search

UARK-AICV / AISFormer

[BMVC 2022] AISFormer: Amodal Instance Segmentation with Transformer
https://uark-aicv.github.io/AISFormer/
Apache License 2.0
28 stars 10 forks source link

Model evaluation problem <KeyError: 'inmodal_seg'> #4

Closed mingheyuemankong closed 1 year ago

mingheyuemankong commented 1 year ago

Hello, I trained and evaluated the model on D2SA and COCOA cls datasets. The training process on the two datasets is normal, but the following problems appear in the evaluation process. Taking COCOA cls as an example, how to solve this problem? I look forward to your reply.@trqminh

[02/09 23:14:01 d2.evaluation.evaluator]: Start inference on 1223 batches [02/09 23:14:03 d2.evaluation.evaluator]: Inference done 11/1223. Dataloading: 0.0005 s/iter. Inference: 0.0381 s/iter. Eval: 0.0236 s/iter. Total: 0.0623 s/iter. ETA=0:01:15 [02/09 23:14:08 d2.evaluation.evaluator]: Inference done 93/1223. Dataloading: 0.0007 s/iter. Inference: 0.0392 s/iter. Eval: 0.0215 s/iter. Total: 0.0614 s/iter. ETA=0:01:09 [02/09 23:14:13 d2.evaluation.evaluator]: Inference done 184/1223. Dataloading: 0.0007 s/iter. Inference: 0.0379 s/iter. Eval: 0.0197 s/iter. Total: 0.0584 s/iter. ETA=0:01:00 [02/09 23:14:19 d2.evaluation.evaluator]: Inference done 267/1223. Dataloading: 0.0007 s/iter. Inference: 0.0381 s/iter. Eval: 0.0202 s/iter. Total: 0.0590 s/iter. ETA=0:00:56 [02/09 23:14:24 d2.evaluation.evaluator]: Inference done 350/1223. Dataloading: 0.0007 s/iter. Inference: 0.0381 s/iter. Eval: 0.0205 s/iter. Total: 0.0594 s/iter. ETA=0:00:51 [02/09 23:14:29 d2.evaluation.evaluator]: Inference done 439/1223. Dataloading: 0.0007 s/iter. Inference: 0.0379 s/iter. Eval: 0.0200 s/iter. Total: 0.0587 s/iter. ETA=0:00:46 [02/09 23:14:34 d2.evaluation.evaluator]: Inference done 518/1223. Dataloading: 0.0008 s/iter. Inference: 0.0383 s/iter. Eval: 0.0204 s/iter. Total: 0.0594 s/iter. ETA=0:00:41 [02/09 23:14:39 d2.evaluation.evaluator]: Inference done 602/1223. Dataloading: 0.0008 s/iter. Inference: 0.0382 s/iter. Eval: 0.0204 s/iter. Total: 0.0595 s/iter. ETA=0:00:36 [02/09 23:14:44 d2.evaluation.evaluator]: Inference done 679/1223. Dataloading: 0.0008 s/iter. Inference: 0.0385 s/iter. Eval: 0.0209 s/iter. Total: 0.0601 s/iter. ETA=0:00:32 [02/09 23:14:49 d2.evaluation.evaluator]: Inference done 765/1223. Dataloading: 0.0008 s/iter. Inference: 0.0384 s/iter. Eval: 0.0207 s/iter. Total: 0.0599 s/iter. ETA=0:00:27 [02/09 23:14:54 d2.evaluation.evaluator]: Inference done 850/1223. Dataloading: 0.0008 s/iter. Inference: 0.0384 s/iter. Eval: 0.0207 s/iter. Total: 0.0599 s/iter. ETA=0:00:22 [02/09 23:14:59 d2.evaluation.evaluator]: Inference done 935/1223. Dataloading: 0.0008 s/iter. Inference: 0.0384 s/iter. Eval: 0.0206 s/iter. Total: 0.0598 s/iter. ETA=0:00:17 [02/09 23:15:04 d2.evaluation.evaluator]: Inference done 1020/1223. Dataloading: 0.0008 s/iter. Inference: 0.0383 s/iter. Eval: 0.0206 s/iter. Total: 0.0598 s/iter. ETA=0:00:12 [02/09 23:15:09 d2.evaluation.evaluator]: Inference done 1103/1223. Dataloading: 0.0008 s/iter. Inference: 0.0383 s/iter. Eval: 0.0206 s/iter. Total: 0.0598 s/iter. ETA=0:00:07 [02/09 23:15:14 d2.evaluation.evaluator]: Inference done 1177/1223. Dataloading: 0.0008 s/iter. Inference: 0.0385 s/iter. Eval: 0.0210 s/iter. Total: 0.0603 s/iter. ETA=0:00:02 [02/09 23:15:17 d2.evaluation.evaluator]: Total inference time: 0:01:13.711335 (0.060518 s / iter per device, on 1 devices) [02/09 23:15:17 d2.evaluation.evaluator]: Total inference pure compute time: 0:00:46 (0.038536 s / iter per device, on 1 devices) [02/09 23:15:17 d2.evaluation.amodal_visible_evaluation]: Preparing results for COCO format ... [02/09 23:15:17 d2.evaluation.amodal_visible_evaluation]: Saving results to ../YF-AIS-out/train_outputs/aisformer_R_50_FPN_1x_amodal_cocoa_cls/inference/coco_instances_amodal_results.json [02/09 23:15:17 d2.evaluation.amodal_visible_evaluation]: Saving results to ../YF-AIS-out/train_outputs/aisformer_R_50_FPN_1x_amodal_cocoa_cls/inference/coco_instances_amodal_occ_results.json [02/09 23:15:17 d2.evaluation.amodal_visible_evaluation]: Saving results to ../YF-AIS-out/train_outputs/aisformer_R_50_FPN_1x_amodal_cocoa_cls/inference/coco_instances_visible_results.json [02/09 23:15:18 d2.evaluation.amodal_visible_evaluation]: number of small occlusion instances in prediction:70 [02/09 23:15:18 d2.evaluation.amodal_visible_evaluation]: number of medium occlusion instances in prediction:2 [02/09 23:15:18 d2.evaluation.amodal_visible_evaluation]: number of heavy occlusion instances in prediction:0 /home/jsj308/anaconda3/envs/AISF/lib/python3.7/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1659484809535/work/aten/src/ATen/native/TensorShape.cpp:2894.) return _VF.meshgrid(tensors, *kwargs) # type: ignore[attr-defined] Traceback (most recent call last): File "tools/train_net.py", line 171, in args=(args,), File "/home/jsj308/D/YYF/AISFormer/detectron2/engine/launch.py", line 82, in launch main_func(args) File "tools/train_net.py", line 141, in main res = Trainer.test(cfg, model) File "/home/jsj308/D/YYF/AISFormer/detectron2/engine/defaults.py", line 617, in test results_i = inference_on_dataset(model, data_loader, evaluator) File "/home/jsj308/D/YYF/AISFormer/detectron2/evaluation/evaluator.py", line 204, in inference_on_dataset results = evaluator.evaluate() File "/home/jsj308/D/YYF/AISFormer/detectron2/evaluation/amodal_visible_evaluation.py", line 253, in evaluate self._eval_predictions(set(self._tasks)) File "/home/jsj308/D/YYF/AISFormer/detectron2/evaluation/amodal_visible_evaluation.py", line 399, in _eval_predictions if coco_api_eval_visible.anns[key][visible_name].len() > 2: KeyError: 'inmodal_seg'

mingheyuemankong commented 1 year ago

I noticed that the author wrote the fields about segmentation, i_segmentation and bg_object_segmentation in https://github.com/UARK-AICV/AISFormer/blob/master/detectron2/data/detection_utils.py , but it seems that the author did not write the above fields in https://github.com/UARK-AICV/AISFormer/blob/master/detectron2/evaluation/amodal_visible_evaluation.py . I guess this is the reason for the problems in the evaluation. I look forward to your reply. @trqminh

trqminh commented 1 year ago

I prepared these scripts to handle this error, you could run them before running the evaluation

https://github.com/UARK-AICV/AISFormer/blob/master/tools/proc_cocoa.py https://github.com/UARK-AICV/AISFormer/blob/master/tools/proc_d2sa.py

li-pengcheng commented 1 year ago 
data = proc_vis_mask_iseg(data)
ann['inmodal_seg'] = ann.pop('visible_mask')

KeyError: 'visible_mask'

Noiredd commented 1 year ago

@li-pengcheng The first secret is that you're not supposed to run detectron2.data.datasets.process_data_amodal on validation data, only the training split. The second secret - you have to run the above scripts before the preprocessing. So the complete procedure is:

  1. run proc_cocoa.py/proc_d2sa.py on all your data
  2. run detectron2.data.datasets.process_data_amodal on your training split

@trqminh Why not just update detectron2.data.datasets.process_data_amodal and fix it all there? This tends to catch by surprise ;)