No predictions from the model! (CondInst)

[zeeshanalipanhwar]

Hi,

I am using CondInst for Multi-class Instance Segmentation for a custom dataset.

Here is what the the log for training looks like from the bottom.

.
.
.
[01/04 08:30:47 fvcore.common.checkpoint]: Saving checkpoint to training_dir/CondInst_MS_R_50_1x/model_final.pth
[01/04 08:30:48 d2.utils.events]:  eta: 0:00:00  iter: 8  total_loss: 3.187  loss_fcos_cls: 1.137  loss_fcos_loc: 0.3397  
loss_fcos_ctr: 0.7088  loss_mask: 0.9942  time: 1.6907  data_time: 0.1392  lr: 8.992e-08  max_mem: 6693M
[01/04 08:30:48 d2.engine.hooks]: Overall training speed: 6 iterations in 0:00:11 (1.9725 s / it)
[01/04 08:30:48 d2.engine.hooks]: Total training time: 0:00:13 (0:00:01 on hooks)
.
.
.
[01/04 08:30:59 d2.evaluation.evaluator]: Start inference on 1024 images
/content/AdelaiDet/adet/modeling/fcos/fcos_outputs.py:460: UserWarning: This overload of nonzero is deprecated:
    nonzero()
Consider using one of the following signatures instead:
    nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  per_candidate_nonzeros = per_candidate_inds.nonzero()
[01/04 08:31:01 d2.evaluation.evaluator]: Inference done 11/1024. 0.0762 s / img. ETA=0:01:18
[01/04 08:31:06 d2.evaluation.evaluator]: Inference done 75/1024. 0.0768 s / img. ETA=0:01:14
[01/04 08:31:11 d2.evaluation.evaluator]: Inference done 139/1024. 0.0770 s / img. ETA=0:01:09
[01/04 08:31:16 d2.evaluation.evaluator]: Inference done 203/1024. 0.0771 s / img. ETA=0:01:04
[01/04 08:31:21 d2.evaluation.evaluator]: Inference done 267/1024. 0.0772 s / img. ETA=0:00:59
[01/04 08:31:26 d2.evaluation.evaluator]: Inference done 331/1024. 0.0773 s / img. ETA=0:00:54
[01/04 08:31:31 d2.evaluation.evaluator]: Inference done 394/1024. 0.0775 s / img. ETA=0:00:49
[01/04 08:31:36 d2.evaluation.evaluator]: Inference done 457/1024. 0.0777 s / img. ETA=0:00:44
[01/04 08:31:41 d2.evaluation.evaluator]: Inference done 520/1024. 0.0778 s / img. ETA=0:00:39
[01/04 08:31:46 d2.evaluation.evaluator]: Inference done 583/1024. 0.0780 s / img. ETA=0:00:34
[01/04 08:31:51 d2.evaluation.evaluator]: Inference done 646/1024. 0.0781 s / img. ETA=0:00:30
[01/04 08:31:56 d2.evaluation.evaluator]: Inference done 709/1024. 0.0782 s / img. ETA=0:00:25
[01/04 08:32:01 d2.evaluation.evaluator]: Inference done 771/1024. 0.0783 s / img. ETA=0:00:20
[01/04 08:32:06 d2.evaluation.evaluator]: Inference done 833/1024. 0.0784 s / img. ETA=0:00:15
[01/04 08:32:11 d2.evaluation.evaluator]: Inference done 895/1024. 0.0785 s / img. ETA=0:00:10
[01/04 08:32:16 d2.evaluation.evaluator]: Inference done 957/1024. 0.0786 s / img. ETA=0:00:05
[01/04 08:32:21 d2.evaluation.evaluator]: Inference done 1019/1024. 0.0787 s / img. ETA=0:00:00
[01/04 08:32:22 d2.evaluation.evaluator]: Total inference time: 0:01:21.761356 (0.080237 s / img per device, on 1 devices)
[01/04 08:32:22 d2.evaluation.evaluator]: Total inference pure compute time: 0:01:20 (0.078751 s / img per device, on 1 devices)
[01/04 08:32:22 d2.evaluation.coco_evaluation]: Preparing results for COCO format ...
[01/04 08:32:22 d2.evaluation.coco_evaluation]: Saving results to 
training_dir/CondInst_MS_R_50_1x/inference/coco_instances_results.json
[01/04 08:32:22 d2.evaluation.coco_evaluation]: Evaluating predictions with unofficial COCO API...
WARNING [01/04 08:32:22 d2.evaluation.coco_evaluation]: No predictions from the model!
WARNING [01/04 08:32:22 d2.evaluation.coco_evaluation]: No predictions from the model!
[01/04 08:32:22 d2.engine.defaults]: Evaluation results for data_in_mscoco_format_test in csv format:
[01/04 08:32:22 d2.evaluation.testing]: copypaste: Task: bbox
[01/04 08:32:22 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl
[01/04 08:32:22 d2.evaluation.testing]: copypaste: nan,nan,nan,nan,nan,nan
[01/04 08:32:22 d2.evaluation.testing]: copypaste: Task: segm
[01/04 08:32:22 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl
[01/04 08:32:22 d2.evaluation.testing]: copypaste: nan,nan,nan,nan,nan,nan
[01/04 08:32:22 d2.utils.events]:  eta: 0:00:00  iter: 8  total_loss: 3.187  loss_fcos_cls: 1.137  loss_fcos_loc: 0.3397  
loss_fcos_ctr: 0.7088  loss_mask: 0.9942  time: 1.6907  data_time: 0.1392  lr: 8.992e-08  max_mem: 6693M

[tianzhi0549]

How many iterations did you train the model?

[zeeshanalipanhwar]

I have tried 9, and 90 iterations. Got the same results.

[zeeshanalipanhwar]

Getting empty coco_instances_results.json.

[tianzhi0549]

@zeeshanalipanhwar We generally train the model for 90K iterations.

[zeeshanalipanhwar]

Right. I have another question. In coco_instances_results.json, what would segmentation be, a list of coordinates as in my train annotations.json or a dictionary with keys counts: list and size: list[h, w]? Thank you.

[zeeshanalipanhwar]

Because when I use Detectron2's MaskRCNN on the same data, I am not getting no results. I get bboxes right. But, segmentation values are dictionaries with keys counts: string (expected list according to this format) and size: list[h, w]. Could I then get the same issue with CondInst when I get some results.

[zeeshanalipanhwar]

Notice in the log that I am getting loss values for training as well as inference. But no predictions.

[zeeshanalipanhwar]

Resolved. I might have modified code somewhere such that it might have raised any exception or something. IDK. One other possible reason could be that I had some unexpected samples, ie there were instances, chunks of them in the patches, with only one or two pixels.

[Dandelionym]

Hi, I also met this problem. Could you please give me some suggestions?

[an99990]

have you resolved this @Dandelionym ?

[zxf29]

Hi, I also met this problem. Could you please give me some suggestions?