KeyError: 'labels' in demo.ipynb (with custom model)

[HireTheHero]

Summary of the Issue

Hi, thanks for all your effort spent on this repo.

Trying to setup a working example of your demo.ipynb in Google Colaboratory, with some custom settings. For example, model used in my code is different from original one - full code is temporarily pushed to my own repository so that you can reproduce it *1 .

In the last cell(*1) of the demo, I got KeyError: 'labels' (full error message in *2) . Looks like #947 of maskrcnn-benchmark is the same or similar problem, but so far unfortunately I didn't find the answer for it.

I believe this is the last piece of puzzle for completing working example, and when fixed happily make pull request to add my notebook as working example, since at least a few issues seems related to demo.ipynb .

Thanks!

Reference

*1 (last cell)

image_path = 'demo/1.jpg'
features, infos = feature_extractor.extract_features(image_path)

img = PIL.Image.open(image_path).convert('RGB')
img = torch.tensor(np.array(img))

plt.axis('off')
plt.imshow(img)
plt.show()

query = "swimming elephant"
task = [9]
custom_prediction(query, task, features, infos)

*2 (full error message)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-29-af2245ff7ddc> in <module>()
      4 
      5 image_path = 'demo/1.jpg'
----> 6 features, infos = feature_extractor.extract_features(image_path)
      7 
      8 img = PIL.Image.open(image_path).convert('RGB')

<ipython-input-24-8fae1b994812> in extract_features(self, image_path)
    162     def extract_features(self, image_path):
    163 
--> 164         features, infos = self.get_detectron_features([image_path])
    165 
    166         return features, infos

<ipython-input-24-8fae1b994812> in get_detectron_features(self, image_paths)
    135 
    136         with torch.no_grad():
--> 137             output = self.detection_model(current_img_list)
    138 
    139         feat_list = self._process_feature_extraction(

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/content/vilbert-multi-task/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py in forward(self, images, targets, proposals)
     54             proposal_losses = {}
     55         if self.roi_heads:
---> 56             x, result, detector_losses = self.roi_heads(features, proposals, targets)
     57         else:
     58             # RPN-only models don't have roi_heads

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/content/vilbert-multi-task/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py in forward(self, features, proposals, targets)
     21         losses = {}
     22         # TODO rename x to roi_box_features, if it doesn't increase memory consumption
---> 23         x, detections, loss_box = self.box(features, proposals, targets)
     24         losses.update(loss_box)
     25         if self.cfg.MODEL.MASK_ON:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/content/vilbert-multi-task/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/box_head.py in forward(self, features, proposals, targets)
     49         if self.return_feats:
     50             if self.has_attributes:
---> 51                 out_dict = self.predictor(x, proposals)
     52                 attr_logits = out_dict["attr_score"]
     53             else:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/content/vilbert-multi-task/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/roi_box_predictors.py in forward(self, x, proposals)
     78 
     79                 # get labels and indices of proposals with foreground
---> 80                 all_labels = cat([prop.get_field("labels") for prop in proposals], dim=0)
     81                 fg_idx = all_labels > 0
     82                 fg_labels = all_labels[fg_idx]

/content/vilbert-multi-task/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/roi_box_predictors.py in <listcomp>(.0)
     78 
     79                 # get labels and indices of proposals with foreground
---> 80                 all_labels = cat([prop.get_field("labels") for prop in proposals], dim=0)
     81                 fg_idx = all_labels > 0
     82                 fg_labels = all_labels[fg_idx]

/content/vilbert-multi-task/vqa-maskrcnn-benchmark/maskrcnn_benchmark/structures/bounding_box.py in get_field(self, field)
     38 
     39     def get_field(self, field):
---> 40         return self.extra_fields[field]
     41 
     42     def has_field(self, field):

KeyError: 'labels'

[snowyrain]

I has the same problem