search

microsoft / scene_graph_benchmark

image scene graph generation benchmark
MIT License
382 stars 86 forks source link

Index error when accessing box features for RelDN model #35

Open aleSuglia opened 3 years ago

aleSuglia commented 3 years ago

Hi there, I'm trying to use the model sgg_configs/vg_rvd/rel_danfeiX_FPN50_reldn.yaml for generating scene graphs for a custom dataset. The bounding box proposals works fine however seems that there is a bug in the way the proposal_pairs are computed. In particular, I get the following exception:

Traceback (most recent call last):
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 103, in profile
    yield action_name
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1088, in run_predict
    self.predict_loop.predict_step(batch, batch_idx, dataloader_idx)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/pytorch_lightning/trainer/predict_loop.py", line 111, in predict_step
    predictions = self.trainer.accelerator.predict_step(args)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 265, in predict_step
    return self.training_type_plugin.predict_step(*args)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 167, in predict_step
    return self.lightning_module.predict_step(*args, **kwargs)
  File "/Users/asuglia/workspace/scene_graph_benchmark/tools/video/extract_features.py", line 311, in predict_step
    predictions = self.model(images)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/asuglia/workspace/scene_graph_benchmark/scene_graph_benchmark/scene_parser.py", line 319, in forward
    x_pairs, prediction_pairs, relation_losses = self.relation_head(features, predictions, targets)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/asuglia/workspace/scene_graph_benchmark/scene_graph_benchmark/relation_head/relation_head.py", line 211, in forward
    = self.rel_predictor(features, proposals, proposal_pairs)
  File "/Users/asuglia/opt/anaconda3/envs/devel/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/asuglia/workspace/scene_graph_benchmark/scene_graph_benchmark/relation_head/reldn/reldn.py", line 103, in forward
    sub_vert_per_image = proposal_per_image.get_field("subj_box_features")[rel_ind_i[:, 0]]
IndexError: index 49 is out of bounds for dimension 0 with size 38

This seems to be due to the fact the indexes of bounding boxes in rel_ind_i cause out of bounds error because there are fewer bounding boxes in proposal_per_image.get_field("subj_box_features"). In this specific case I can see that proposal_per_image.get_field("subj_box_features") has the following shape: torch.Size([38, 1024]). While, rel_ind_i[:, 0] has the following indexes:

tensor([49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50,
        50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51,
        51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52,
        52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52,
        52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53,
        53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53,
        53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54,
        54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55,
        55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56,
        57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58,
        58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59,
        59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61,
        61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61,
        61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 63,
        63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64,
        64, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65,
        65, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67,
        67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67,
        67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68,
        68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68,
        68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69,
        69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71,
        71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72,
        72, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 73,
        73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74,
        74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75,
        75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75,
        75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76,
        76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76,
        76, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78,
        78, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80,
        80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80,
        80, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 81,
        81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82,
        82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82,
        82, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83,
        83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85,
        85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86,
        86, 86, 86, 86, 86, 86, 86, 86])

I'm running this code with batch size 2. I thought that the error could have been in the way the proposal_pairs are generated. However, the exception happens when either of these lines are executed to generate the proposal_pairs:

  1. https://github.com/microsoft/scene_graph_benchmark/blob/main/scene_graph_benchmark/relation_head/relation_head.py#L187
  2. https://github.com/microsoft/scene_graph_benchmark/blob/main/scene_graph_benchmark/relation_head/relation_head.py#L185

Do you think that offset is required here: https://github.com/microsoft/scene_graph_benchmark/blob/main/scene_graph_benchmark/relation_head/reldn/reldn.py#L98

@hanxiaotian Could you please advise?