Custom dataset, dataloader returns list[list[dict]]

[holmbuar]

I am having a curious problem with a custom dataset, registered with a custom dataset function.

The Detectron2 setup and cfg is copied from Detic Colab tutorial

A snippet of my code:

from detectron2.modeling import build_model
from detectron2.data import build_detection_test_loader
import torch

def my_dataset_generator():
    data_list = []
    for index, observation in df.iterrows():
        data_dict = {}
        data_dict["file_name"] = observation['file_path']
        data_dict['image_id'] = observation['image']
        data_list.append(data_dict)
    return data_list

DatasetCatalog.register("my_whales3", my_dataset_generator)

model = build_model(cfg)  # returns a torch.nn.Module
test_loader = build_detection_test_loader(cfg, 'my_whales3')

which returns

[03/01 12:05:54 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')]
[03/01 12:05:54 d2.data.common]: Serializing 51033 elements to byte tensors and concatenating them all ...
[03/01 12:05:54 d2.data.common]: Serialized dataset takes 6.42 MiB

Note that the output format of my_dataset_generator() is list[dict]. Also, the df in the function is the actual image metadata Pandas dataframe.

Running

model.eval()
with torch.no_grad():
    outputs = model(test_loader)

produces this traceback:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_3448/2278559883.py in <module>
      1 model.eval()
      2 with torch.no_grad():
----> 3     outputs = model(test_loader)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

~/Detic/detic/modeling/meta_arch/custom_rcnn.py in forward(self, batched_inputs)
    113         """
    114         if not self.training:
--> 115             return self.inference(batched_inputs)
    116 
    117         images = self.preprocess_image(batched_inputs)

~/Detic/detic/modeling/meta_arch/custom_rcnn.py in inference(self, batched_inputs, detected_instances, do_postprocess)
     94         assert detected_instances is None
     95 
---> 96         images = self.preprocess_image(batched_inputs)
     97         features = self.backbone(images.tensor)
     98         proposals, _ = self.proposal_generator(images, features, None)

/opt/conda/lib/python3.7/site-packages/detectron2/modeling/meta_arch/rcnn.py in preprocess_image(self, batched_inputs)
    222         Normalize, pad and batch the input images.
    223         """
--> 224         images = [x["image"].to(self.device) for x in batched_inputs]
    225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)

/opt/conda/lib/python3.7/site-packages/detectron2/modeling/meta_arch/rcnn.py in <listcomp>(.0)
    222         Normalize, pad and batch the input images.
    223         """
--> 224         images = [x["image"].to(self.device) for x in batched_inputs]
    225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)

TypeError: list indices must be integers or slices, not str

So the x in batched_inputs is a list, not a dict as expected. When I try

data_iter = iter(test_loader)
i = 0
for x in data_iter:
    if i % 1000 == 0:
        print(type(x))
    i = i + 1

the output is

<class 'list'>
<class 'list'>
(...)
<class 'list'>
<class 'list'>

In other words, my data loader seems to return list[list[dict]], instead of list[dict]. Have I goofed, or is this error due to something else?

[holmbuar]

If it is of any help, here is the output of calling outputs, after the error is thrown:

outputs

{'instances': Instances(num_instances=3, image_height=671, image_width=804, fields=[pred_boxes: Boxes(tensor([[  2.0434, 116.0116, 802.5441, 665.5936],
         [  2.0434, 116.0116, 802.5441, 665.5936],
         [  2.0434, 116.0116, 802.5441, 665.5936]], device='cuda:0')), scores: tensor([0.8476, 0.6173, 0.5370], device='cuda:0'), pred_classes: tensor([382, 935, 559], device='cuda:0'), pred_masks: tensor([[[False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          ...,
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False]],

         [[False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          ...,
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False]],

         [[False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          ...,
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False],
          [False, False, False,  ..., False, False, False]]], device='cuda:0')])}

[holmbuar]

Calling

data_iter = iter(test_loader)
i = 0
for x in data_iter:
    if i % 1000 == 0:
        print(x)
    i = i + 1

returns

[{'file_name': '/home/jupyter/happy-whale-and-dolphin/train_images/00021adfb725ed.jpg', 'image_id': '00021adfb725ed.jpg', 'width': 804, 'height': 671, 'image': tensor([[[ 74,  74,  73,  ...,  71,  72,  73],
         [ 76,  76,  77,  ...,  63,  64,  65],
         [ 80,  81,  83,  ...,  60,  60,  61],
         ...,
         [ 44,  42,  43,  ...,  50,  51,  51],
         [ 43,  41,  42,  ...,  49,  50,  50],
         [ 43,  41,  41,  ...,  50,  50,  50]],

        [[120, 120, 119,  ..., 113, 114, 115],
         [122, 122, 123,  ..., 104, 104, 105],
         [126, 127, 130,  ...,  99,  99, 100],
         ...,
         [ 45,  43,  43,  ...,  48,  49,  50],
         [ 46,  44,  43,  ...,  49,  50,  50],
         [ 45,  43,  43,  ...,  50,  50,  50]],

        [[170, 170, 169,  ..., 163, 164, 165],
         [172, 172, 173,  ..., 155, 155, 156],
         [175, 177, 178,  ..., 149, 149, 150],
         ...,
         [ 54,  53,  53,  ...,  58,  59,  60],
         [ 55,  53,  53,  ...,  57,  58,  58],
         [ 57,  55,  55,  ...,  58,  58,  58]]], dtype=torch.uint8)}]