[Docs] Lazy loading annotations

[austinmw]

Branch

master branch https://mmdetection.readthedocs.io/en/latest/

📚 The doc issue

Does MMDetection currently support lazy loading annotations when they don't all fit into memory (ex. loading a coco train_2017.json results in OOM)? If so, what's the easiest way to enable this?

Suggest a potential alternative/fix

No response

[PeterVennerstrom]

The annotations could be saved as per image files containing a single image's annotations and read in during training. The custom_dataset class could be adjusted not to pre-load the annotations and the get_ann_info() method adjusted to read them in from file by index.

If masks aren't used they could be removed. RLE encoding for masks may reduce memory. COCO panoptic datasets store the mask data in png files which are read in as needed.

[austinmw]

So to make sure, creating a custom dataset does not require the CustomDataset.load_annotations method to return all annotations? I see that the __init__ uses it to create data_infos, so I could remove that. I just want to make sure there's no other dependencies related to callbacks, evaluation, etc. that requires loading everything upfront.

In general, does the CustomDataset require map-style with __getitem__, and __len__ or could we use an iterable-style and __iter__ instead?

Also, if I use a sharded tar format like WebDataset (so there might be something like 100 images and annotations per file), is there anything I should take care of in terms of distributed training? Can anyone point me to the code that paritions the dataset for distributed training?

[PeterVennerstrom]

I'd guess most of the dependancies are in terms of the images not the annotations. The existing code stores image/annotations in per image dictionaries within the list assigned to self.data_infos.

CustomDataset inherits torch Dataset which requires __getitem__ and __len__. They have some nice documentation. Torch IterableDataset is also option to build your dataset, but I think the mmdet datasets are built on the map-style only.

The partitioning for distributed training is here. It inherits from the torch DistributedSampler.

WebDataset requires a torch IterableDataset.

[hhaAndroid]

@austinmw The main branch is currently not supported, but the 3.x branch supports it

test_dataset_cfg = copy.deepcopy(config.test_dataloader.dataset)
# lazy init. We only need the metainfo.
test_dataset_cfg['lazy_init'] = True

DATASETS.build(test_dataset_cfg)

[Looottch]

@hhaAndroid can you give a code link for lazy_init here

@austinmw The main branch is currently not supported, but the 3.x branch supports it

test_dataset_cfg = copy.deepcopy(config.test_dataloader.dataset)
# lazy init. We only need the metainfo.
test_dataset_cfg['lazy_init'] = True

DATASETS.build(test_dataset_cfg)

[jamesbc123]

So to make sure, creating a custom dataset does not require the CustomDataset.load_annotations method to return all annotations? I see that the __init__ uses it to create data_infos, so I could remove that. I just want to make sure there's no other dependencies related to callbacks, evaluation, etc. that requires loading everything upfront.

In general, does the CustomDataset require map-style with __getitem__, and __len__ or could we use an iterable-style and __iter__ instead?

Also, if I use a sharded tar format like WebDataset (so there might be something like 100 images and annotations per file), is there anything I should take care of in terms of distributed training? Can anyone point me to the code that paritions the dataset for distributed training?

@austinmw Did you make the changes to load the annotations just when needed and it worked out okay? As in by making changes to the code, and not using MMDetection 3.0.0 as hhaAndroid mentioned.

[Bing1002]

Any plan to integrate webdataset as another dataloader option? @hhaAndroid

[KyanChen]

Any plan to integrate webdataset as another dataloader option?