Data preparation and modification of config file for training with custom dataset

[mr-spielberg]

Hey, Thank you very much for sharing your great work! I am currently investigating how state-of-the-art panoptic segmentation networks perform on fisheye images. I have obtained good results by applying your network with the published weights to fisheye images using the "demo.py" script in detectron2. Now, I would like to examine if training with fisheye images can boost the performance further. I intend to use the recently published WoodScape dataset (https://github.com/valeoai/WoodScape).

I have already worked through the issues. Please let me know if I understood the process for preparing the dataset correctly:

1. The dataset needs to be transformed into the COCO panoptic format specified on https://cocodataset.org/#format-data
2. The function "separate_coco_semantic_from_panoptic" from https://github.com/facebookresearch/detectron2/blob/master/datasets/prepare_panoptic_fpn.py needs to be run since you use the PanopticFPN format.
3. The dataset needs to be registered using the function "register_coco_panoptic_separated" from https://github.com/facebookresearch/detectron2/blob/master/detectron2/data/datasets/coco_panoptic.py. Do I need to provide extra metadata here?

Did I miss anything?

Concerning the modification of the config file, I could only find issues regarding the Cityscapes and Mapillary Vistas datasets. Could you please explain which parameters need to be adjusted for the use of custom datasets and how I can choose the right values (e.g., for POSITION_HEAD/THING/POS_NUM, KERNEL_HEAD/INSTANCE_SCALES, TENSOR_DIM, etc.)?

Can I then just start the training with "python3 projects/PanopticFCN/train.py --config-file --num-gpus " or are there further adjustments of the code required?

I am looking forward to your response. Thank you very much!

[yanwei-li]

Hi, thanks for your interest in our work! You need to modify the following aspects in your own dataset:

1. Dataset

   • Except the things you mentioned above, you also need to add the image or annotation processing file like that in detectron2/master/detectron2/data/datasets/coco_panoptic.py.

2. Config

   • The dataset register DATASETS.TRAIN according to your definition.
   • Positive numbers within each object for optimization POSITION_HEAD.THING.POS_NUM. (This value is up to you. The default one is enough for most cases.)
   • The optimized scale ranges in FPN KERNEL_HEAD.INSTANCE_SCALES. If your instance scale is too large, try to turn this range up, and vice versa.
   • The max number of object in each image during optimization TENSOR_DIM.
   • Number of background stuff categories MODEL.POSITION_HEAD.STUFF.NUM_CLASSES and MODEL.SEM_SEG_HEAD.NUM_CLASSES. It's up to your predefined annotation.
   • Number of foreground thing categories MODEL.POSITION_HEAD.THING.NUM_CLASSES. It's up to your predefined annotation.