About nuScenes config file.

[Andhow0603]

Hi, Thank you so much for sharing your impressive work! Could you please send the nuScenes config to me? E-mail: howard8682442@gmail.com

Many Thanks!!!

[mmmfffzz]

Hi!@xhjiang-pixel, Thank you for sharing your outstanding work. Could you please share the nuScenes config file with me mingfanggzhang@163.com ? Thanks very much!!!

[rlin1538]

Hello @xhjiang-pixel , thank you for bringing us such good work. I would like to train on the nuScenes dataset. Can you share a copy of your configuration file? Thank you very much! E-mail: zwlin@nuaa.edu.cn

[Manato1fg]

Hello @xhjiang-pixel Thank you so much for your great works. I also want to try this project on nuScenes. Can you share the config file for nuScenes with me? Thank you.

Mail: manato-y@g.ecc.u-tokyo.ac.jp

[Manato1fg]

It's not perfect yet, but here are some tips for training on the nuScenes dataset. You should copy the config file for Argoverse2 and follow the instructions below to modify it.

I mainly referred StreamPETR.

Class Names:

Since nuScenes has different class names compared to Argoverse2, you must update the configuration file accordingly.

class_names = [
    'car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier',
    'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'
]

Additionally, the num_classes value is hardcoded in the configuration, so you need to change it from 26 to 10. I recommend using len(class_names) instead of hardcoding 10 to make it easier to extend the class list later.

Image Normalization Configuration (img_norm_cfg):

The camera parameters are also different between the two datasets.

img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)

Bounding Box:

Ground truth bounding boxes in nuScenes include additional velocity information, which is not used in Far3D. However, you still need to modify the config to ensure the model runs properly.

model = dict(
    type='Far3D',
    ...
    pts_bbox_head=dict(
        type='FarHead',
        ...
        code_size=10,  # It's 8 in Argoverse2
        code_weights = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0], # Two 0.0 values added
...
)

Dataset Path:

You should have created the nuScenes dataset using tools/create_data_nusc.py. After that, you'll need to update the dataset path and PKL file names.

dataset_type = 'CustomNuScenesDataset'
data_root = 'path/to/nuscenes'

data = dict(
    samples_per_gpu=1,
    workers_per_gpu=4,
    train=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + 'nuscenes2d_temporal_infos_train.pkl', # <- here
        num_frame_losses=num_frame_losses,
        random_length=1, #for sliding window training
        pipeline=train_pipeline,
        classes=class_names,
        modality=input_modality,
        collect_keys=collect_keys + ['img', 'prev_exists', 'img_metas'],
        queue_length=queue_length,
        test_mode=False,
        use_valid_flag=True,
        filter_empty_gt=False,
        box_type_3d='LiDAR'),
    val=dict(type=dataset_type, pipeline=test_pipeline, collect_keys=collect_keys + ['img', 'img_metas'], queue_length=queue_length, ann_file=data_root + 'nuscenes2d_temporal_infos_val.pkl', classes=class_names, modality=input_modality), 
    test=dict(type=dataset_type, pipeline=test_pipeline, collect_keys=collect_keys + ['img', 'img_metas'], queue_length=queue_length, ann_file=data_root + 'nuscenes2d_temporal_infos_val.pkl', classes=class_names, modality=input_modality),
    shuffler_sampler=dict(type='InfiniteGroupEachSampleInBatchSampler'),
    nonshuffler_sampler=dict(type='DistributedSampler')
)

Modify some code

You need to modify some code due to certain limitations. There may be alternative solutions to this problem within the config file, but this approach works.

In projects/mmdet3d_plugin/models/detectors/far3d.py: projects/mmdet3d_plugin/models/detectors/far3d.py

@DETECTORS.register_module()
class Far3D(MVXTwoStageDetector):
        ...
        def forward_train(self,
                      img_metas=None,
                      gt_bboxes_3d=None,
                      gt_labels_3d=None,
                      gt_labels=None,
                      gt_bboxes=None,
                      gt_bboxes_ignore=None,
                      depths=None,
                      centers2d=None,
                      **data):
        """
        ...
        """

        # NuScenes dataset has multidimensional array.
        if type(img_metas[0]) is list:
            img_metas = img_metas[0]
            gt_bboxes_3d = None if gt_bboxes_3d is None else gt_bboxes_3d[0]
            gt_labels_3d = None if gt_labels_3d is None else gt_labels_3d[0]
            gt_labels = None if gt_labels is None else gt_labels[0]
            gt_bboxes = None if gt_bboxes is None else gt_bboxes[0]
            centers2d = None if centers2d is None else centers2d[0]
            depths = None if depths is None else depths[0]
        ...

Please feel free to ask any questions or suggest any updates you may have.