Training models with different amount of scenes

[BoynextDoor114514]

Hello :) I want to train my models with different amount of scenes(100, 300, 500, 700), all I should do is to delete scenes from train_detect and train_track in split.py, is that right?

Thank you sincerely!! : )

[whyekit-motional]

@BoynextDoor114514 in addition to deleting the scenes you don't want from the splits, you might also need to remove the corresponding samples from nusc (if you are doing eval), e.g.: https://github.com/nutonomy/nuscenes-devkit/blob/9df79f48bac3cc713f4dbd2b48049be845605c51/python-sdk/nuscenes/eval/common/loaders.py#L69

[BoynextDoor114514]

Thank you so much, it works good for me! And I have another question, at the evaluation phase after loading NuScenes tables, the usage of memory became so large and caused the OOM Killer to kill the process : ( My RAM is 16G, do I have to expand my RAM or there is any way to fix this problem?

[whyekit-motional]

Perhaps you could first check if there are any other (potentially heavy) processes that are occupying your RAM

Otherwise, yes, you would probably have to use a machine with more RAM :smile:

[michael-hoss]

Hi @BoynextDoor114514, if relevant for you, once my PR https://github.com/nutonomy/nuscenes-devkit/pull/1046 merges, you can evaluate on your own splits in a custom splits.json without needing to touch the splits.py code.

[BoynextDoor114514]

@BoynextDoor114514 in addition to deleting the scenes you don't want from the splits, you might also need to remove the corresponding samples from nusc (if you are doing eval), e.g.:

https://github.com/nutonomy/nuscenes-devkit/blob/9df79f48bac3cc713f4dbd2b48049be845605c51/python-sdk/nuscenes/eval/common/loaders.py#L69

Unfortunately, I may have failed. I use OpenPCDet to train the models with different amount of scenes: 5%(35 scenes) and 10%(70 scenes). But I've noticed that in these 2 different conditions, the total items each epoch has not changed(in the image is 6110), I thought this parameter should be different. My steps are: 1. deleting the scenes from split.py, 2. use date preparation tools to create data_info, 3. train the model. Do you have any idea to help me locate the problem?

Sorry for disturbing you and thank you sincerely!

[whyekit-motional]

@BoynextDoor114514 you could try isolating the problem by initializing the data-loader, and checking its length

[BoynextDoor114514]

@BoynextDoor114514 in addition to deleting the scenes you don't want from the splits, you might also need to remove the corresponding samples from nusc (if you are doing eval), e.g.:

https://github.com/nutonomy/nuscenes-devkit/blob/9df79f48bac3cc713f4dbd2b48049be845605c51/python-sdk/nuscenes/eval/common/loaders.py#L69

hello, I've completed training process with 140 scenes. But at the evaluation phase, an error occurred in "assert set(self.pred_boxes.sample_tokens) == set(self.gt_boxes.sample_tokens)" : "AssertionError: Samples in split doesn't match samples in predictions."

I try to fix this problem and in my opinion, the gt_boxes come from here https://github.com/nutonomy/nuscenes-devkit/blob/4df2701feb3436ae49edaf70128488865a3f6ff9/python-sdk/nuscenes/eval/detection/evaluate.py#L96C13-L96C93 so since I haven't changed the scenes in validation set in split.py, the samples for evaluation still remained, so maybe the error shouldn't happen.

I'm confused and could you plz give me some idea to fix it? Thank you sincerely!

[whyekit-motional]

@BoynextDoor114514 if you did not change the scenes to be used in the val split, then you should check whether you have predictions for each sample in each scene of the val split

One way to do this could be to load up your JSON file containing all your predictions and checking that each sample token in the val split is present in your JSON file

[BoynextDoor114514]

@BoynextDoor114514 if you did not change the scenes to be used in the val split, then you should check whether you have predictions for each sample in each scene of the val split

One way to do this could be to load up your JSON file containing all your predictions and checking that each sample token in the val split is present in your JSON file

I've checked my evaluation process and I noticed that the predicted samples are much more than the samples in val-set. Nearly the rest samples of the full trainval dataset are predicted so the result.pkl is very large. The amount of gt-samples seems good as I wish. So which scenes or samples will be predicted during the evaluation and then written in result_nusc.json? I haven't find which part of code determine it. :-(

[whyekit-motional]

@BoynextDoor114514 the samples which get inferred (a.k.a. predicted) on and written into a json is entirely determined by the user's code - the DetectionEval in nuscenes-devkit only consumes the json file provided by the user