How the training cycle acturally works

[luoyuchenmlcv]

Hi, thanks a lot for presenting your great work!

I am struggling to figure out how the loop actually works, hope you could provide more insights.

I am trying to understand how your code actually works, please help me point out any faulty usage and resolve some of my confusions. Let's take an example by using naive method MAXLogit and MSP and dataset is nuscenes, there are four files involved:

1.train_cylinder_asym_nuscenes_naive.py 2.val_cylinder_asym_nusc.py 3.val_cylinder_asym_nusc_generate_incre_labels.py 4.train_cylinder_asym_nuscenes_ood_incremental.py

By following your training scheme mentioned in paper, there should be: K_new = {barrier, consuction_vehicle, traffic_cone, trailer} K_old = all_classes \ K_new

Denote M_c0 as the initially trained close-set model. If I understand your code correctly, unlike your REAL framework, now trying to train M_c0 to M_c1 with an additional prediction head directly.

###############################################################################

1. run train_cylinder_asym_nuscenes_naive.py. And {barrier, consuction_vehicle, traffic_cone, trailer} labels ignored. And the entire training data should be swept through.

2. run val_cylinder_asym_nusc.py to save ID prediction results and uncertainty scores, but in your code the data_loader is from validation set rather than training set, the saved prediction and uncertainty score are from validation set. but in train_cylinder_asym_nuscenes_ood_incremental the training data still comes from training set.

3. run val_cylinder_asym_nusc_generate_incre_labels.py to generate and save the pseudo labels of the training set, but in your code the data_loader is still from validation set, similar issue with above.

4. run train_cylinder_asym_nuscenes_ood_incremental.py, you are using training set to perform incremental learning, but the saved prediction comes from validation set. ############################################################################### Then train M_c(t) to M_c(t+1), similar to the process above, repeatedly run step 2,3,4 only the "unknown_clss" in all these files should be modified with one less class.

[Jun-CEN]

Hi,

Thanks for your interest about our work. Our work can be divided into 2 tasks including open-set segmentation and incremental learning. Open-set segmentation is to detect unknown classes, while incremental learning is to learn unknown classes. MSP and MaxLogit are methods of open-set segmentation.

1. This step is to train a base model M_c0, and 4 classes {barrier, consuction_vehicle, traffic_cone, trailer} are unknown for the model.
2. This step is to evaluate the performance of closed-set mIoU and open-set segmentation. So it is correct that we use the validation set. We get saved prediction and uncertainty score of the validation set, and then we use our modified SemanticKITTI-api to evaluate the closed-set and open-set performance. So this step is to evaluate how well M_c0 can detect unknown classes.
3. After open-set segmentation in step 2, we begin incremental learning in step 3 to learn unknown classes. So we need to use the old model M_c0 to generate the pseudo labels of the training set. It seems like we are still using validation set in val_cylinder_asym_nusc_generate_increlabels.py, but the default corresponding config file is config/nuScenes_ood_generate_incre_labels.yaml_, where validation set is actually defined as the training set https://github.com/Jun-CEN/Open_world_3D_semantic_segmentation/blob/f7041d6b50b431a510e308d54f359917d7350e26/config/nuScenes_ood_generate_incre_labels.yaml#L51-L53
4. After we obtain the pseudo labels in step 3, now we can finetune the model M_c0 to M_c1.

Note that during incremental learning, for the first new class barrier, we use val_cylinder_asym_nusc_generate_incre_labels.py to generate pseudo labels, and for the last three new classes, we use val_cylinder_asym_nusc_generate_incre_labels_1.py to generate pseudo labels, as illustrated in the README.md.

[luoyuchenmlcv]

Thanks for your detailed explanation! But I am still confused with this one and need your guidance:

To train M_c(0) to M_c(1), with use the description from above, with new class barrier

To train M_c(1) to M_c(2), M_c(2) to M_c(3), M_c(3) to M_c(4), corresponding to your paper training scheme, should I loop the similar process 3 more times, using val_cylinder_asym_nusc_generate_incre_labels_1.py, like training M_c(0) to M_c(1)?

Or using val_cylinder_asym_nusc_generate_incre_labels_1.py one time ? So that M_c(1) to M_c(4) directly?

And in all these steps, "unknown_clss" should be reduced as well right ?

[Jun-CEN]

I learn the new class one by one. But of course you can learn three new classes in one time, but you have to modify the code.

What do you mean "unknown_clss" should be reduced?

[luoyuchenmlcv]

I see that from your code:

unknown_clss = [1,5,8,9] for unknown_cls in unknown_clss: point_label_tensor[point_label_tensor == unknown_cls] = 0

this part exist in all related files, unknown_clss = [1,5,8,9] should be reduced to unknown_clss = [5,8,9] after barrier class has been learned, and will be further reduced to unknown_clss = [8,9] and unknown_clss = [9] when more classes are learned, right?

[Jun-CEN]

No. The only thing needed to change is args.incremental_class in the corresponding file. You can check the logic of the code carefully.

[luoyuchenmlcv]

Ok, thank you for the clarification!