the number of dummy classfiers for nuscenes

[luoyuchenmlcv]

Hi, I see there are 4 unknown classes, {barrier, traffic cone, trailer, construction vehicle}, but the default number of dummy classifiers=3 in your code. If I understand correctly, every incremental class should occupy one dummy classifier once introduced, hence it should be 4 dummy classifiers, could you explain why it is 3 ?

[Jun-CEN]

Hi,

For SemanticKITTI dataset, the number of dummy classifiers should be 3, as we find it has the best performance on the open-set task. For nuScenes dataset, we set the number of dummy classifier as 3 at first for the open-set task. Then we set the dummy classifier as 5 during incremental learning, as the number of dummy classifiers should be larger than unknown classes. Basically, you can set the number of dummy classifiers as 5 for both the open-set and incremental learning task.

[luoyuchenmlcv]

Thanks for your clarification!

[luoyuchenmlcv]

By the way, it seems that the evaluation code only test the mIOUclose, The code for testing AUROC and AUPR for open-set recognition ability is not provided. The most important thing to try to reproduce your reported results is the lambda_th, to classify if it is a novel class or not. How do you set the value? I guess it comes from maximizing classification accuracy between the synthesized and normal point during training time?

[Jun-CEN]

AUROC and AUPR can be evaluated through semantickitti_api and nuscenes_api, which are mentioned in the first 3 lines of README.md. You should store the uncertainty score files and closed-set evaluation files and then use api to get both closed-set and open-set performance.

AUORC and AUPR are threshold independent, so we do not need lambda_th to evaluate the open-set performance. You can check the definition of these two metrics.

[luoyuchenmlcv]

Thanks again! I will carefully understand this api.