search

hendrycks / anomaly-seg

The Combined Anomalous Object Segmentation (CAOS) Benchmark
MIT License
154 stars 20 forks source link

Some questions about the CAOS benchmark #16

Closed zhouhuan-hust closed 3 years ago

zhouhuan-hust commented 3 years ago

Hello, thanks for your hard work! I want to do some experiments with the StreetHazards and BDD-anomaly dataset，but I have some confusion during the experiment. In your《Scaling Out-of-Distribution Detection for Real-World Settings》paper, for StreetHazards dataset, it is stated that there are 12 classes used for training, but the dataset label shows that there are a total of 14 classes(label:1-14). Except for the anomaly class, the training set includes another 13 classes. Is there any class that does not participate in training and testing during the experiment? For BDD-anomaly dataset, in your paper, it is stated that there are 18 original classes, but the dataset label shows that there are a total of 20 classes (label:0-18,255). Is there any class that does not participate in training and testing during the experiment? Could you give me some help so that I can better understand the benchmark? Thank you very much and looking forward to your reply.

xksteven commented 3 years ago

Hello huanhuan, thanks for your question. I believe this may just be an issue with clarity in the paper. There are 12 distinct classes, 1 anomalous class, and then an "other" or "background" class for the StreetHazards dataset. We utilize the 12 classes + background class during training. Depending on who you talk to (in the research community) the background class is sometimes not considered itself as a real class because it is a conglomeration of other classes. The only class not used during training is the anomalous class. In the paper we also experimented with excluding the background class during training but achieved worse results.

As for your second question concerning the BDD-anomaly dataset we have the code here that list the classes and on these lines here you can observe that we exclude the classes train, motorcycle and bicycle from the dataset. For convenience we also include all of the preprocessed labels here.

I believe this answers all of your questions about the dataset. Feel free to reopen and add any additional questions or details if I missed something :)

zhouhuan-hust commented 3 years ago

Thank you for your answer, but I still have some points to confirm. 1.For StreetHazards dataset, there are a total of 14 classes(0:unlabeled,1:building,2:fence,3:other,...13:anomaly).The label in the label file is from 1 to 14, minus 1 corresponds to 0-13 here. Unlabeled Corresponding background.Use 0-12 for training(a total of 13 categories), including 0(0:unlabeled) and 3(3:other), and use 0-13 for testing(a total of 14 categories), right? 2.For BDD-anomaly dataset, there are a total of 20 classes (0:road,1:sidewalk,...18:bicycle,255:other). Use 0-15 and 255(other) for training(a total of 17 categories), only does not contain 16-18, and use 0-18 and 255 for testing(a total of 20 categories), right? Thank you very much and looking forward to your reply.