Clarification on datasets

[mdsmith-cim]

For both Track 1 and Track 2, my understanding was that training was strictly limited to the Cityscapes or one or more of [Cityscapes, BDD100k, Mapillary Vistas, India Driving Dataset, WildDash 2, GTA5 Dataset, SHIFT Dataset]. My understanding was that anything beyond this, such as COCO or various OOD training data was prohibited.

However, I noticed that some of the baselines seem to use the COCO dataset as OOD training. As such I have three questions:

1. Is COCO an allowed dataset for Track 2?
2. What OOD datasets are allowed to be trained on?
3. If OOD data is allowed, do they have to be existing datasets or can we create our own e.g. in the typical fashion of inserting objects into existing datasets?

Thanks!

[dreavjr]

Thank you for calling this our attention, @mdsmith-cim! The offending baseline would be RbA, right?

For this edition of the challenge, we would prefer to be stricter on the allowed datasets. Depending on the participants feedback we might be more flexible in the future. We will remove or adapt the problematic baselines.

[mdsmith-cim]

@dreavjr RbA with the COCO dataset, yes, which it looks like you have already removed. If they were to be allowed, I imagine similar to other OOD challenges e.g. SegmentMeIfYouCan there would need to be a separate track for training with OOD data.

On a somewhat related note, is ImageNet pretraining allowed? While I can't speak for every model or baseline, one of the models I'm looking at uses it for pretraining/transfer learning, as does Mask2Former which RbA uses. Overall, I'm not sure there are many segmentation methods that don't use it for pretraining.

[tuanhungvu]

@mdsmith-cim Indeed, we have removed the RbA baselines that use COCO objects for OOD augmentation from the leaderboard . Thank you again for bringing this to our attention. Depending on participant feedback, we will consider adding a new track in the future.

ImageNet pretraining is a common practice and is allowed in the BRAVO challenge

[dreavjr]

With the clarification above that no other datasets are acceptable except for ImageNet pretraining, I am considering this closed for now.

[mdsmith-cim]

@dreavjr I noticed that you removed the RbA baseline - might I ask why? The RbA paper/code presents two approaches, one trained on Cityscapes and the other which uses COCO for OOD supervision (see the models at https://github.com/NazirNayal8/RbA/blob/main/MODEL_ZOO.md) . Only the latter would violate the policy as far as I am aware.

[tuanhungvu]

Hello @mdsmith-cim, The model trained exclusively on Cityscapes still serves as the baseline, referred to as Mask2Former-SwinB in https://benchmarks.elsa-ai.eu/?ch=1&com=evaluation&view=method_info&task=1&m=719

We will update the description and repository to make it clearer. Regards,

[mdsmith-cim]

Ah I see thanks - for some reason I overlooked that.