search

mseg-dataset / mseg-api

An Official Repo of CVPR '20 "MSeg: A Composite Dataset for Multi-Domain Segmentation"
Other
249 stars 45 forks source link

Test dataset annotation #28

Closed lxtGH closed 3 years ago

lxtGH commented 3 years ago

Hi! @johnwlambert
Since the train datasets have been relabeled. image

Why the test sets are not relabed? How to test the these test test?

lxtGH commented 3 years ago

Did you just transfer these gt mask into unified taxonomy of 194 classes using NaiveTaxonomyConverter?

johnwlambert commented 3 years ago

Hi @lxtGH, please refer to section 3.2 of our paper: "We re-annotate only the datasets used for learning, leaving the evaluation datasets intact."

We did this for fair comparison with other results on these established benchmarks. Indeed, we were the runner-up at the 2020 Robust Vision Challenge that used several of these benchmarks.

Regarding how to project results in the universal taxonomy into a test dataset's taxonomy, please refer to the section entitled "Using the MSeg taxonomy on a held-out dataset." of our paper on page 6. We discuss this in detail.