Reintroducing the crosswalk class back into MSeg

[ionicaML]

Hello, I want to re-add the crosswalk class to the MSeg dataset. I have seen that mapillary vistas dataset contains the crosswalk class, and that it is relabeled to the road class. Also, I know that other datasets (such as cityscapes) contains crosswalks but they are not labeled and are considered plain road. My question is: How would you add the crosswalk class back into the dataset? . I'm thinking of 3 options, and I would appreciate your insight:

1. Use mseg-turk to label each dataset which does not contain crosswalks. This might be the best and most straight-forward option, but this can prove labor-intensive and costly
2. For the datasets that do not contain crosswalks, ignore the road class. For example, ignore all possible mislabels of crosswalks in the cityscapes dataset. I expect this to lower the confidence for the road class, as there will be considerably less data.
3. Simply add the crosswalk class and revert the relabel action applied on mapillary. I woudn't try this, because it might be problematic for the model to learn crosswalks only sometimes (in case of mapillary frames) while ignoring them alltogether in other cases(e.g. cityscapes frames) - which may also create confusion between the road class and crosswalk class.

Thanks!

[johnwlambert]

Hi @ionicaML, thanks for your interest in our work. There are a few different ways to go about it:

Regarding (1), mseg-mturk is designed for re-assigning categories to instances, not for re-drawing polygon boundaries. So you would need to use a different polygon-drawing tool (Amazon MTurk has example code for this, and there's an active area of research about building tools for human-in-the-loop segmentation annotation).

Regarding (2), this is feasible. You would set "road" in 5 datasets (ADE20K, COCO, IDD, BDD, and Cityscapes) to "unlabeled", and then add "crosswalk" to the universal taxonomy as a new class, and route Mapillary's crosswalk marking to it, instead of to "road", as we had it previously. This would hurt generalization for the "road" class, but if you need crosswalk predictions urgently enough, maybe the tradeoff is worth it for you.

Regarding (3), this is another reasonable and possible approach. Allowing for this inconsistency clashes with the principles/decision tree we designed in Figure 3 of our paper, and we haven't evaluated the effect on lane marking or crosswalk labels. But it's feasible.

A fourth option is to use the equivalent of a logical OR gate in your loss function, like SwiftNet did, summing over multiple logits using multi-label ground truth (See Equation 4 of their paper).

In short, this is still an open research area and we welcome your feedback and seeing what you discover.

[jdavidd]

Hi, @johnwlambert I want to make something similar with the dataset, but also add curb and curb cut. So for this until now i figured out, that I should modify the tsv files of dataset you mentioned above. But I dont know if this is complete, so I have two questions: 1) When you said add 'crosswalk' to universal taxonomy you refer to add a new line in MSeg_master.tsv for this class? I should do this also for, curb (curb, curb cut)? are there other files, i should change?

2) I want to keep only 64 classes, so this means I only remove them (their lines) from MSeg_master.tsv and add to final line to unlabeled, and also modify the tsv file for every dataset (ex. ade20k-151_to_ade20k-150.tsv and ade20k-151_to_ade20k-150-relabeled.tsv).

Waiting for you response.