search

ErikEnglesson / GJS

The official code for the NeurIPS 2021 paper Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels (https://arxiv.org/abs/2105.04522)
21 stars 1 forks source link

Transferability to semantic segmentation #4

Closed JaWeyl closed 9 months ago

JaWeyl commented 9 months ago

Thanks for the excellent paper - I enjoyed reading it and think it addresses an important task.

Do you have any experience about applying the proposed GJS loss to semantic segmentation?

From my experience, there are many noisy labels in the semantic annotations of images, e.g., at the contour of objects. Thus, intuitively, the proposed loss should also help for this task - I'd appreciate it if you can share any experience in case you tested it also for semantic segmentation.

Best,

JaWeyl

ErikEnglesson commented 9 months ago

Hi JaWeyl,

I am glad to hear you enjoyed our paper!

Unfortunately, I have almost no experience with semantic segmentation yet. However, I think semantic segmentation is an exciting application of robust noisy labels methods, as there are noisy labels essentially between any transition from one label to the other. In fact, we have just started a project on applying noisy label methods to the problem of land cover prediction from satellite images, which is a segmentation task. We are in very early stages though, and are still working on getting a good cross-entropy baseline. Hopefully, we can get a paper out of this soon, where we will share all the experiences we have learned.

If you allow me to speculate about challenges with applying GJS to semantic segmentation, I think the main challenge would be to find a good augmentation strategy. In the GJS paper, we showed that strong augmentation greatly improves the performance of GJS. Strong augmentations in segmentation tasks I think could be challenging, as for example, if the strong augmentations uses rotation, then one must make sure that the GJS loss encourages consistency between the correct pixel in the first augmented image and the second (M=2). I haven't thought much about this problem, but one of my colleagues have: Dense FixMatch: a simple semi-supervised learning method for pixel-wise prediction tasks. Another alternative is to choose strong augmentations that only change pixel values, such as colour transformations. An interesting example of such an augmentation strategy is PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures.

I wish I had more experience with semantic segmentation, so I could have help in a better way. Thanks again for the kind words about our work.

Best, Erik

JaWeyl commented 9 months ago

Thanks for your prompt response and intuitions about the GJS for semantic segmentation!

I agree that the GJS loss feels intuitive for semantic segmentation and that appropriate data augmentation can be challenging since they often depend on the dataset. I'll give this loss a try and share my experience. However, this may take a while since its rather a side project.

Thanks and cheers,

JaWeyl