search

zhang-haojie / wesam

[CVPR 2024] Code for "Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation"
MIT License
104 stars 7 forks source link

About Contrastive Loss in your method #7

Open AprilT0621 opened 2 months ago

AprilT0621 commented 2 months ago

Hello! I am currently attempting to replicate the work presented in your paper and I am very grateful for the code you have provided. However, I have encountered an issue regarding the contrastive loss.

During my training process, the contrastive loss remains consistently at 0. After discovering this problem, I checked the inputs in the contrastive loss function (soft_image_embeds[i], anchor_image_embeds[i], soft_res_masks[i], anchor_res_masks[i]) and found that the values ​​of soft_image_embeds[i] and anchor_image_embeds[i] are almost the same, the same goes for soft_res_masks[i] and anchor_res_masks[i]. loss_contra += contra_loss(soft_image_embeds[i], anchor_image_embeds[i], soft_res_masks[i].clone().detach(), anchor_res_masks[i].clone().detach()) Here is my tensorboard visualization of the contrastive loss using the ISIC dataset and 'box' prompt, the model is ViT-b: 20240417170340

I was wondering if you could provide some insights into possible reasons for this occurrence or directions I could explore for troubleshooting? Perhaps I may have encountered some common pitfalls, but I am currently unable to ascertain them.

I look forward to your guidance and suggestions.

zhang-haojie commented 2 months ago

In cases with only one target in an image, the absence of negative samples in the contrast loss construction results in a constant 0 loss. We have used a method to treat the background as a special instance, and the code will be updated soon.

lfxx commented 2 months ago

In cases with only one target in an image, the absence of negative samples in the contrast loss construction results in a constant 0 loss. We have used a method to treat the background as a special instance, and the code will be updated soon.

looking forward to this update since i can not get 80.01 miou on ISIC,i just get 76.1.