search

HealthX-Lab / MedCLIP-SAM

Official implementation for MedCLIP-SAM (MICCAI 2024)
22 stars 1 forks source link

some issues bout grads #2

Closed sjlljs176 closed 1 month ago

sjlljs176 commented 1 month ago

I noticed that the grads_list was nan when using the BiomedCLIP. Is it normal? I feel weired.

This is the line. https://github.com/HealthX-Lab/MedCLIP-SAM/blob/38e4cadfd607e32d8144427fc9d6f5cc72f64c09/saliency_maps/pytorch_grad_cam/base_cam.py#L192C13-L193C75

TahaKoleilat commented 1 month ago

Hi @sjlljs176,

Can you please provide snapshots of the output you are getting? I am running the code on my machine and I am not getting any NaN values in the grads_list. This could be maybe due to the input. So, if you can also please provide sample inputs where you are getting NaN values, it will be easier to reproduce and pinpoint the problem.

Thank you.

sjlljs176 commented 1 month ago

the values in activations_and_grads.gradients hava the shape like (1,197,768) but the reshape_transfrom make it to be a list with nothing image

sjlljs176 commented 1 month ago

when debug in the get_cam_weights, the "importance" become a tensor with NaN values. I used the images of BUSI dataset as inputs. image

TahaKoleilat commented 1 month ago

The issue was with the gradients shape. I added the fix here.

Thank you for pointing this out @sjlljs176.

sjlljs176 commented 1 month ago

Thanks a lot for your time and assistance!

sjlljs176 commented 1 month ago

b.t.w, I found the value of loss became < 0 (negative number), when I fine-tuned the BiomedCLIP with MedPix dataset. Is it normal?

TahaKoleilat commented 1 month ago

It's totally normal. Typical contrastive losses will be bounded between 0 and 1, however when the positive and negative pairs are decoupled the loss will be unbounded and thus can become negative.

sjlljs176 commented 1 month ago

thanks a lot.