RyanWangZf / MedCLIP

EMNLP'22 | MedCLIP: Contrastive Learning from Unpaired Medical Images and Texts
394 stars 41 forks source link

Cannot run example code in colab #16

Closed erichan1 closed 1 year ago

erichan1 commented 1 year ago

Would love help fixing this - I can't run the example CLIP inference code on the example image on a clean google colab.

I opened a brand new colab, uploaded the example image into the colab filesystem, and ran the following code:

I get the following error:

AttributeError                            Traceback (most recent call last)
[<ipython-input-2-d97046b8898c>](https://localhost:8080/#) in <module>
      6 processor = MedCLIPProcessor()
      7 image = Image.open('./view1_frontal.jpg')
----> 8 inputs = processor(
      9     text=["lungs remain severely hyperinflated with upper lobe emphysema", 
     10         "opacity left costophrenic angle is new since prior exam ___ represent some loculated fluid cavitation unlikely"], 

2 frames
[/usr/local/lib/python3.8/dist-packages/medclip/dataset.py](https://localhost:8080/#) in <listcomp>(.0)
    103         # transformations (convert rgb + resizing + center cropping + normalization)
    104         if self.do_convert_rgb:
--> 105             images = [self.convert_rgb(image) for image in images]
    106 
    107         if self.do_pad_square:

AttributeError: 'MedCLIPFeatureExtractor' object has no attribute 'convert_rgb'

I solved this and a subsequent padding error by turning off rgb conversion and padding:

processor.feature_extractor.do_convert_rgb = False
processor.feature_extractor.do_pad_square = False

I convert the grayscale example image into RGB manually using example_image.convert('RGB'), converting it to numpy, and moving the channels dim from last dim to first dim because model expects channels-first.

But then we get a new issue where all the pixel_values become inf and nan after the processor is called on the inputs.

RyanWangZf commented 1 year ago

Hi, thanks for posting this problem. I checked and find it is the problem due to the version of transformers. Please try to install transformers<=4.24.0 then it should fix the bug. You can also refer to the example I create: https://colab.research.google.com/drive/1jwaHrHmF8RZ1m31jtzgWeccB55g1z0Z1?usp=sharing