search

opanhw / MM_Neurons

Code for "Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers" (Findings of ACL 2024)
MIT License
4 stars 0 forks source link

No definition about variable 'activations' used for getting heatmap and mask #1

Closed msra-jqxu closed 2 weeks ago

msra-jqxu commented 2 weeks ago

Hi, I noticed that the variable 'activations' doesn't seem to be defined here. Could you update the code? Thanks!

opanhw commented 2 weeks ago

Thank you for raising this issue. I have updated the trainer.py file. Please check if it works now.

msra-jqxu commented 2 weeks ago

Thanks! It works for me.

What's more, I'm curious why there is an img_tensor in addition to image_tensor here? In fact, the key of self.processor.image_processor.preprocess(image, return_tensors='pt') only has 'pixel_values' but no 'pixel'.

If I use img_tensor = image_tensor, then an error will be reported when saving ‘image.png’ because the dimension is (3, 224, 224). After I modify img_tensor to (224, 224, 3), the saved image below does not seem correct. Do I need to normalize the img_tensor to 0-255? image

opanhw commented 2 weeks ago

In fact, 'img_tensor' and 'image_tensor' are not the same. The pixel key is manually added, to get image tensors before being rescaled and normalized. Please replace transformers/models/blip/image_processing_blip.py in your own python libary with src/image_processing_blip.py, a newly uploaded file.

msra-jqxu commented 2 weeks ago

Hi, @opanhw, thanks for your efforts! Now the code works fine.

However, I hope the code could be better. For example, self.args.preparation_path and self.cal_activations are not defined in src/trainer.py and open_source_model/LLaVA/llava_llama.py respectively. And some python packages are not listed in requirements.txt. I spent some effort to fix them while running the code.

Thanks again！