Closed makemecker closed 3 months ago
I also encounter the same issue!
Can you try casting the tensor to float()
in utils_attn.py
line 486?
ax2 = seaborn.heatmap([attn_image_patch.float()]
Hi, I have made some fixes. Also, the correct steps to plot attention should be:
Hi, I have made some fixes. Also, the correct steps to plot attention should be:
- Upload an image and input text for the model.
- Go to the "Attentions" section.
- Click "Plot Attention" in top section to load the image and attentions.
- Click "Reset Patch Selector" in bottom section to initialize the patch selector.
- Select desired image patches
- Click "Plot attention matrix".
Hi,
Thank you for the quick response and the fixes!
Following your updated steps, I was able to successfully plot the attention matrix without any errors. Your instructions were very helpful)
First, I want to express my gratitude for this fantastic tool that provides a powerful way to visualize the workings of multimodal models. It’s incredibly helpful for understanding and analyzing complex model behaviors. Thank you for your hard work on LVLM-Interpret!
I encountered an issue while trying to visualize the attention matrix using the LVLM-Interpret tool. The problem occurs when attempting to plot the attention matrix after selecting image patches using the patch selector in the "Attentions" section of the tool and clicking the "Plot attention matrix" button. An error message appears in the tool interface:
And the following error traceback is logged in the console:
The error seems to be caused by an unsupported ScalarType BFloat16 when attempting to convert a PyTorch tensor to a NumPy array. My setup indicates that BFloat16 should be supported, as verified by running torch.cuda.is_bf16_supported() which returns True.
Steps to Reproduce
python app.py --model_name_or_path Intel/llava-gemma-2b --share
Any insights or fixes would be greatly appreciated. Thank you!