Error while plotting attention matrix due to unsupported BFloat16 ScalarType

[makemecker]

First, I want to express my gratitude for this fantastic tool that provides a powerful way to visualize the workings of multimodal models. It’s incredibly helpful for understanding and analyzing complex model behaviors. Thank you for your hard work on LVLM-Interpret!

I encountered an issue while trying to visualize the attention matrix using the LVLM-Interpret tool. The problem occurs when attempting to plot the attention matrix after selecting image patches using the patch selector in the "Attentions" section of the tool and clicking the "Plot attention matrix" button. An error message appears in the tool interface:

And the following error traceback is logged in the console:

INFO:utils_attn:Loaded attention from /tmp/tmpqlas_jbu_attn.pt Traceback (most recent call last): File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/gradio/queueing.py", line 532, in process_events response = await route_utils.call_process_api( File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/gradio/route_utils.py", line 276, in call_process_api output = await app.get_blocks().process_api( File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/gradio/blocks.py", line 1928, in process_api result = await self.call_function( File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/gradio/blocks.py", line 1514, in call_function prediction = await anyio.to_thread.run_sync( File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread return await future File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run result = context.run(func, args) File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/gradio/utils.py", line 832, in wrapper response = f(args, kwargs) File "/home/tyugunov/modules/lvlm-interpret/utils_attn.py", line 486, in plot_text_to_image_analysis ax2 = seaborn.heatmap([attn_image_patch], File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/seaborn/matrix.py", line 446, in heatmap plotter = _HeatMapper(data, vmin, vmax, cmap, center, robust, annot, fmt, File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/seaborn/matrix.py", line 109, in init plot_data = np.asarray(data) File "/home/tyugunov/venvs/lvlm_interpret/lib/python3.10/site-packages/torch/_tensor.py", line 1087, in array return self.numpy() ⚠️TypeError: Got unsupported ScalarType BFloat16**⚠️

The error seems to be caused by an unsupported ScalarType BFloat16 when attempting to convert a PyTorch tensor to a NumPy array. My setup indicates that BFloat16 should be supported, as verified by running torch.cuda.is_bf16_supported() which returns True.

Steps to Reproduce

1. Run LVLM-Interpret with the following command:

python app.py --model_name_or_path Intel/llava-gemma-2b --share

2. Upload an image and input text for the model.
3. Go to the "Attentions" section.
4. Upload an image in window "Preprocessed Image".
5. Click "Reset patch selector".
6. Select desired image patches.
7. Click "Plot attention matrix".
8. Observe the error message in the interface and check the console for the traceback.

Any insights or fixes would be greatly appreciated. Thank you!

[shijian2001]

I also encounter the same issue!

[shaoyent-IL]

Can you try casting the tensor to float() in utils_attn.py line 486? ax2 = seaborn.heatmap([attn_image_patch.float()]

[shaoyent-IL]

Hi, I have made some fixes. Also, the correct steps to plot attention should be:

1. Upload an image and input text for the model.
2. Go to the "Attentions" section.
3. Click "Plot Attention" in top section to load the image and attentions.
4. Click "Reset Patch Selector" in bottom section to initialize the patch selector.
5. Select desired image patches
6. Click "Plot attention matrix".

[makemecker]

Hi, I have made some fixes. Also, the correct steps to plot attention should be:

1. Upload an image and input text for the model.
2. Go to the "Attentions" section.
3. Click "Plot Attention" in top section to load the image and attentions.
4. Click "Reset Patch Selector" in bottom section to initialize the patch selector.
5. Select desired image patches
6. Click "Plot attention matrix".

Hi,

Thank you for the quick response and the fixes!

Following your updated steps, I was able to successfully plot the attention matrix without any errors. Your instructions were very helpful)