[FR] Visualization of extracted feature maps or attention maps

voxel51 / fiftyone

Refine high-quality datasets and visual AI models

https://fiftyone.ai

Apache License 2.0

8.84k stars 557 forks source link

[FR] Visualization of extracted feature maps or attention maps #1680

Open oneseer opened 2 years ago

oneseer commented 2 years ago

Hi, I noticed that fiftyone currently does not support visualization of extracted feature maps and attention for cv models, such as Vit, MVit, etc. Do you currently have any plans for this feature? I hope to be able to participate and contribute together to enrich the visualization and attribution capabilities of fiftyone. here is an example about mvit:

original image in video
feature map
attention map

brimoor commented 2 years ago

Hi @oneseer 👋

FiftyOne does have a Heatmap label type that you can use to store arbitrary numeric data and do things like configure colormaps and transparencies when visualizing these fields in the App.

You would need to store each map in a separate Heatmap field, say, xxx_layer0, xxx_layer1, ... but then you could toggle each map on/off as desired via the App's controls.