where is ckpt_path = "focalnet_base_lrf.pth" and attention or focus scores

[WenY2020]

Hello, Jianwei:

I am trying to use focalnet to get some attention/focus scores for my images, I see it perform well here in the modulation map of the below images, and this is exactly what I need, I want to calculate each pixel's attention/focus score, I see in your image, the yellow is with highest attention scores (not sure attention score is the right name, basically I want to see which pixels get more attention from people's eyes). And I have two questions:

1. I try to run the code in visualizaiton.ipynb but cannot find "focalnet_base_lrf.pth", where to download it please?
2. I wonder can you give a hint, from which code part, I can get attention scores which shows in the modulation map.

Thank you very much for your help in advance!

Best wishes, Wen

[WenY2020]

I think I get the answer to my first question: "focalnet_base_lrf.pth" is actually one model of ImageNet-1K Pretrained FocalNet-B. Not sure the second one...

[jwyang]

Hi, @WenY2020 ,

thanks for your interest in our work. Regarding your two questions:

1. You can download the 'focalnet_base_lrf.pth' from this link: https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_base_lrf.pth. actually, you can find all the downloadable ckpts from here: https://github.com/microsoft/FocalNet#imagenet-1k-pretrained

This checkpoint is hierarchical which is slightly worse for visualization. I recommend you can also try to use the isotropic focalnets.

2. For visualize the modulation map, you can simply refer to this line of code: https://github.com/microsoft/FocalNet/blob/6de0705ffd4cdb26217c680461e0390a8d5f5c41/classification/focalnet.py#L96

It stores the modulator for each focal modulation block so that you can visualize the magnitude of the modulation maps for all focal modulation layers.

thanks, Jianwei

[jwyang]

And if you just want to visualize your images, you can directly use this hugging face demo!

https://huggingface.co/spaces/jw2yang/focalnet-modulators

[WenY2020]

thank you very much, Jianwei, those are very helpful, I wll look into here: x_out = q*self.modulator, Great, I need not only the visualization, mainly I need the attention numbers/scores, will need to use it as a feature to feed other models.

[jwyang]

Awesome, let me know if you have any further questions.

[WenY2020]

thanks you, will do:).