Modify token ratio loss

[King4819]

Hi, in your work you utilize dynamic pruning ratio loss (averaged across all tokens and across images in a batch, resulting in more tokens allocated for complex images and fewer tokens for simpler images), which implements the below equation:

I would like to ask that how to modify the token ratio loss to fixed (does not average token usage across images within a batch, thereby penalizing each image towards the same keeping ratio) as most pruning methods (e.g., DynamicViT)

Thanks a lot. This is such an awesome work !!!

[kaikai23]

Hi, it is defined here: https://github.com/uzh-rpg/svit/blob/master/mmdet_custom/models/backbones/adapter_modules.py#L436-L441

[King4819]

@kaikai23 Thanks for your reply. If I change the arg self.ratio_per_sample to False (default True), it will perform fixed token ratio loss (does not average token usage across images within a batch, thereby penalizing each image towards the same keeping ratio) ?

[kaikai23]

self.ratio_per_sample=False corresponds to dynamic ratio loss, and Ture correpsondes to fixed ratio loss. I think we used dynamic ratio loss by default? https://github.com/uzh-rpg/svit/blob/d2472c13e667f4ef4c098898faec2e6cbe6c8e01/mmdet_custom/models/backbones/base/selective_vit.py#L100

[King4819]

Oh I get it. Thanks for your reply