Caformer finetuning. Adapter/LoRa?

[kacwin]

Hey, great job with this repo. Caformer with 100M parameters is really powerful, though I am struggling with the finetuning due to hardware limitations. Did you already make experiments with something like adapter finetuning or LoRA? At first glance, the code looks like one would need to rewrite a lot for this.

[yuweihao]

Hi @kacwin ,

Many thanks for your attention. I did not conduct experiments with adapter finetuning or LoRA. For hardware limitations, since all models in the paper do not use Batch Norm, we can set --grad-accum-steps.

[kacwin]

Thanks for the info, using gradient accumulation would be kind of a last resort; there simply is too much data. I will try some LoRA experiments in the near future, can give an update on that if you want.

[yuweihao]

Many thanks!

[kacwin]

Hello, we did some experiments with LoRa finetuning.

• We started with Caformer_b36_384 and did linear probing (freezing the network aside from the mlp head) on a classification task with domain shift ---> accuracy ~30%
• We repeated the experiment with a full finetuning ---> accuracy ~65%
• Lastly, we implemented parallel LoRa layers for all linear layers in Caformer. We then froze the original linear weights and only trained the LoRa layers as well as the Conv2d layers (in the downsampling stages and in the SepConv blocks). This left us with 10M trainable parameters, compared to 100M parameters at full finetuning ---> accuracy 61%

So with a fraction of trainable parameters, we achieved similar results. However, training time/GPU space sadly did not decrease that much (maybe factor 0.66). I think there is some potential here :)

[yuweihao]

Thanks for sharing valuable experiment results.