search

huggingface / peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
https://huggingface.co/docs/peft
Apache License 2.0
16.35k stars 1.61k forks source link

LoRA support for image classification and segmentation #2052

Closed namrahrehman closed 3 weeks ago

namrahrehman commented 2 months ago

I had a question regarding LoRA support for image classification and segmentation. I understand that LoRA support is available for both as specified in the following tutorials: https://github.com/huggingface/peft/blob/main/examples/semantic_segmentation/semantic_segmentation_peft_lora.ipynb https://huggingface.co/docs/peft/main/en/task_guides/image_classification_lora

but is LoHa, LoKr, AdaLoRA, and QLoRA support available for image classification and segmentation? Or can we only use the traditional LoRA?

I could not find a definite answer to my question anywhere in the official documentation.

BenjaminBossan commented 2 months ago

AdaLoRA does not work. LoHa and LoKr have support for Conv2d layers, as such they should work. However, they don't support quantization. Therefore, LoRA is most feature rich when it comes to support for image models.

I agree that this information is not easily figured out, I'd have to think a bit about how to best document this. From a user's perspective, the easiest way is probably to just try it out.

namrahrehman commented 2 months ago

Okay and by "image models" do you mean ViTs (Dino, Swin, DeiT, etc) as well? As my experiments involve the use of ViT-based backbones.

BenjaminBossan commented 2 months ago

Ah yes, sorry, vision transformers should generally work, as they use linear layers. All methods, except for prompt-tuning methods, implement linear layers. So even AdaLoRA should work there.

namrahrehman commented 2 months ago

Thanks for your response @BenjaminBossan , I will get back to you with an implementation and then we can discuss it further.

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.