search

alexriggio / BERT-LoRA-TensorRT

This repository contains a custom implementation of the BERT model, fine-tuned for specific tasks, along with an implementation of Low Rank Approximation (LoRA). The models are optimized for high performance using NVIDIA's TensorRT.
Apache License 2.0
56 stars 7 forks source link

THANK YOU; Q-LORA for vision language models #3

Open Abhiram-kandiyana opened 8 months ago

Abhiram-kandiyana commented 8 months ago

Thank you for documenting everything you learned. It is very helpful. I have been trying to find a pre-coded Q-LORA for BiomedCLIP but I couldn't so I have to do it on my own. BioMedCLIP uses a BERT model as text encoder and ViT as vision encode with contrastive learning. Do you think we can apply Q-Lora to BERT and ViT separately and then join them?

alexriggio commented 8 months ago

Sounds reasonable. I am not familiar with the specific architecture but would think LoRA could target any linear layer.