-
Do we have a general sense on this? Has LoRA/QLoRA fine tuning been attempted on this, and if so, any guidance?
-
Currently, for every llama SKU, we have 6 configs:
- LoRA single device
- LoRA distributed
- QLoRA single device
- QLoRA distributed (after FSDP2)
- Full distributed
- Full single device
We d…
-
I see that [PEFT brought in](https://github.com/huggingface/peft/releases/tag/v0.10.0) QLoRA with FSDP support in their latest release.
Any plans to incorporate this into litgpt?
-
Evidence:
We finetune without Unsloth in Qlora with rank 32, targets all linear layers AND embed/lm_head (smaller 10x lr, same padđing right key as Unsloth) in a total of **1,134,559,232 trainable …
-
When working on my customized LoRAFinetuneRecipeSingleDevice recipe and upgrading from `torchtune` version 0.1.1 to 0.2.1 and `torchao` 0.1 to 0.3.1, I noticed that model loading times went up dramati…
-
### System Info
---
**Setup Summary for LoRAX Benchmarking with Llama-2 Model:**
- **Hardware**: A100 40 GB (a2-highgpu-2g) on Google Kubernetes Engine (GKE)
- **Image**: ghcr.io/predibase…
-
```
oading checkpoint shards: 0%| | 0/2 [00:00
-
Hello,
Does anyone have any advice on how to fine-tune the model using lora technique?
Thank you
-
### Question
Nice work!
I just wonder when could you finish LoRA/QLoRA training? It may help a lot to finetune.
-
Fully supported! Scroll down on our latest Mistral notebook: https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing
For 16bit merging:
```
model.save_pretrained_mer…