-
When I tried
```
!python qlora.py –learning_rate 0.0001 --model_name_or_path EleutherAI/gpt-neox-20b --trust_remote_code
```
in colab, i got following errors
```
2023-06-03 13:54:17.113623: W t…
-
在用
torchrun --nproc_per_node=4 train.py --train_args_file train_args/sft/qlora/qwen2-7b-sft-qlora.json
训练qwen2+qlora+unsloth时(use_unsloth=true)出现错误:
ValueError: You can't train a model that has bee…
-
Hi the team, great work!
QDoRA seems to be better than QLoRA, refer to [Efficient finetuning of Llama 3 with FSDP QDoRA](https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html)
I wonder w…
-
# Understanding LoRA and QLoRA - The Powerhouses of Efficient Finetuning in Large Language Models - Musings of Murali
Delving into the math behind LoRA and QLoRA
[http://gitlostmurali.com/machine-le…
-
### System Info
```
bitsandbytes==0.43.1
sentencepiece==0.1.97
huggingface_hub==0.23.2
accelerate==0.30.1
tokenizers==0.19.1
transformers==4.41.1
trl==0.8.6
peft==0.11.1
datasets==2.14.6
``…
-
![image](https://github.com/gauss5930/AlpaGasus2-QLoRA/assets/76432120/7f40e304-e7db-4f47-8ef4-700b8a86eaac)
I found these two models on OpenAI's LLM, and they exhibit significant differences in perf…
MDK-L updated
6 months ago
-
Do we have a general sense on this? Has LoRA/QLoRA fine tuning been attempted on this, and if so, any guidance?
-
Hello,
I was trying to see if the Opacus library can be used with the Trainer module of huggingface. I see a code snippet in the readme that says to do the callback ```dp_transformers.PrivacyEngine…
-
### Please check that this issue hasn't been reported before.
- [X] I searched previous [Bug Reports](https://github.com/OpenAccess-AI-Collective/axolotl/labels/bug) didn't find any similar reports.
…
-
### System Info
- `transformers` version: 4.41.2
- Platform: Linux-5.15.0-1044-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.0
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.2…