CUDA Memory is not enough

AkariAsai / self-rag

This includes the original implementation of SELF-RAG: Learning to Retrieve, Generate and Critique through self-reflection by Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.

https://selfrag.github.io/

MIT License

1.76k stars 162 forks source link

CUDA Memory is not enough #84

Open zhongruizhe123 opened 3 months ago

zhongruizhe123 commented 3 months ago

Ask the Critic training how much video memory is required at least, I used two 3090ti 24G, but to tip that video memory is not enough. Is there a way to adjust some parameters to make my program work

fate-ubw commented 1 month ago

24G of RAM can't fully weight finetune 7B models, which usually requires 4*80G GPUs. If you only have two 3090ti 24G I recommend you to use LoRA or QLORA, but this will lose a little bit of performance, if you don't know how to configure LORA and QLORA you can refer to the LoRA finetune script I configured https://github.com/fate-ubw/RAGLAB/blob/main/run/rag_train/script_finetune-llama3-8B-baseline-Lora.sh All you need to do is switch the training data and output model name