konabuta
commented
5 months ago The section 2.5 for Fine-Tuning. The data for fine-tuning is creatd by Llama2-13b-chat with RAG.
- FSDP using 8 H100 GPUs.
- Automatic mixed precision with BFloat16
- Adam optimizer
- Learning rate of 2e-5, and consine learning rate scheduler with a linear warmup comprising 4% ofthe total number of steps.
- FastChat training script with flash-attention monkeypatching
- Low Rank Adaption (LoRA)
RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture
Link: https://arxiv.org/abs/2401.08406
Start reading it. In my understang, PEFT is not for knowldge, but for format. I will check what kind of fine-tuning method is used in this paper.