BF16 PEFT training references

[heeju-kim2]

Reference code

• Llama-recipes code https://github.com/meta-llama/llama-recipes/tree/b7fd81c71239c67345d897c0eb6529eba076e8b8

[heeju-kim2]

Reference Paper

• Revisiting BFloat16 Training (2021, sambanova)

  https://arxiv.org/pdf/2010.06192

• Doubling Neural Network Finetuning Efficiency with 16-bit Precision Techniques

  https://lightning.ai/blog/doubling-neural-network-finetuning-efficiency-with-16-bit-precision-techniques/

  https://github.com/rasbt/ViT-finetuning-scripts/tree/main

• Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments

  https://lightning.ai/pages/community/lora-insights/

• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training (https://arxiv.org/abs/1909.06842)

[heeju-kim2]

Evaluation metrics

• loss per iteration
• efficiency and accuracy tradeoff
• memory cost vs accuracy
• training throughput speedup vs accuracy

[dylee-furiosa]

llama2 3runs wnadb: https://wandb.ai/debiasing/llama_recipes?nw=nwuserbig_whale

mixed : use_fp16 = False, mixed_precision = True fp16 : use_fp16 = True, mixed_precision = False fp32 : use_fp16 = False, mixed_precision = False