Example code for supervised finetuning using epochraft + HF FSDP

[fujiki-1emon]

Description

• I mainly personally interested in scalable codebase both for (continued) pre-training and supervised fine-tuning using the popular HF libraries.
• Out of (pure) technical curiosity, scalable approach to SFT like this one Fine-tuning Llama 2 70B using PyTorch FSDP seems interesting and should be useful for continued pretraining and/or supervised finetuning of multi-B models to some extent.
• So, having such scalable-to-multi-B codebase based on HF and FSDP plus using epochraft should be helpful.

Related

• Fine-tuning Llama 2 70B using PyTorch FSDP

[fujiki-1emon]

@iwiwi Thanks for taking care of this issue! I wonder if there's any room for me to customize things around dataset to allow SFT training? If so, I'm willing to help on it. Thank you.