ScatterMoE to support LoRA Adapters - Githubissues

foundation-model-stack / fms-acceleration

🚀 Collection of libraries used with fms-hf-tuning to accelerate fine-tuning and training of large models.

Apache License 2.0

6 stars 12 forks source link

ScatterMoE to support LoRA Adapters #103

Open fabianlim opened 2 weeks ago

fabianlim commented 2 weeks ago

This issue to be done after #99

the kernel-hyperdrive has support for LoRA adapters, but need to install it properly into the plugin

perhaps have the prepare happen during augmentation (still use the model_loader). in augmentation we intercept the peft_config
ensure that checkpoint saving works correctly with LoRA. Maybe its easier to just modify the fsdp_save and load functions to support adapters.

In addition we defer the following things

handle the mixed precision here, as we can now get the --bf16 or --fp16 flags easily.