search

daniel-furman / sft-demos

Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.
https://huggingface.co/dfurman
Apache License 2.0
64 stars 8 forks source link

Plans on enabling multi-GPU training for OpenOrca finetuning? #1

Closed matthiasgeihs closed 1 year ago

matthiasgeihs commented 1 year ago

Hey, found this repository via your hugging face model dfurman/mpt-7b-instruct-openorca. This looks very useful!

I am currently mostly working with 2xA100 40GB. Are there any plans to enhance the scripts to make finetuning work on multiple GPUs?

daniel-furman commented 1 year ago

Yes - thanks for the question! I am planning on adapting the multi-GPU tutorial from Hugging Face that uses the FDSP method for multi-GPU training. Planning on doing sft with mpt-30b and perhaps falcon-40b with this multi-gpu tutorial. Don't have a timeline on when this will be ready though, could be a week or a month depending on my free time.

https://github.com/daniel-furman/sft-demos/blob/main/sft/multi_gpus/fsdp/hf_tutorials/fsdp_with_peak_mem_tracking.py

^ you could try to edit the above yourself - if you do, lmk!

matthiasgeihs commented 1 year ago

Thx for the response. I'll let you know if I try and how things are going.