search

loubnabnl / santacoder-finetuning

Fine-tune SantaCoder for Code/Text Generation.
Apache License 2.0
184 stars 23 forks source link

Ways to reproduce this approach #24

Open Yuyz0112 opened 11 months ago

Yuyz0112 commented 11 months ago

Hi @loubnabnl, thanks for this great repo.

I've seen a blog from the VMware OCTO, which described their works on fine-tuning star-coder, but modified the code provided by the [SantaCoder](https://github.com/loubnabnl/santacoder-finetuning) git repository for fine-tuning as it is focused on the code generation task..

There are some more details like:

  1. Accelerate and DeepSpeed are used to improve fine-tuning performance.
  2. Fine-tuning generates a small PEFT model.

I think this is not the best place to discuss their approach, but since you are the expert on fine-tuning santacoder/star-coder, are there any hints we can reproduce the approach in the blog on top of the current open-source code? I also checked the star-coder fine-tuning repo, but it looks like it suggests using instruction-based fine-tuning.

loubnabnl commented 11 months ago

Hi, you can use the StarCoder repo for using Peft or DeepSpeed you just need to change how you build the dataset samples e.g the prepare_sample_text can just be your code file instead of a question and answer like it's done for instruction tuning, same for the deespeed code.

Yuyz0112 commented 11 months ago

@loubnabnl Thank you for the help! I've started following your suggestion. BTW, could you give some hints on the hardware requirement for fine-tuning the starcoder? Issues in the starcoder repo seem not to have a clear answer. I have 4 A16 GPUs for fine-tuning.