search

huggingface / alignment-handbook

Robust recipes to align language models with human and AI preferences
https://huggingface.co/HuggingFaceH4
Apache License 2.0
4.18k stars 354 forks source link

Add `scripts/run_kto.py` #165

Open alvarobartt opened 1 month ago

alvarobartt commented 1 month ago

Description

As briefly discussed with @lewtun this morning, this PR adds the scripts/run_kto.py script to fine-tune LLMs using the trl.KTOTrainer from the alignment-handbook.

The script should work as is, but still needs to be tested, cc in case you're interested @jiwooya1000 and @nlee-208

The main reference used to put the script together has been https://github.com/huggingface/trl/blob/main/examples/scripts/kto.py.

HuggingFaceDocBuilderDev commented 1 month ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.