[Question] ORPO Fine-tuning Data Format

[nooobodynose]

Hey!

Trying to understand the prompt format we need to prepare for ORPOTrainer in trl.

In trl ORPO doc it presents the dataset in this format (shortened for visibility):

orpo_dataset_dict = {
    "prompt": [
        "hello"
    ],
    "chosen": [
        "hi nice to meet you"
    ],
    "rejected": [
        "leave me alone"
    ],
}

Let's say we're preparing the ORPO dataset for Mixtral / Mistral Instruct:

should we pass train_dataset param to ORPOTrainer as :

[1] Plain Messages

from datasets import Dataset
# orpo_dataset_dict as defined above...
train_dataset = Dataset.from_dict(orpo_dataset_dict)

or

[2] Messages formatted in its chat template.

from datasets import Dataset

orpo_dataset_dict = {
    "prompt": [
        "<s>[INST] hello [/INST]"
    ],
    "chosen": [
        "hi nice to meet you</s>"
    ],
    "rejected": [
        "leave me alone</s>"
    ],
}

train_dataset = Dataset.from_dict(orpo_dataset_dict)

Thanks in advance for your clarification : )

[jiwooya1000]

Hello @nooobodynose, the second option is the proper way to prepare the dataset for ORPOTrainer in TRL!

For further clarification, the prompt could be preprocessed through tokenizer.apply_chat_template(..., tokenize=False, add_generation_prompt=True) as in your example🙂

[nooobodynose]

Thanks for your quick response, it is clear now : ) Great work 👍