Does it support Preference data (for training Reward / DPO)?

[ericxsun]

🚀 Feature Request

The preference data looks like this:

{
    "chosen":
    [
        {"role": "user", "content": "abcd"},
        {"role": "assistant",  "content": "abcef"},
        ...
    ],
    "rejected":
    [
        {"role": "user", "content": "abcd"},
        {"role": "assistant", "content": "abcef"},
        ...
    ]
}

This data is used to train a Reward Model or DPO

I'm wondering if it's possible to use streaming for this kind of situation. And How? Thanks very much.

[XiaohanZhangCMU]

Yes it should work out of the box. Use MDSWrite to convert your preference data to MDS , and create a streaming dataset out of it. Use json as the encoding method. Let me know if you see any issue.

[ericxsun]

Yes it should work out of the box. Use MDSWrite to convert your preference data to MDS , and create a streaming dataset out of it. Use json as the encoding method. Let me know if you see any issue.

Thank you for quickly explain, I'll try it.

[karan6181]

@ericxsun Wondering, have you tried the @XiaohanZhangCMU suggestion? Did it work?

[XiaohanZhangCMU]

Hi @ericxsun want to follow up here before closing this issue.