PygmalionAI / data-toolbox

Our data munging code.
GNU Affero General Public License v3.0
34 stars 9 forks source link

Add ChatML format #38

Closed TearGosling closed 1 year ago

TearGosling commented 1 year ago

Quick job I did to add OpenAI's ChatML format. This is the basic plaintext version - no fancy JSON objects with this one. One variant is the regular ChatML format, and the other is a modified variant of their last example in that document that includes names (and thus the Turn object's name attribute.) The original purpose was just for designating few-shot prompting, but a named variant can come in handy here.

Merging quickly since it's tested and working.