search

allenai / open-instruct

Apache License 2.0
1.08k stars 140 forks source link

Migrate to tokenizer + fastchat conv templates #155

Open natolambert opened 2 months ago

natolambert commented 2 months ago

Realistically, doing our own is a losing battle, let's migrate to something maintained.

hamishivi commented 1 month ago

There is a start at this here: https://github.com/allenai/open-instruct/blob/main/eval/templates.py#L112 But more formal integrations with fastchat would be good. I think supporting more flexible generation in general is good. E.g. llama 3 with its multiple eos tokens.