Fix tokenizer setup in eval.

allenai / open-instruct

Apache License 2.0

1.1k stars 145 forks source link

Fix tokenizer setup in eval. #119

Closed hamishivi closed 4 months ago

hamishivi commented 5 months ago

Fix tokenizer issue with vllm, have to load manually. Also add budget to the eval job script, and send to multiple clusters.

yizhongw commented 5 months ago

Could you provide some context on why manual tokenizer loading is necessary for vllm? The changes look good to me though.

hamishivi commented 5 months ago

Ah, it strictly isn't, but I feel it's preferable to expose it rather than rely on vllm internals? Happy to revert it back. There are some other small fixes in this PR too (bug in toxigen eval).