Update all versions for new OLMo models (and others)

allenai / open-instruct

Apache License 2.0

1.21k stars 166 forks source link

Update all versions for new OLMo models (and others) #151

Closed natolambert closed 3 months ago

OyvindTafjord commented 5 months ago

I see a RUN pip install -r requirementstxt in there, missing period?

natolambert commented 5 months ago

Yeah @OyvindTafjord , but regardless you need to hardcode that line to requirements-olmo.txt. I couldn't get docker build-args working for it.

natolambert commented 5 months ago

Note, this will also close #136, #143, and #145 when merged (pushing soon after this comment).

natolambert commented 5 months ago

In the meantime, you can use the image olmo-adapt-v1 at ai2

hamishivi commented 4 months ago

It would be good to merge this sometime soon, probably we should do a test and see if there are any big performance differences with this branch and our old one when it comes to evaluations.

hamishivi commented 3 months ago

Okay, I cleaned up this PR a tonne. I've confirmed:

Evaluations match old evaluation numbers within some tolerance
Finetuning matches old finetuning (although you have to use sum loss to match jax land)
Llama 3 finetuning + eval works, and actually increasing the bsz beyond 2 (by hacking the code a bit) gives results closer to, if not outright better, than my jax-trained models.
Finetuning + evaluating olmo works out of box
Chat template is now automatically added to HF tokenizer during finetuning

hamishivi commented 3 months ago

For reference (it's in the docs now too) the commit to use for the 'old' open-instruct is: f3424591638ed63b31d5869abd867932c359c1ed