search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.79k stars 487 forks source link

Adding script for processing many intermediate checkpoints at once for offline evals #731

Open IanMagnusson opened 1 month ago

IanMagnusson commented 1 month ago

Making a draft PR for this so we can consider merging this in to main. It would be nice if we could do this so we don't run into version issues if we train models in the future that are not compatible with the version of the code forked here.

jenahwang commented 1 month ago

@soldni , don't you already have a checkpoint converter script that runs in Beaker?

He does. What this one does is very similar but the focus is on batch conversion and wildcard acceptance. And it was written for oe-eval consistent ranking project with expediting its pipeline in mind.

soldni commented 1 month ago

@jenahwang would it be possible to merge in your changes to the other script? or consolidate the two?

it's confusing to have two conversion scripts, and it doubles maintenance.