search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.24k stars 399 forks source link

Optionally load trainer state #573

Open Muennighoff opened 2 months ago

Muennighoff commented 2 months ago

I may be missing some nuances with the checkpointing but can we do sth akin to this PR to avoid trying to load the trainer state when the file is not present? Currently, I get FileNotFoundErrors when I try to load a CKPT where I only have the model file.

Muennighoff commented 2 months ago

Type checks are still failing --- do you understand why?

epwalsh commented 2 months ago

Type checks are still failing --- do you understand why?

Looks like you'll need to assert trainer_state is not None in the Trainer when load_trainer_state=True.