Live eval support for pre-training runs - Githubissues

AnswerDotAI / bert24

Apache License 2.0

66 stars 4 forks source link

Live eval support for pre-training runs #120

Closed rbiswasfc closed 4 weeks ago

rbiswasfc commented 2 months ago

Changes

Added download_artifacts_from_wandb.py to download checkpoints in eval compute
Added wandb_log_live_eval.py to log live eval summary
Added run_live_eval.sh to run live eval jobs
MLMMLU hyperparam changes in misc_jobs.py
ruff formatting in superglue_jobs.py
Minor config related changes in generate_eval_config_from_checkpoint.py and run_evals_from_checkpoints.py

Discussions

If any, please include references to the relevant issues/previous PR/discord discussions around these changes.

Tests

[ ] Is the new feature tested? (Not always necessary for all changes -- just adding to the checklist to keep track)
[ ] Have you ran all the tests?
[ ] Do the tests all pass?
[ ] If not, have you included an explanation of which tests this PR breaks and/or why (below this checklisT)