Would be better to save generations on the fly

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.

Apache License 2.0

745 stars 193 forks source link

Would be better to save generations on the fly #58

Open Muennighoff opened 1 year ago

Muennighoff commented 1 year ago

This would be a bigger refactoring but imo it'd be better to save generations after each generation is done & along with that offer restarting from previously unfinished generations (e.g. if it's interrupted or sth)

just leaving this here if someone is interested