varunsatish / llama-recipes-fertility

1 stars 0 forks source link

Allow for evaluation by number of steps, similar to Hugging Face logging_steps argument #22

Closed varunsatish closed 2 months ago

varunsatish commented 2 months ago

Llama-recipes supports evaluation at each epoch, but not the number of steps natively. It would be useful for us to add evaluation by number of steps.

varunsatish commented 2 months ago

Previously, we were only able to do evaluation at the end of each epoch. This was not ideal for us given that our experiments were going to begin using just one epoch.

I amended the code (with a lot of Claude-ing) to do evaluation at intervals determined by the logging_steps flag. For example, we can choose to do evaluation every 10, 1000, 1000 steps.

We can still do evaluation at each epoch by determining the number of steps in each epoch. This is determined by the quantity: # training sample / batch size.

CC: @msalganik

msalganik commented 2 months ago

👍