search

pytorch / torchtune

PyTorch native finetuning library
https://pytorch.org/torchtune/main/
BSD 3-Clause "New" or "Revised" License
4.27k stars 426 forks source link

Explain magic numbers in our tests #1324

Open SalmanMohammadi opened 2 months ago

SalmanMohammadi commented 2 months ago

If we want to help set a high bar for our contributors (and ourselves), we should be clearer about where many of the magic numbers in our tests come from (e.g. loss values in recipe integration tests). For example, when implementing a new recipe, we should have an uncomplicated system for that contributor to document and provide tests for that recipe.

If these magic numbers come from a particular point in time, e.g. when the recipe was implemented and validated, and prevents further modifications from breaking this, would it help to be more explicit about this? Commenting with the PR?

We can document this both in our CONTRIBUTING.md and our tests themselves.

felipemello1 commented 2 months ago

sharing a reproducible notebook would be interesting, even if its a gist. Not sure if this should be in the PR, or in the unittest file itself.

joecummings commented 2 months ago

Shhhhh it's a secret