ChEB-AI / python-chebai

GNU Affero General Public License v3.0

11 stars 4 forks source link

Runs should be reproducable #12

Closed sfluegel05 closed 1 month ago

sfluegel05 commented 6 months ago

Tasks

[x] Make sure the underlying data splits can be reproduced -> see #10
[x] Reproduce specific runs by some seed
- Lightning should already be using seeds by default, find out where they are stored and how to set seeds manually
- Check if these seeds lead to reproducible runs (according to @MGlauer's experience, they do not) or if there are other options to guarantee reproducability

sfluegel05 commented 2 months ago

I reproduced runs using the same seed and it worked perferctly. I got models with the exact same validation performance. Also, I found out that pytorch lightning changed the default behaviour for seeds. Before v2.2.0: if no seed is specified, set random seed After v2.2.0: if no seed is specified, set seed to 0 Thus, one has to set a seed manually with --seed_everything if one does want a different seed.