The PR makes a couple tweaks to improve the repeatability of the results. In particular it:
Provides the option to specify a random seed in the config file. I wasn't able to get None to pass into the training function, so I put in -9999 and a line in river-dl/train.py to convert the -9999 to None. If someone else has a suggestion to avoid this, that would be great.
Removes a potential source of non-determinism in the gw loss
With these settings (and a specified seed), I did 25 runs of base river-dl (no gw loss function) and the losses were identical to 16 decimals across all epochs
The PR makes a couple tweaks to improve the repeatability of the results. In particular it:
With these settings (and a specified seed), I did 25 runs of base river-dl (no gw loss function) and the losses were identical to 16 decimals across all epochs