search

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 15 forks source link

determinism adjustments #145

Closed janetrbarclay closed 2 years ago

janetrbarclay commented 2 years ago

The PR makes a couple tweaks to improve the repeatability of the results. In particular it:

  1. Provides the option to specify a random seed in the config file. I wasn't able to get None to pass into the training function, so I put in -9999 and a line in river-dl/train.py to convert the -9999 to None. If someone else has a suggestion to avoid this, that would be great.
  2. Adds an environment variable (https://github.com/janetrbarclay/river-dl/blob/8554393adac7ab3e72edcca4ac4da497731aa9f5/river_dl/train.py#L186) for choosing the deterministic operations in tf >= 2.1
  3. Removes a potential source of non-determinism in the gw loss

With these settings (and a specified seed), I did 25 runs of base river-dl (no gw loss function) and the losses were identical to 16 decimals across all epochs