search

m2lines / gz21_ocean_momentum

Stochastic-Deep Learning Parameterization of Ocean Momentum Forcing
MIT License
5 stars 0 forks source link

Enable training model with a seeded PRNG #44

Open mondus opened 1 year ago

mondus commented 1 year ago

There is randomness involved in training the model (e.g. the DataLoader shuffling). Being able to control this may be useful.

This should be assertable by training twice on the same input data with the same seed: in theory, we should train two identical models with the identical weights.

(Original issue text)

This would aid some cases within the testing. Care must be taken for cases where stochasticity needs to be evaluated as part of testing.

raehik commented 1 year ago

A file testing the FCN model in Cem's fork includes the line torch.manual_seed(1). Seems reasonable.

https://github.com/CemGultekin1/gz21/blob/5c612e5dde6dcb03b07add09ccbfe54a6a6c4a32/gz21/adam_test.py#L5

raehik commented 1 year ago

66 adds code to seed numpy and everything else that might touch a (P)RNG. It's not yet plugged into the rest of the code like Cem's fork is.

raehik commented 12 months ago

It's not clear how much randomness is actually involved. It may just be DataLoader shuffling...? Need a bit more PyTorch expertise, and understanding of any transformations that we perform.