search

TuringLang / docs

Documentation and tutorials for the Turing language
https://turinglang.org/docs/
MIT License
229 stars 99 forks source link

Reproducibility between Julia < 1.7 and >= 1.7 #378

Closed xukai92 closed 1 year ago

xukai92 commented 1 year ago

The default RNG has changed since Julia 1.7 (https://docs.julialang.org/en/v1.7-dev/NEWS/#Random) and some of notebooks with an explicit RNG (e.g. the Bayesian ODE one) are no longer reproducible as the tutorials are generated using Julia < 1.7. Is there a way we can code the notebook such that we set some RNG globally which works for all Julia versions?

devmotion commented 1 year ago

The only way to really guarantee reproducibility of the random numbers is to use StableRNGs. Other RNGs may vary between Julia versions and different architectures (even if the type of the RNG is the same). But probably even that is not sufficient to completely reproduce the tutorials if you switch Julia versions - some packages (such as Turing) dropped support for Julia < 1.7, and hence new releases (which might change the results, contain bug fixes etc.) are not necessarily available on all Julia versions. Another point is that Julia also changes numerical algorithms between different versions (IIRC e.g. exponentiation recently), and hence it is not guaranteed that you get exactly the same results even if no randomness is involved.

Hence given these limitations and other reproducibility problems, I'm not sure if we should try to use a more explicit RNG in the tutorials. IMO it would make the tutorials less readable (since we have to pass it around all the time) and still we might not be able to reproduce the results exactly on different Julia versions.

xukai92 commented 1 year ago

Thanks for the note David. In this case, can we make sure the tutorials are generated using Julia >=1.7? We might need to update the TOML files as well to include the Julia versions.

devmotion commented 1 year ago

I can look into that this evening. IIRC it was very annoying before because one had to go through all the Manifest.toml files and adjust them manually. But with the newly introduced weekly action that updates the Manifest.toml files automatically, it should be much easier and less work for us if we want to keep up with the latest stable Julia release.