Closed patnr closed 1 year ago
Update: I wrote random numbers to file and uploaded it as artefact, and confirmed that it is the random numbers that have changed. Strangely though, the first several (hundreds?) of numbers remain the same.
I should have tried a new install locally first. The problem was not the platform or python version, but simply the dependency 🤦 Specifically, upgrading scipy
to 1.11 causes the error. Probably some part of scipy
is using the global state differently than before. This type of problems are bound to happen with global random state, unless we pin everything, which we dont want, and therefore seem like a good reason to use new random stream/generator system.
BUT, the new system seems problematic for exact reproduction of pseudo-random results (e.g. tests_example_2.py
), which appear liable to change quite often and depending on platform. From docs:
Because of the change in algorithms, it is not possible to reproduce the exact random values using
Generator
for these distributions or any distribution method that relies on them.
More info at stackoverflow.
The upgrade of scipy
also revealed another weakness: Computing matrix square roots has a degree of freedom to it: the sign of each row/column. And for some reason scipy.linalg.svd
sometimes returned different signs from version 1.10 to 1.11. So if you use the svd to compute rectangular square roots then you get different random samples when they're colored by that square root. Fixed in https://github.com/nansencenter/DAPPER/commit/deb07f78070b136d5d8beb5ff056d55a475c99fb
I'm guessing the tests that just failed are due to some change in recent Python patches affecting the random numbers generated. But it's difficult to debug when I cannot seem to access the most recent Python's locally...
DAPPER is/was using numpy's legacy/global
RandomState
system, accessed vianumpy.random.seed/randn/rand/choice/...
. Maybe upgrading to theGenerator
local stream would fix it, and just be a good forward-looking choice?