search

hddm-devs / hddm

HDDM is a python module that implements Hierarchical Bayesian parameter estimation of Drift Diffusion Models (via PyMC).
http://ski.clps.brown.edu/hddm_docs/
Other
268 stars 119 forks source link

Reproducibility issue #45

Open mikecroucher opened 7 years ago

mikecroucher commented 7 years ago

Hi

I'm attempting to seed the random number stream to get identical results every time. A notebook showing what I mean is at

https://github.com/mikecroucher/hddm_test/blob/master/hddm_test.ipynb

That notebook performs the same calculation 4 times while setting random number seed. Runs 2 and 3 are identical as I would hope but these are different from runs 1 and 4.

My guess is that some sort of parallelisation is going on which may cause numbers to be drawn from the random stream in different orders sometimes.

Any ideas? Should I be initialising the seed anywhere else?

twiecki commented 7 years ago

That is quite puzzling. I suppose it could be in pymc somewhere but we set the seed for tests and also in our simulation experiments and didn't encounter the issue (I think, perhaps we didn't run it in the same process repeatedly).

mikecroucher commented 7 years ago

My main concern at the moment is 'Am I seeding everything correctly?' Can you advise on that please?

mikecroucher commented 7 years ago

Can I ask that if you run my notebook, do you get the same behaviour as me?

twiecki commented 7 years ago

CC @janfreyberg @madslupe

madslupe commented 7 years ago

Hi Mike,

I get the same error when running your code. I tried running different versions, changing the seed number and other basic things, but I was never able to get the same estimates for all four runs. I googled it and saw others had reported the same problem with pymc, but I didn’t find any explanation. I did notice that generating data with hddm.generate gave different results the first time you run it after having estimated data, but I’m not sure if that’s a clue.

Mads

On Wed,1-Nov,2017, at 2:48 , Thomas Wiecki notifications@github.com wrote:

CC @janfreyberg https://github.com/janfreyberg @madslupe https://github.com/madslupe — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hddm-devs/hddm/issues/45#issuecomment-341004512, or mute the thread https://github.com/notifications/unsubscribe-auth/ADT4IR7n0Z6qzd5hynDU6ERDht-zPQIaks5syBQxgaJpZM4P-HSe.

drussellmrichie commented 5 years ago

Is there any update on this? I'd love to be able to set a random seed and exactly reproduce my results.