search

tapios / risk-networks

Code for risk networks: a blend of compartmental models, graphs, data assimilation and semi-supervised learning
Other
2 stars 2 forks source link

Shared memory & improved parallelism #184

Closed dburov190 closed 4 years ago

dburov190 commented 4 years ago

This PR implements a refactored integrator for master equations, including a parallel implementation with shared memory. The speed ups are now such that on 8 CPUs with 128G memory, 10k network's 7-day sweep takes ~570 seconds (out of which 300s are spent in master_eqn_ensemble.simulate, out of which 270s are spent in eval_closure).

I'm currently running tests to verify that the results are not affected (i.e. I haven't introduced a bug somewhere) but I've been playing with this for a week or so, and to me everything seems okay, so feel free to review.

In order to run this code, you need to install ray package.

odunbar commented 4 years ago

Once you have resolved the conflicts this looks good to me, I should note that i have never used python paralellism really i wouldn't be able to spot bugs( @jinlong83 , would you be ok with your parallel version being deleted in this PR as it is replaced here?)

dburov190 commented 4 years ago

Alright, since it's been a while now, I'm gonna merge.

Don't forget to install ray when using parallel mode! @odunbar @jinlong83