Closed 3fon3fonov closed 3 years ago
I haven't seen a BrokenPipeError yet, and there's nothing I see from the tracebacks that can help me. However, if I were you, I'd google for the error you are seeing: https://discourse.pymc.io/t/multiprocessing-windows-10-brokenpipeerror-errno-32-broken-pipe/2259. It seems that others have seen it in multiprocess
and multiprocessing
.
The code you posted produces the error... but takes 30+ hours to do so? Yikes, that's not conducive to testing. You don't see the behavior on any shorter runs?
Hi @mmckerns ! Thanks for the prompt reply!
After I posted I found this pathos issue https://github.com/uqfoundation/pathos/issues/143 where you had suggested to use pool._clear() and/or pool.restart() together with ProcessPool. And thats what I did now:
sampler.run_nested(print_progress=print_progress,dlogz_init=stop_crit,nlive_init=nwalkers,
maxiter = ns_maxiter, maxcall = ns_maxcall,use_stop = ns_use_stop, wt_kwargs={'pfrac': ns_pfrac}) #nlive_batch=1
thread.close()
thread.join()
thread._clear()
thread.restart()
I hope I did it right. I started another run and it seems to work. However, I am still waiting for the outcome (10 hours running already). Since I narrow a bit the parameter priors, I expect that this run will be shorter.
Generally, my tool works perfectly with pathos (and dill). Shorter runs are not a problem. I suspect it is a memory issue, although not resource-related, since I am using really expensive servers with plenty of CPUs and RAM.
I will post when the run is done.
Ok, narrowing down the priors had some effect and now the run seems to converge for about 15 h....
[15:02:00, 9.19it/s, batch: 0 | bound: 2072 | nc: 25 | ncall: 4264414 | eff(%): 2.585 | loglstar: -inf < 31572.672 < inf | logz: 31365.917 +/- 0.853 | dlogz: 0.111 > 0.100]
However, it stuck in the last step (I am waiting to progress already over 2 h)... It should print the results and save a dill session when dlogz: > 0.100. I suspected it went there, but cannot progress further anymore.
"Top" says that my python3 process had used 103 GB of RAM. Quite large, yes, but definitely not a problem for the hardware, I still have a lot of RAM left in this machine...
So it seems that pool._clear() and/or pool.restart() is not a real solution... What do you think?
pathos
stores the pools in a singleton, and for fast calculations, it reduces overhead significantly. However, in certain cases, like yours, it can lead to memory issues if clear
is not used. You could try to use a pathos.pools._ProcessPool
, which doesn't utilize the caching... and has an interface identical to multiprocessing.Pool
.
Side note, I remember seeing your tool before -- I built an optimization-based sampling technique that has been demonstrated to produce valid models with at least an order of magnitude speed-up (paper is under review now) -- but the base of the code is in this module: https://github.com/uqfoundation/mystic/blob/master/mystic/samplers.py. May be of interest to you.
What I'd also suggest is to monitor the memory, if possible, and see if that becomes an issue with your current code. And if _ProcessPool
doesn't work, then we keep digging.
Thanks a lot, @mmckerns ! Your mystic
package looks awesome! I will definitely try it out! I am already using scipy.optimize, which seems to be already included in mystic
, s the transition should be smooth!
Yet, let's first see what to do with the reason for this issue.
These days I was trying all possible _ProcessPoos/ProcessPool combinations and still no success. I do not see an error just the dynesty
sampler seems to stuck at the end.... I personally think this could be related to dynesty
, because sampling via emcee
(MCMC) executes with no problem... I wonder what kind of tests I can do with dynesty
and pathos
to monitor the CPU/memory behavior.
It really sucks because I really need dynesty-like nested Sampling instead of the MCMC. "Lighter" runs are fine; the "heavy" runs, however, fail and I cannot understand why.
What do you mean, it gets "stuck" at the end? Like execution "hangs"... ? This can happen if there's in parallel if a queue is expected to be pulled from, and its not happening... and that execution is in blocking mode. Hard to tell without seeing it.
Might I suggest you also try the ThreadPool
, if you are having ProcessPool
issues... that would at least let you know where you stand at running in parallel.
Also, you might want to include a monitor in the scipy
optimization, like this: https://github.com/uqfoundation/mystic/blob/master/examples2/constrained_scipy.py
Sorry for not reporting my progress. I ended up busy with other stuff. After all, I was able to get my run done but in only special conditions of the dynesty
sampler, which have nothing to do with pathos. I think, my problems are related to dynesty
, but I am still debugging. I also open an issue in the dynesty
GitHub. Josh's answer could be informative, you may like to check:
Looks like this can be closed. Please reopen if needed.
I am getting serious troubles with some really expensive runs, and I think the problem could be in pathos, but I am not 100% sure. I am using the "dynesty" nested-sampling sampler to get prsteriors of really heavy model. like this:
And over a week now, no three different machines I am hitting into this:
In the above example over 34.5 hours of running on a 80 CPU machine are lost! Shorter runs usually compleate with no problem.
Help is really needed and highly appreciated!