jeff-regier / Celeste.jl

Scalable inference for a generative model of astronomical images
MIT License
183 stars 28 forks source link

segfault in MCMC test_infer with multiple threads #720

Closed kbarbary closed 6 years ago

kbarbary commented 6 years ago

Looks like there's some printing in the MCMC code that might be causing a problem with threads.

$ echo $JULIA_NUM_THREADS 
4

$ runtests.jl infer
make: Nothing to be done for 'all'.
make: Nothing to be done for 'all'.
make: Nothing to be done for 'all'.
make: Nothing to be done for 'fetch'.
WARNING: Skipping long running tests.  To test everything, run tests with the flag --long-running
Running test_infer.jl
[1]<1> INFO: 18:54:30.806: 614 primary sources, 5 target sources in 164.39, 164.41, 39.11, 39.13

Running AIS on with patch size 
Running AIS on with patch size 
Running AIS on with patch size 
Running AIS on with patch size 
signal (11): Segmentation fault

signal (11): Segmentation fault
while loading /home/kyle/code/Celeste.jl/test/test_infer.jl, in expression starting on line 9
julia: /home/centos/buildbot/slave/package_tarball64/build/deps/srccache/libuv-52d72a52cc7ccd570929990f010ed16e2ec604c8/src/unix/stream.c:956: uv__write_callbacks: Assertion `(((const QUEUE *) (&(stream->loop)->active_reqs) == (const QUEUE *) (*(QUEUE **) &((*(&(stream->loop)->active_reqs))[0]))) == 0)' failed.

signal (6): Aborted
while loading /home/kyle/code/Celeste.jl/test/test_infer.jl, in expression starting on line 9

[...more output...]

Allocations: 22556853 (Pool: 22553469; Big: 3384); GC: 47
Segmentation fault (core dumped)
jeff-regier commented 6 years ago

I've always run the tests single threaded since they don't really parallelize. We should probably get the print statements removed though since we do want to use multithreading for mcmc.

andymiller commented 6 years ago

Ok, I can incorporate that into the branch I'm working on. I'll add a verbose flag to turn off printing.

kbarbary commented 6 years ago

If we need logging there, I guess the Celeste mini-logging module is thread-safe?

jeff-regier commented 6 years ago

Yeah, Log.info is thread safe. @andymiller You just have to add import ..Log to the MCMC header--the Log module is part of Celeste.