Open yebai opened 7 years ago
@yebai: I've reproduced the code on the issue https://github.com/JuliaLang/julia/issues/19450 you opened, and I do not get any segfault. Yet, it seems that there is still this issue https://github.com/JuliaLang/julia/issues/10441.
I can't find the parallelsmc
branch, do you still have the code ?
The solution posted here https://discourse.julialang.org/t/how-do-i-deal-with-random-number-generation-when-multithreading/5636/2 seems to be a good solution to deal with random numbers while using@threads
.
I can't find the
parallelsmcbranch
, do you still have the code?
Thanks for investigating into this. Please find my code in the following branch:
https://github.com/yebai/Turing.jl/tree/hg/parallelsmc
Ps. it might be broken since it was written long before.
I just implemented a version which seems to work on branch multithreaded-PG
and will have a further look into this issue.
See: https://github.com/TuringLang/Turing.jl/blob/multithreaded-PG/src/core/container.jl#L152-L180
FYI I implemented a few months ago a parallel version of IPMCMC: https://github.com/emilemathieu/Turing.jl/blob/7c72a238b4d278720409d845880738d5d2c44ed3/src/samplers/ipmcmc.jl#L70
Great! I’ll have a look at it.
The code on branch https://github.com/TuringLang/Turing.jl/tree/multithreaded-PG now implements a multi-threaded version of the ParticleContainer and an adaptation of the code in https://github.com/emilemathieu/Turing.jl/blob/7c72a238b4d278720409d845880738d5d2c44ed3/src/samplers/ipmcmc.jl#L70 for Julia 0.6. A test for the distributed IPMCMC is currently failing, see below.
As far as I understand the compiler code, it seems that the compiler currently does not generate the inner callback functions, *_model, on all processes. I'll, therefore, have a deeper look at the compiler implementation.
@emilemathieu Did you encounter the same issue? cc @yebai
Great work !
I'm not sure to completely understand what you mean by inner callback functions, *_model
.
I did not encounter such an issue. The only "tricky" part I had was that within @parallel
, objects defined outside the scope of the loop couldn't be updated. Thus I had to explicitly return the objects needed to be updated within the loop.
If I'm running a simple example to test the parallel implementation, e.g.
addprocs(1)
@everywhere using Turing
srand(125)
x = [1.5, 2]
@everywhere @model gdemo(x) = begin
s ~ InverseGamma(2, 3)
m ~ Normal(0, sqrt(s))
for n in 1:length(x)
x[n] ~ Normal(m, sqrt(s))
end
s, m
end
inference = IPMCMC(30, 500, 4)
chain = sample(gdemo(x), inference)
, I get an error that ###gdemo_model#700
is not defined on the second worker. Am I'm doing something wrong here?
Thanks!
@trappmartin You need to use @everywhere
in all places, i.e.
addprocs(1)
@everywhere using Turing
srand(125)
@everywhere x = [1.5, 2]
@everywhere @model gdemo(x) = begin
s ~ InverseGamma(2, 3)
m ~ Normal(0, sqrt(s))
for n in 1:length(x)
x[n] ~ Normal(m, sqrt(s))
end
s, m
end
@everywhere inference = IPMCMC(30, 500, 4)
@everywhere mf = gdemo(x)
chain = sample(mf, inference)
Supporting paralliesm using processes is a pain since it involves data transfer. It's probably better to stick with threads for now until we have a cleaner and more organised code base.
Right, thanks for the tip!
I wanted to have both supported so that one can use multi-threading locally and also distributed computation if necessary. Unfortuanitelly, I'm currently having a broken test for the distributed code and see a similar problem as in TuringLang/Turing.jl#463 even though the non-distributed code works fine. Working on it...
For shared-memory parallelism, I suggest making use of KissThreading.jl https://github.com/bkamins/KissThreading.jl which is shared-memory parallelism free of the closure bug hassle. That package is not registered yet, but I can work on getting it registered if it turns out to be useful here. GPU support is probably also worth considering at some point, but that's a much larger commitment.
Is multiprocessing SMC (that is, not multithreading) supported still? I see some mentions here but it's difficult to find documentation.
I'll ask also, what is the current status of this? The current IPMCMC
implementation doesn't look very parallel... unless I am missing something.
@FredericWantiez we can finally re-visit this functionality based on SSMProblems
...
I looked into the newly released
Threads.@threading
construct in Julia 0.5. It seems that adding parallelization feature to SMC is simple - could be done in a few lines of code (see theparallelsmc
branch).However, the
threading
feature in Julia is still quite fragile (see e.g. here). I've also filed a bug in the Julia repo.Maybe we should wait until the Julia team fix these threading bugs and revisit to this feature in a few months time.