Open amitmurthy opened 9 years ago
Some progress:
addprocs(2)
rr = RemoteRef(2)
put!(rr, :OK)
Base.remote_do(3, x->nothing, rr)
rr=nothing
@everywhere gc()
@everywhere gc()
Base.remote_do(2, ()->println(Base.PGRP.refs[(1,4)].clientset))
# Execute a dummy remote_do again. This collects the previous ref
Base.remote_do(3, myid)
@everywhere gc()
Base.remote_do(2, ()->println(Base.PGRP.refs))
The second remote_do
results in the reference finally being collected.
I tried changing https://github.com/JuliaLang/julia/blob/dbe94d156bbb07f0c30af6b49a42ab09416f5df7/base/multi.jl#L838-L846 to
elseif is(msg, :do)
f = deserialize(r_stream)
args = deserialize(r_stream)
#print("got args: $args\n")
let f=f, args=args
@schedule begin
run_work_thunk(RemoteValue(), ()->f(args...))
f = nothing
args = nothing
end
end
f = nothing
args = nothing
but that doesn't help.
Do let
blocks also keep references? How do we clear them?
Simpler example:
function foo(rr)
while true
b=take!(rr)
let b=b
f = x->nothing
@schedule ()->f(b)
b = nothing
end
b=nothing
end
end
rr = RemoteRef()
@schedule foo(rr)
put!(rr, ones(10^8));
gc()
gc()
gc()
A reference to the array is held till the loop is entered again, say by a put!(rr, :OK)
. The remote ref does not actually have a reference as evidenced by
julia> isready(rr)
false
julia> Base.PGRP.refs
Dict{Any,Any} with 3 entries:
(1,0) => Base.RemoteValue(false,nothing,Condition(Any[Task (waiting) @0x00007f8d3187f850]),Condition(Any[]),IntSet([1]),0)
(1,2) => Base.RemoteValue(false,nothing,Condition(Any[Task (waiting) @0x00007f8d325fb6c0]),Condition(Any[]),IntSet([1]),0)
(1,1) => Base.RemoteValue(false,nothing,Condition(Any[]),Condition(Any[]),IntSet([1]),0)
Removing the let
statement makes the problem go away.
Forgot to add a comment : as we discussed yesterday this seems to be because the value is stored into a temporary gensym. @vtjnash ?
@yuyichao this - https://github.com/JuliaLang/Distributed.jl/issues/25 - is still an issue. Any ideas?
Scenario:
I have tracked it down to finalizers not being called on the RemoteRef. The finalizer sends a del_msg to the processes actually holding the value.
Finalizers are not being called for regular objects too, when they are serialized to a remote worker.
As can be seen, Foo was not finalized on worker 3.
cc: @carnaval , @JeffBezanson