Closed mkitti closed 1 year ago
I guess the solution here is to close all properties objects when they are done?
Following up on https://github.com/JuliaIO/HDF5.jl/pull/1013#discussion_r1003866863, how about we define a function
function with(fn, args...)
try
fn(args...)
finally
for arg in args
finalize(arg)
end
end
end
so e.g. https://github.com/JuliaIO/HDF5.jl/blob/6022be0e1e44073152a84b5063baf5bed6eaf19b/src/groups.jl#L24-L35 could be written as
function create_group(parent::Union{File,Group}, path::AbstractString; pv...)
with(_link_properties(path), GroupCreateProperties()) do lcpl, gcpl
pv = setproperties!(lcpl, gcpl; pv...)
isempty(pv) || error("invalid keyword options $pv")
return create_group(parent, path, lcpl, gcpl)
end
end
or perhaps a macro to do the same, e.g.:
function create_group(parent::Union{File,Group}, path::AbstractString; pv...)
@with lcpl=_link_properties(path) gcpl=GroupCreateProperties() begin
pv = setproperties!(lcpl, gcpl; pv...)
isempty(pv) || error("invalid keyword options $pv")
return create_group(parent, path, lcpl, gcpl)
end
end
I was thinking something similar, but calling close
rather than finalize
.
@simonbyrne that sounds a lot better than the current setup and remind me of something similar in the D language for finalizers. The function form seems simpler than going with a macro, unless there is an intrinsic benefit of the macro.
Does calling finalize run the finalizers on the current task/thread or does it schedule it on another task/thread?
I'm pretty sure it runs on the current task (i.e. it calls it immediately). finalize
also has the advantages of working on any object (even immutable ones), and being cleared once called (i.e. the finalizers from the object aren't later called again by the GC), e.g.:
julia> mutable struct Foo
end
julia> close(f::Foo) = println("done")
close (generic function with 1 method)
julia> f = Foo()
Foo()
julia> finalizer(close, f)
Foo()
julia> close(f)
done
julia> f = nothing; GC.gc() # close called again
done
julia> f = Foo()
Foo()
julia> finalizer(close, f)
Foo()
julia> finalize(f)
done
julia> f = nothing; GC.gc() # close not called again
I'm pretty sure it runs on the current task (i.e. it calls it immediately)
This seems more general than HDF5.jl. While I'm happy to experiment with it here, perhaps we should consider putting it in a small package at some point? Maybe we should incubate it as a subdirectory package or at least put it within it's own module.
The @with
macro looks quite like a let
statement. Perhaps the macro should act on a let
statement rather than a begin
block?
Closed by #1021
When running long jobs, I'm occasionally running into this traces like this. What I think is happening is that the finalizer is running to close out a property while another thread is trying to access the HDF5 library. This can be be mitigated by running
GC.gc()
when between jobs where threads are used.Note I'm doing this while using HDF5.jl on a single spawned thread.