Open jpsamaroo opened 3 years ago
I was thinking on how to add this in the best way. I assume we want it to be opt-in to begin with?
Would it suffice with some convenience which basically does this:
julia> tt = maketree("root" => ["next" => [(name=string(x), value=1:10) for x in 'a':'k']]);
julia> function myvcat(x, y)
sleep(1)
vcat(x,y)
end
myvcat (generic function with 1 method)
julia> @time fetch(reducevalues((v1,v2) -> Dagger.@spawn(myvcat(v1, v2)), tt));
4.159940 seconds (110.59 k allocations: 5.883 MiB, 2.32% compilation time)
For example, letting exec
also fetch EagerThunk
s and wrapping @spawn
in some utility function, e.g. parallel(f) = (args...) -> Dagger.@spawn(f(args...))
?
After https://github.com/JuliaParallel/Dagger.jl/pull/223 gets merged, Dagger's eager API (
Dagger.@spawn
) should be suitable for use in packages. I would recommend we use it for non-lazy computations in FileTrees so that we can get the benefits of Dagger 100% of the time.