shashi / FileTrees.jl

Parallel computing with a tree of files metaphor
http://shashi.biz/FileTrees.jl
Other
88 stars 6 forks source link

Use Dagger's eager API #55

Open jpsamaroo opened 3 years ago

jpsamaroo commented 3 years ago

After https://github.com/JuliaParallel/Dagger.jl/pull/223 gets merged, Dagger's eager API (Dagger.@spawn) should be suitable for use in packages. I would recommend we use it for non-lazy computations in FileTrees so that we can get the benefits of Dagger 100% of the time.

DrChainsaw commented 2 years ago

I was thinking on how to add this in the best way. I assume we want it to be opt-in to begin with?

Would it suffice with some convenience which basically does this:

julia> tt = maketree("root" => ["next" => [(name=string(x), value=1:10) for x in 'a':'k']]);

julia> function myvcat(x, y)
           sleep(1)
           vcat(x,y)
       end
myvcat (generic function with 1 method)

julia> @time fetch(reducevalues((v1,v2) -> Dagger.@spawn(myvcat(v1, v2)), tt));
  4.159940 seconds (110.59 k allocations: 5.883 MiB, 2.32% compilation time)

For example, letting exec also fetch EagerThunks and wrapping @spawn in some utility function, e.g. parallel(f) = (args...) -> Dagger.@spawn(f(args...))?