shashi / FileTrees.jl

Parallel computing with a tree of files metaphor
http://shashi.biz/FileTrees.jl
Other
88 stars 6 forks source link

`combine=f` does not respect laziness #3

Closed shashi closed 4 years ago

shashi commented 4 years ago

I was trying to use combine to do a tree-reduce where I bin files using regex search and replace (mv), but if the data is lazy loaded, combine will get the value as a Thunk.

In spirit we should be able to split out the combine function into two parts: that which works on file name field and that which works on the value, and use _lazy_if_lazy(f_value) to combine the values.

But so far my API thoughts have been awkward about this.

shashi commented 4 years ago

Maybe by default combine=f f should be applied on the value, but there can be a wrapper NodeFunction(f) which makes it apply on the node itself. It's usually the case that we don't want to change the file name, we just want to merge the contents.