search

richfitz / remake

Make-like declarative workflows in R
Other
340 stars 32 forks source link

List/array targets #8

Open richfitz opened 10 years ago

richfitz commented 10 years ago

Implement parts of the split/apply/merge pattern.

Simplest to imagine for object -> object targets. Start with an object that has a length and apply a rule to each element in the list. Merge goes the other way.

For object -> file, each element could become a file within a directory, but getting the file naming is going to be tricky.

For file -> object we could go file in directory to object, fairly easily.

For file -> file we could go from one directory to another.

krlmlr commented 7 years ago

What about iterative processes that can't be parallelized, but where we'd like to keep intermediate results?

richfitz commented 7 years ago

So, somewhere I implemented a whole pile of list target support, got into a massive tangle, threw my hands in the air in despair and ripped it all out again. The work would have been spread between remake and storr. So anywhere you see "list target" it's referring to that excursion.

You're right; there are a huge number of corner cases and bits that are tricky. I'd been thinking of list targets purely as a lapply/map operation but things like fold are going to be non-parallelisable. It also appeared that it was going to mean that too much logic ended up in the remake files.