search

richfitz / remake

Make-like declarative workflows in R
Other
340 stars 32 forks source link

remake for workflows implemented in R packages #85

Open wlandau opened 8 years ago

wlandau commented 8 years ago

I just found out about remake, and it looks incredibly useful. I would love to use it for my own reproducible workflows. However, for portability and convenience, I like to encapsulate my workflows in R packages. Is there a way to designate an installed package's built-in files, functions, and datasets as dependencies? For example, if I change mypackage::myfunction(), rebuild, and reinstall, is there a way for remake to recognize the change and update the targets accordingly? Also, does remake respond to newly installed versions of external packages?

richfitz commented 8 years ago

Package versioning is not supported - there's an old issue I'll crossreference here (#13)

For a package, this would be easy enough to do by depending on the result of packageVersion("yourpackage") manually for everything but that's going to get tiresome.

In theory, an interface could look like another entry in the .yml for which packages to check and then I could just use all the usual dependency checking machinery that is there (currently I stop descending if a function has a namespace).

Do you have a public example of a package based workflow this might apply to?

wlandau commented 8 years ago

My packages that implement workflows are private because they reproduce papers whose publication is pending. I was thinking of generating output only for the parts of the package that change in an update. For example, if I update one function in a workflow package, I would only want to regenerate the output files that depend on that function. Relying on the package version would likely trigger a lot of redundant work, so I agree with your decision in issue 13 to remove dependence on package versions.

richfitz commented 8 years ago

The current code dependency bits do work out which target depend on the updated functions, so that should work.

Totally understand the private repo thing.

I'm not currently doing a lot of remake work at the moment but I may get to this later in the month unless you want to take a stab at a PR.

wlandau commented 8 years ago

That's really cool that changes to functions trigger smart remakes. (I tested it out with a small bunch of functions implemented in a script.) Is that because you use storr to keep track of objects in the .remake folder?

By the way, I added a MWE to show how I was hoping remake would work with reproducible workflows implemented as packages. Updates to one of the functions inside the toy package do not trigger remakes. I hope that's just a problem with my usage of remake. I realize that you don't plan to work on this now, but just know that the MWE is available if you feel like looking.

richfitz commented 8 years ago

Thanks for the MWE on this - I've started a bit of work on remake so may get this done in the next month or so

wlandau commented 8 years ago

Sure thing, I look forward to the updates. I'll be sure to follow the issues.