search

richfitz / remake

Make-like declarative workflows in R
Other
340 stars 32 forks source link

FR: Reuse old results #136

Open krlmlr opened 8 years ago

krlmlr commented 8 years ago

In my project, I have a RATE parameter (set by a function) which controls if I'm working with the full dataset or only with a small sample. Many other targets depend on it directly or indirectly. When I change it, almost everything is rebuilt -- fine. It would be awesome if remake did less work when I set the parameter back to a value for which targets have been computed recently. (Perhaps this is implemented already, and I'm just missing something.)

This interferes with #55 (garbage collection).

richfitz commented 8 years ago

Oh, this is a seriously great idea. There will be some serious new work involved with getting the files to behave (object targets will be easier). The extra storage for non file targets is pretty small.

Garbage collection has been implemented in storr since #55 I think. The only thing to do is to activate it.

Will think more about this, but it seems like a fun thing to implement

krlmlr commented 8 years ago

I forgot about file targets. If we have a hash-based cache for file targets, gives us another layer of security: User deletes file, remakes, wham! it's back there. delete() is also affected.

I guess we need a way to assign a name to results based on the inputs and the code used. Would it work if we hash the hashes of the dependencies (relevant code and data items)?

Probably too much for the immediate future, given the complexity.

richfitz commented 8 years ago

I agree, but it's a good thing to think about. After the first CRAN release perhaps...