uptake / updraft

R package for building flexible workflows
Other
13 stars 12 forks source link

Intermediate cache clearing #17

Closed jayqi closed 6 years ago

jayqi commented 6 years ago

Addresses #12

codecov-io commented 6 years ago

Codecov Report

Merging #17 into master will increase coverage by 0.42%. The diff coverage is 84%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #17      +/-   ##
==========================================
+ Coverage   82.45%   82.88%   +0.42%     
==========================================
  Files          14       14              
  Lines         610      625      +15     
==========================================
+ Hits          503      518      +15     
  Misses        107      107
Impacted Files Coverage Δ
R/workflow_interface.R 11.11% <0%> (-0.66%) :arrow_down:
R/execution.R 96.38% <100%> (+0.33%) :arrow_up:
R/workflow_dag.R 92.09% <82.35%> (+0.74%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b8a8042...7161039. Read the comment docs.

jayqi commented 6 years ago

Note that this implementation may sometimes fail to clear a module's cache during execution. For example, suppose you had this workflow:

A -> B -> C
 ↘
  D

If module C completes, then it will clear module B's cache. But if this happens before module D completes, then hasCompletedAllDownstreamModules(A) will always evaluate to FALSE because B has already been cleared.

This is because module$hasCompleted() checks for the existence of the cache to determine completion. We'd need to maintain global states of completion at the workflow level in order to have complete, correct information.

jayqi commented 6 years ago

@cwschultz88 in case you missed it. ^

jameslamb commented 6 years ago

@jayqi I really appreciate the way you ASCII-arted that example. Super clear