Open mr-c opened 5 years ago
Has anybody found a solution for this? This code has been merged into master branch but it is still not working as expected.
Does someone found a solution to this? I am running a workflow that produces huge amount of data, the final results are a small portion of the intermediary files, and while running some of then could be deleted.
We can implement a routine to delete it at the end but running multiple instance can easily be a problem for storage.
Is anyone working on this issue?
I think this issue was in reference to a specific bug where things that were supposed to be deleted at the end of the run were not.
Deleting intermediates that are no longer needed while the workflow is running is a little bit harder. You need logic that reasons about how intermediate results are going to be used by downstream steps and when they are no longer needed, and handles cases where an input is passed through and returned in the output (keeping it "live"). This is all feasible, CWL workflows definitely contain enough information required to figure this out, but not completely trivial.
@tetron Got it, thanks for the reply. I guess I misinterpret the issue title. But even at the end those files persist in my case and I am trying to debug it. I have considered that was my running conditions but I guess the problem still exists.
From https://github.com/common-workflow-language/cwltool/pull/898#issuecomment-431154202 by @WESClarke