Closed V-FEXrt closed 2 months ago
How does the cleanable list interact with the FnOutputs, for existing code? I definitely prefer how much more explicit this setup is, but I want to be sure that existing code which obviously doesn't interact with CleanableOutputs doesn't break, seeing as the function only seems to be applied as
def fileOutputs = fnOutputs cleanable
in the runners:
Existing code calls CleanableOutputs
All_Outputs
or something and is always populated with the unfiltered outputs in runAlways
. User code that implements its own runner will now have to fill in this field in the RunnerOutput to update to this version of wake which should not be a high burden given they are maintaining their own Runner already.
And, actually, do we want to introduce a new Plan.FnCleanable to provide end users with more control over that list as well?
I think this is a good idea but needs to be carefully described as something even less likely the user wants to play with than Plan.FnOutputs.
How does the cleanable list interact with the FnOutputs, for existing code? I definitely prefer how much more explicit this setup is, but I want to be sure that existing code which obviously doesn't interact with CleanableOutputs doesn't break
Yep its a tiny bit subtle but the tl;dr is that previously Runners
were required to return "All outputs" that they knew about then runAlways
did two things. 1. Pass the unfiltered list of all outputs into the now renamed but identical "AllOutputs" 2. run fnOuputs over the all outputs list and pass that into "outputs". Now the runner directly controls both lists but trivially does the exact same thing.
One subtle and interesting thing is that "AllOutputs" was a very strong misnomer since that list is always empty for the localRunner while the "filtered" outputs of the local runner has the actual discovered outputs. So in the runtime we actually get something like
cleanable_outputs = "all outputs as reported by runner"
hashed_real_outputs = FnOutputs cleanable_outputs (which may increase the size of the list)
unhashed_real_cleanable = cleanable_outputs - hashed_real_outputs
So the only things actually processed separately are items that are in cleanable_outputs but not in "filtered" outputs
And, actually, do we want to introduce a new Plan.FnCleanable to provide end users with more control over that list as well?
Lets punt on that for a bit. I can see it being useful but this change is already large, not one is currently asking for it, and its rather hard to explain what should go into it (esp since the cache won't store files in that list).
Moves the fnInput/fnOutput functions into the runners so they can directly control them. Allows things like the cache runners to cache only actual output files