Open hannahilea opened 9 months ago
On more thought, this should maybe instead be a kwarg to transform!
... there are various other options we want to pass to a nothrowdag, and having a bunch of independent functions wouldn't let those compose nicely. (nor would a big old list of kwargs, but at least it would be useable!)
What about a separate function like get_intermediates(dag)
to allow getting whatever intermediates exist at call time? (Assuming they are stored in the dag; if not then this doesn’t make sense). Then one could query for them after running transform!
.
That would be a good idea, except that the dag is stateless and stores nothing. :)
Currently, NoThrowDag's are nice and simple; their return type is either the return type of their last step (if everything succeeds) or a
NoThrowReturn{missing}
if they fail.In a private project, we've run into a case where---even on failure---it would be useful to have the intermediate results of upstream steps.
There are assorted ways this could be accomplished; one that could work (proposed by @kendal-s) is on failure to return a special type that the internal state of the results up to the point of failure, in addition to the basic
NoThrowResult{missing}
(which should still be returned, so that all warnings/errors are still captured.I'd kind of like to keep the original basic implementation, but maybe we can add a special alternative function to
transform!(dag, input)
that's liketransform_and_return_intermediates!(dag, input)
or something, that always returns the intermediates (on success or failure) in addition to the originalNoThrowResult
? (I know use of "and" in function names is often a code smell; perhaps someone can come up with a better/alternative name.)