beacon-biosignals / TransformSpecifications.jl

Structured processing elements with I/O specifications
MIT License
0 stars 0 forks source link

Add `NoThrowDAG` return type #29

Open hannahilea opened 9 months ago

hannahilea commented 9 months ago

Currently, NoThrowDag's are nice and simple; their return type is either the return type of their last step (if everything succeeds) or a NoThrowReturn{missing} if they fail.

In a private project, we've run into a case where---even on failure---it would be useful to have the intermediate results of upstream steps.

There are assorted ways this could be accomplished; one that could work (proposed by @kendal-s) is on failure to return a special type that the internal state of the results up to the point of failure, in addition to the basic NoThrowResult{missing} (which should still be returned, so that all warnings/errors are still captured.

I'd kind of like to keep the original basic implementation, but maybe we can add a special alternative function to transform!(dag, input) that's like transform_and_return_intermediates!(dag, input) or something, that always returns the intermediates (on success or failure) in addition to the original NoThrowResult? (I know use of "and" in function names is often a code smell; perhaps someone can come up with a better/alternative name.)

hannahilea commented 9 months ago

On more thought, this should maybe instead be a kwarg to transform!... there are various other options we want to pass to a nothrowdag, and having a bunch of independent functions wouldn't let those compose nicely. (nor would a big old list of kwargs, but at least it would be useable!)

ericphanson commented 9 months ago

What about a separate function like get_intermediates(dag) to allow getting whatever intermediates exist at call time? (Assuming they are stored in the dag; if not then this doesn’t make sense). Then one could query for them after running transform!.

hannahilea commented 9 months ago

That would be a good idea, except that the dag is stateless and stores nothing. :)