search

arcalot / arcaflow-engine

Arcaflow is a highly-portable workflow engine enabling modular and validated pipelines through containerized plugins.
https://arcalot.io/arcaflow/
Apache License 2.0
7 stars 9 forks source link

Use dependency System: Part 2; Marking as unresolvable #197

Closed jaredoconnell closed 3 months ago

jaredoconnell commented 4 months ago

Changes introduced with this PR

The way this works is by marking all outputs that failed as unresolvable.

A callback called OnStepStageFailure was created to allow the providers to self-report failure of specific stages within their lifecycle for a step. The callback is required because the DAG itself doesn't know everything.

Once all outputs are marked unresolvable, the entire workflow exits at that time. It will not keep things running. The old way of determining failure is still implemented, but any activation of it is considered a bug, either due to a false activation, or due to a missing mark of unresolvability.

With the bugs fixed in the updated dependencies, the tests are very resilient based on my testing. This will need testing with real-world workflows.

By contributing to this repository, I agree to the contribution guidelines.

jaredoconnell commented 4 months ago

I need to test if this works with subworkflow/foreach. If it doesn't, I don't think that has to block these changes, but it should be an immediate follow-up PR.

dbutenhof commented 3 months ago

Are we missing a commit here? I thought you were making a few minor changes after the meeting, and I don't see anything.

jaredoconnell commented 3 months ago

Are we missing a commit here? I thought you were making a few minor changes after the meeting, and I don't see anything.

Yes. I will push the commit.