search

arcalot / arcaflow-engine

Arcaflow is a highly-portable workflow engine enabling modular and validated pipelines through containerized plugins.
https://arcalot.io/arcaflow/
Apache License 2.0
7 stars 9 forks source link

Possible Rewrite To Prevent Locking Issues #81

Open jaredoconnell opened 1 year ago

jaredoconnell commented 1 year ago

One possible redesigned architecture is to make it so that for every dependency in the DAG, there is a channel from the dependency to the dependent node. The advantage of this redesign is it solves a lot of the locking issues that cause bugs right now.

However, it is essential that we add more tests that test the DAG before doing this, since this has the potential to break stuff.

jaredoconnell commented 1 year ago

One section of code that could benefit from some changes:

For where it calls ProvideStageInput, look into unifying, and calling an intermediate object (interface an impl) to check if it needs change the state, and lock if it makes sense. This would be helpful if we add more step providers, since the current setup is prone to locking errors.