Use a workflow engine or state machine?

[alahiff]

The deployment workflow has been getting more and more complex. It would be good to use (if possible) an existing workflow system / business process manager instead. Ideal features would include:

• reproduce existing functionality, i.e. generate list of clouds, rank them, try each in turn until one works, also deal with obtaining tokens, deploying Ansible nodes if necessary, ...
• automatically handle timeouts and retries
• be resilent, i.e. if it's restarted or crashes it can pick up where it left off (the current code will not do this)
• handle deletions & reversions

Possible options:

• https://wiki.openstack.org/wiki/TaskFlow
• https://airflow.apache.org/ (similar usecase: https://cloud.google.com/composer/)
• https://github.com/pytransitions/transitions

[alahiff]

Wait for https://github.com/prominence-eosc/imc/issues/17 to be completed first

[alahiff]

Other options:

• Uber Cadence: https://cadenceworkflow.io/
• Event sourcing: https://eventsourcing.readthedocs.io/en/stable/