search

honeydipper / honeydipper

Basically, your Swiss army knife for systems engineering and operations. An event-driven, policy-based orchestration system, with a pluggable open architecture.
MIT License
59 stars 14 forks source link

Making workflow engine truly stateless #225

Open Charles546 opened 4 years ago

Charles546 commented 4 years ago

The workflow engine currently has an in memory data structure that holds all in-fly workflow sessions. We can't safely restart the daemon when there are workflows running. We have an issue to have a way to gracefully restart the daemon #190, but for a busy daemon, it will be challenging to find a window to do upgrade/maintenance etc.

The proposed solution is to persist the data into some kind of database such as consul, redis, memcached, couchdb etc. This way, we can safely/gracefully shutdown the daemon for upgrade or maintenance.

Charles546 commented 4 years ago

The issue #226 is the first step in resolving this issue. We still need to change the code in internal/workflow to use the cache, and the communication between the workflow engine and operator needs to be node neutral (currently, we use engine IP to identify the node of the engine like below.)

https://github.com/honeydipper/honeydipper/blob/ade16d76df0df06302899af99e767fd23deb270d/drivers/cmd/redisqueue/main.go#L100-L111