Summary - Architecture & Model

[kymr]

Link

• https://netflix.github.io/conductor/intro/

Contents

• High Level Architecture
• The API and storage layers are pluggable and provide ability to work with different backend and queue service providers.

Installing & Running

• Skipped
• For production level use, we need to fix some layers like dynomite.

Runtime Model

• rpc based communication model
• workers are running on a separate machine from the server
• Workers communicate with server over HTTP based endpoints
• employs polling model for managing work queues
• Workers are remote systems and communicates over HTTP (or any supported RPC mechanism) with conductor servers
• Task Queues are used to schedule tasks for workers. We use dyno-queues internally but it can easily be swapped with SQS or similar pub-sub mechanism
  • we need to swap queue layer with rabbitmq
• conductor-redis-persistence module uses Dynomite for storing the state and metadata along with Elasticsearch for indexing backend
  • we need to use base redis cluster instead of dynomite

High Level Steps

• Define task definitions used by the workflow.
• Create the workflow definition
• Create task worker(s) that polls for scheduled tasks at regular interval