kymr
commented
6 years ago Workflow Definition
- Workflows are defined using a JSON based DSL
- includes a set of tasks that are executed as part of the workflows
- control tasks
- fork
- conditional
- application tasks
- executed on a remote machine
Task Definition
- All tasks need to be registered before they can be used by active workflows.
- A task can be re-used within multiple workflows
- 2 Categories
System Task
- are executed within the JVM of the Conductor server
- managed by Conductor for its execution and scalability
- Conductor provides an API to create user defined tasks that are executed in the same JVM as the engine
- WorkflowSystemTask
Dynamic
- A worker task which is derived based on the input expression to the task
- not statically defined as part of the blueprint
DECIDE
- Decision tasks - implements case...switch style fork
- (like decide what qualities to transcode?)
FORK
- Forks a parallel set of tasks.
- Each set is scheduled to be executed in parallel
FORK_JOIN_DYNAMIC
- Similar to FORK, but rather than the set of tasks defined in the blueprint for parallel execution
- FORK_JOIN_DYNAMIC spawns the parallel tasks based on the input expression to this task
- (like transcoding for each quality?)
JOIN
- Complements FORK and FORK_JOIN_DYNAMIC
- Used to merge one of more parallel branches
- (like all quality videos are transcoded)
SUB_WORKFLOW
- Nest another workflow as a sub workflow task
- Upon execution it instantiates the sub workflow and awaits it completion
EVENT
- Produces an event in a supported eventing system (e.g. Conductor, SQS)
- rabbitmq or kafka?
Worker Task
- Worker tasks are implemented by application(s)
- runs in a separate environment from Conductor
- likes transcoder or media processing server
- The worker tasks can be implemented in any language
- These tasks talk to Conductor server via REST API endpoints to poll for tasks
- no queue operation supported like rabbitmq?
- update its status after execution
- like callback
- reasonable
- Worker tasks are identified by task type SIMPLE in the blueprint.
Link