Decoupled Job Completion Pattern in AWS lambda environment

[ryanelee]

We try to follow this Decoupled Job Completion Pattern, use AWS lambda to create our job workers against the self-managed Camunda 8 in the AWS EKS cluster. There is a lambda function that we create for polling the active Camunda task based on the pattern we designed for defining all the service tasks in the BPMN. The task type is always external and workerTypre header is the key for distinguishing the specific job worker since we don't want to change our task poller lambda every time if there is new service task added in the BPMN.

The task poller lambda is triggered by a scheduler every minute, now we're facing an inefficient problem to poll the active tasks from Camunda, the task has potential 1 minute delay to be distributed. It's not acceptable since we have about 15 service tasks for now.

Is there a guidance that we could leverage this pattern in a serverless environment in order to poll the task immediately? Or in order to utilize the long polling mechanism, we have to move the task poller away from lambda and deploy a containerized task for that?

[jwulf]

we have to move the task poller away from lambda and deploy a containerized task for that?

Correct, you need a task poller.

Here is an example of one: https://github.com/jwulf/zeebe-zero-scale