Parallelization Factor for MSK Event Source Mapping

[davidtwomey]

Hi,

Is there a way to set concurrency of the lambda function so that multiple batches can be processed simultaneouly?

From the aws lambda cli docs I see there is a --parallelization-factor which I think is supported for Kinesis streams, but not MSK?

Are there any plans to support this for MSK?

If I understood the MSK event source correctly, it polls a target MSK topic across partitions and invokes a lambda function with the obtained batch synchronously? So in order to poll for another batch, the invoked lambda function must complete execution before a subsequent poll can begin?

Therefore, the throughput that can be handled is a function of:

• The number of topic partitions (more partitions allows for higher throughput)
• The batch size of the MSK event source (10,000 is currently the max)
• The execution time of the lambda function.

For example, if the execution takes 100ms vs. 10secs this makes a difference to the consumer lag in high-throughput topics?

Many thanks,

David

[davidtwomey]

@rcchakr any insight here? Or could you point me in the direction of someone who could answer this?

[rcchakr]

@davidtwomey This is something the Lambda team is looking to do but there are no specific timelines yet.