Question: frequent kafkajs timeouts when using with lambda

[dhvidding-accenture]

Perhaps more of a question than an issue. I'm trying to produce to MSK using lambdas rather than a long-lived compute resource that can have a long-standing connection. I'm running into lots of timeout issues which I believe are due to these limitations with MSK and IAM. However, because I am connecting with lambda I am forced to create multiple connections:

• A cluster that uses IAM access control can have up to 3000 TCP connections per broker at any given time. To increase this limit, you can adjust the listener.name.client_iam.max.connections or the listener.name.client_iam_public.max.connections configuration property using the Kafka AlterConfig API or the kafka-configs.sh tool. It's important to note that increasing either property to a high value can result in unavailability.

• Limits on TCP connections. A cluster that uses IAM access control can accept new connections at a rate of up to 20 TCP connections per broker per second for all broker types, except for the type kafka.t3.small. Brokers of type kafka.t3.small are limited to 4 TCP connections per broker per second.

Are there any best practices for managing connections with kafkajs and this plugin when using lambda to limit timeout errors? For example, is it better to connect outside the lambda handler to preserve the connection across the container hosting the lambda?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.