Reuse connection for lambda serverless

[james4388]

Is your feature request related to a problem? Please describe. I use kafkajs in my serverless application: when request come in It invoke a lambda, create a producer, connect to broker, send messages and then disconnect. That was "fine" for little request, for huge spike of request this is problematic: connection timed out. I'm thinking of reuse a producer connection and keep it a live in memory for a while until lambda is no longer hot.

Something like

const producerCache = {};  // This will be kept in lambda pod memory and survive between requests

export const handler = () => { // This handler is invoke in each request
  let producer = null;
  if (!producerCache[clientId]) {
    const kafka = new Kafka({ clientId });,
    producer = kafka.producer({ ... });
    producerCache[clientId] = producer;
  } else {
    producer = producerCache[clientId]
  }
  await producer.connect(); // I believe calling this multiple time will not cause a problem and only keep 1 connection?
}

Have anyone did this before? Is there any better way?

[Nevon]

This is the normal way you keep any object from being GC'd, whether serverless or not. It's the same as if you're connecting to a database or anything else. You can move the await producer.connect into your if (!producerCache[clientId]) { case, since you only need to connect it when you first instantiate it. If the connection drops, it'll reconnect on demand. The connect/disconnect methods are there to signal intent.

You might still have a problem if you don't control the concurrency of your lambda function, since a spike in requests will cause multiple concurrent lambdas to start, each of which will have its own connections to you Kafka cluster, but this is handled by controlling the concurrency of your lambda - nothing specific to KafkaJS.

The only thing I would add is that you should probably add a listener for SIGTERM and disconnect your client when the lambda actually shuts down. Otherwise you'll leave half-open connections on the brokers, which will take some time to clean up. See https://github.com/aws-samples/graceful-shutdown-with-aws-lambda/blob/main/nodejs-demo/hello-world/app.js#L3-L12

[james4388]

Well look like lambda shutdown is not SIGTERM but SIGKILL so nothing can be done here I guess. Thank you for the answer

[Nevon]

By default, yes, but if you register an extension the lambda does get sigterm - at least according to the documentation.

For a function with registered external extensions, Lambda supports graceful shutdown. When Lambda service is about to shut down the runtime, it sends a SIGTERM signal to the runtime and then a SHUTDOWN event to each registered external extension. Developers can catch the SIGTERM signal in their lambda functions and clean up resources.

[samba2]

Just want to share that to enable SIGTERM in an AWS lambda apparently you have to have some extension installed. What I understand is that as soon as you are running with an extension you get this extended live cycle mgmt. As I really just wanted to enable SIGTERM handling and nothing more, I went with this receipe:

https://github.com/aws-samples/graceful-shutdown-with-aws-lambda/tree/main/nodejs-demo

In a nutshell:

• reference as lambda layer the already existing "Lambda Insights" extension
• add an IAM role required for the extension

In terraform:

resource "aws_lambda_function" "my_func" {
  ...
  layers = ["arn:aws:lambda:eu-central-1:580247275435:layer:LambdaInsightsExtension:14"]
  ...

I am wondering if a slimmer no-op extension would do the same trick 🤔

[EmmanDizon]

Hi @samba2 @Nevon question, do we need to close connection of kafka or lambda function will automatically close it if lambda function died ?