fluent / fluent-logger-node

A structured logger for Fluentd (Node.js)
Apache License 2.0
259 stars 83 forks source link

Lambda timeout occurs when using the sender.end() function with setTimeout #173

Open Shantanu35 opened 4 years ago

Shantanu35 commented 4 years ago

Hi, We are using the library for sending logs to Fluentd via Bunyan Logger.

const sender = require('fluent-logger').createFluentSender('tag-prefix', {
            enableReconnect: false,
            host: 'localhost',
            port: 24224,
            requireAckResponse: true
        });

// bunyan logger instance for sending logs to fluentd.
logger = bunyan.createLogger({
            name: 'FluentdLogger',
            streams: [{ stream: sender.toStream('fluentd') }]
        });

public async end( ms = 2000 ) {
        if (this.sender !== null) {
            return new Promise((resolve) => {
                setTimeout(() => {
                    this.sender.end();
                    resolve();
                }, ms);
            });
        }
}
  1. The logs are being sent to fluentd using the above mentioned approach.
  2. The end() function is called at the end of lambda execution to close the socket connections.
  3. However, the end() function when deployed through a lambda, causes it to timeout due to every callback not being processed in the said invocation.
  4. After looking into the issue further, we found that although the sockets are getting closed, the listener port is still active even after sender.end() function call. This might be a reason for the addHandlers() callback to not get processed, and thus causing a timeout.
  5. Unless and until all callbacks are not processed in a particular invocation, the lambda does not exit successfully, thus causing a timeout when I use this library.
  6. The lambda timeout threshold is set to 180 seconds ~ 3 minutes.

Thus, is there any way to handle the above scenario? I read through issues #91 and #89, but the solutions to those are not working in my case. It would be great if you could help.