EMFILE Errors

[BrandynThornton]

Hello,

I've deployed my express app to production and started seeing consistent errors in the logs related to too many open sockets. This led me to the following issue in the aws-serverless-express repo https://github.com/awslabs/aws-serverless-express/issues/239 which is almost identical to the issue I'm seeing. I believe the root cause is a DNS error and I'm enabling connection pooling to hopefully address that issue.

However, while reviewing that issue I noticed that they recommend not initializing the express server inside of the handler function. I see that the _express/handler.js source code for this component has the same issue creating the server inside of the handler function which is not recommended.

Can we move the createServer call to the global scope as recommended in the linked issue?

For some context, I don't see any issues with the server running under normal load but at peak the application receives around 40k requests a minute and starts to log around 500 errors a minute.

Error messages look like the following

2020-09-12T21:05:59.973Z 60c8a0e0-f3a3-49b5-ba54-80fb1da9e66c ERROR Error: listen EMFILE: too many open files /tmp/server-io7p3itg74o.sock at Server.setupListenHandle [as _listen2] (net.js:1296:21) at listenInCluster (net.js:1361:12) at Server.listen (net.js:1458:5) at startServer (/var/task/_express/node_modules/aws-serverless-express/src/index.js:152:17) at promise (/var/task/_express/node_modules/aws-serverless-express/src/index.js:226:9) at new Promise (<anonymous>) at Object.proxy (/var/task/_express/node_modules/aws-serverless-express/src/index.js:211:14) at Object.exports.handler (/var/task/_express/handler.js:60:42) at Runtime.exports.handler (/var/task/_serverless/handler.js:35:40) at Runtime.handleOnce (/var/runtime/Runtime.js:66:25) { code: 'EMFILE', errno: 'EMFILE', syscall: 'listen', address: '/tmp/server-io7p3itg74o.sock', port: -1 }

2020-09-12T21:05:59.921Z 26b2321b-c97f-424d-ae89-ec74d4a524c0 ERROR Error: connect EMFILE /tmp/server-i5dzkspeeep.sock at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1141:16) { errno: 'EMFILE', code: 'EMFILE', syscall: 'connect', address: '/tmp/server-i5dzkspeeep.sock'}

Thanks

[eahefnawy]

Thanks for suggesting @BrandynThornton ... I believe we could move it outside the handler. Need to look deeper into the codebase to see if that could cause specific issues though

[eahefnawy]

This should now be fixed and published.