search

shelfio / dynamodb-parallel-scan

Scan large DynamoDB tables faster with parallelism
MIT License
71 stars 10 forks source link

ParallelScan | Error: connect EMFILE 52.94.11.16:443 - Local (undefined:undefined) with nodeJs v18 & dynamodb-parallel-scan v3.4 #244

Closed mwj-1992 closed 1 year ago

mwj-1992 commented 1 year ago

Hi, I have faced an issue once upgraded the nodeJs engine to v18.x and the dynamodb-parallel-scan to v3.5,

Here is the error log 2023-09-07T02:22:05.427Z 544fbba4-df65-4dbd-975e-c440cdc16a5e ERROR Gateway | DynamoDB | ParallelScan | Error: connect EMFILE 52.94.11.16:443 - Local (undefined:undefined) at internalConnect (node:net:1067:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:463:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1324:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:110:8) { errno: -24, code: 'EMFILE', syscall: 'connect', address: '52.94.11.16', port: 443, '$metadata': { attempts: 1, totalRetryDelay: 0 } }

After reverting the node to v14.x and the dynamodb-paralled-scan 3.4, the issue gone already, Has anyone faced the same issue ?

mwj-1992 commented 1 year ago

closed the issue, as found that nodev18 enabled keepAlive for Db connections by default, for this reason by adopting parallel scan we initiate plenty of DB connections( as in our current code we initiate the client db connection instance inside the lambda handler) by right should creat the db connection outside of the handler.