Describe the bug
As soon as kafkajs initiates a connection to a Kafka cluster, requestQueue.fullfillRequest is hit which begins an infinite loop in the RequestQueue of the checkPendingRequests command. This is due to the fact that checkPendingRequests schedules a new check every time it finishes, regardless of whether or not any requests are pending.
In addition, the scheduleAt parameter for the next check is set to a negative value by default, so the scheduled check runs immediately, and then the cycle starts again. This results in the process running as fast as your little CPU can handle.
In a bare setup, this does not result in any real issue, it simply consumes a lot of cycles without any benefit (AFAIK). However, if you are using the newrelic package to monitor your code, it will shim setTimeout during an transaction and create a SegmentTrace object. As there are thousands of these being created, if the transaction continues, you will quickly run out of memory in your Node heap or your server.
Expected behaviorcheckPendingRequests should only be run when there are pending requests possible. If nothing has been throttled or queued up, there is no reason to re-run this code, as pushing a new message will still flush the queue appropriately.
Describe the bug As soon as kafkajs initiates a connection to a Kafka cluster,
requestQueue.fullfillRequestis hit which begins an infinite loop in theRequestQueueof thecheckPendingRequestscommand. This is due to the fact thatcheckPendingRequestsschedules a new check every time it finishes, regardless of whether or not any requests are pending.In addition, the
scheduleAtparameter for the next check is set to a negative value by default, so the scheduled check runs immediately, and then the cycle starts again. This results in the process running as fast as your little CPU can handle.In a bare setup, this does not result in any real issue, it simply consumes a lot of cycles without any benefit (AFAIK). However, if you are using the
newrelicpackage to monitor your code, it will shimsetTimeoutduring an transaction and create aSegmentTraceobject. As there are thousands of these being created, if the transaction continues, you will quickly run out of memory in your Node heap or your server.To Reproduce Link to PR with failing tests: https://github.com/tulios/kafkajs/pull/1703 Failing test output: https://github.com/tulios/kafkajs/pull/1703/checks?check_run_id=27018283006 Shows that within one second, upon initial connection, this method is hit 694 times.
Note: Code for a fix is commented out in the PR.
Expected behavior
checkPendingRequestsshould only be run when there are pending requests possible. If nothing has been throttled or queued up, there is no reason to re-run this code, as pushing a new message will still flush the queue appropriately.Observed behavior Infinite loop. See above.
Environment: