Kafka js consumer lag issue

[carboncs16]

We are using kafka js to consume messages from around 350 topics with 1000 1500 messages per second. It works fine but sometimes when the throughput is high, then the consumer starts to lag. Is there anything I can do.

[JxJPu9bD]

{"level":"INFO","timestamp":"2023-08-25T16:40:42.907Z","logger":"kafkajs","message":"[Consumer] Stopped","groupId":""}

I have also encountered similar issues, which may mainly be performance or configuration issues. There are indeed network issues, but my consumption of another Java version on the same host is normal and can maintain connectivity

[jdrago999]

@carboncs16 and @JxJPu9bD thank you for sharing information for this issue.

Other information that could be helpful to solve this:

• what OS?
• what version of NodeJS?
• what version of kafkajs?
• what version of kafka?

[carboncs16]

@jdrago999 OS: Linux (Not sure about flavour, will let you know) Node.js: 14.17.3 Kafkajs: 2.2.4 Kafka: 2.7

[wuriyanto48]

we are experiencing the same issue when the throughput is high

Kafkajs: 2.2.4

{"level":"ERROR","timestamp":"2023-10-19T11:59:31.955Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Request Fetch(key: 1, version: 11) timed out","groupId":"consumer-group-288578","retryCount":5,"stack":"KafkaJSNonRetriableError\n  Caused by: KafkaJSRequestTimeoutError: Request Fetch(key: 1, version: 11) timed out\n    at SocketRequest.timeoutRequest (node_modules/kafkajs/src/network/requestQueue/socketRequest.js:107:19)\n    at /node_modules/kafkajs/src/network/requestQueue/index.js:94:21\n    at Map.forEach (<anonymous>)\n    at Timeout._onTimeout (/node_modules/kafkajs/src/network/requestQueue/index.js:92:23)\n    at listOnTimeout (internal/timers.js:551:17)\n    at processTimers (internal/timers.js:494:7)"}
{"level":"ERROR","timestamp":"2023-10-19T11:59:31.984Z","logger":"kafkajs","message":"[Connection] Response LeaveGroup(key: 13, version: 3)","broker":"10.60.171.54:9093","clientId":"my-app","error":"The coordinator is not aware of this member","correlationId":11,"size":63}
{"level":"INFO","timestamp":"2023-10-19T11:59:31.984Z","logger":"kafkajs","message":"[Consumer] Stopped","groupId":"consumer-group-288578"}
{"level":"ERROR","timestamp":"2023-10-19T11:59:31.985Z","logger":"kafkajs","message":"[Consumer] Restarting the consumer in 8274ms","retryCount":5,"retryTime":8274,"groupId":"consumer-group-288578"}

[ZibaBahari]

I have this issue too There is not any solution???

[PRBMY]

Facing the same issue, any way to resolve this?