search

walmartlabs / cookie-cutter

An opinionated micro-services framework for TypeScript
https://walmartlabs.github.io/cookie-cutter
Other
58 stars 35 forks source link

kafka input source should not be stopped if kafkajs error is retriable by kafkajs #377

Closed YashGajera closed 1 year ago

YashGajera commented 1 year ago

kafkajs CRASH event includes restart flag. if kafkajs error is retriable, consumer will try to restart itself upon crash and input source should not be stopped if error is retriable.

In following example kafkjs crashed with retriable error but input source was stopped immediately reacting to crash event. After that consumer got restarted by kafkajs automatically but it was not consuming any messages because of closed kafka input source pipe. This behavior causes consumer to lag even it is connected to broker and able to consume.

{"time":"2022-12-23T03:13:21.374Z","level":"EROR","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Crashed"],"err":[{"error":{"name":"KafkaJSConnectionError","retriable":true,"broker":"kafka-420262885-13-1196072530.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092","code":"ECONNRESET"},"groupId":"prod-chaperone-task-handler-verify-transaction-mis-scans"}]}}
{"level":"ERROR","timestamp":"2022-12-23T03:13:21.374Z","logger":"kafkajs","message":"[Consumer] Restarting the consumer in 300ms","retryTime":300,"groupId":"prod-chaperone-task-handler-verify-transaction-mis-scans"}
{"time":"2022-12-23T03:13:21.375Z","level":"WARN","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Error iterating input source",{"name":"KafkaJSConnectionError","retriable":true,"broker":"kafka-420262885-13-1196072530.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092","code":"ECONNRESET"}]}}
{"level":"INFO","timestamp":"2022-12-23T03:13:21.675Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"prod-chaperone-task-handler-verify-transaction-mis-scans"}
{"level":"INFO","timestamp":"2022-12-23T03:13:24.452Z","logger":"kafkajs","message":"[ConsumerGroup] Consumer has joined the group","groupId":"prod-chaperone-task-handler-verify-transaction-mis-scans","memberId":"task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247-a4999a9a-c266-4441-95b4-9512de2aa21d","leaderId":"task-handler-verify-trxn-ms-795c6657f5-5qxhc-task-handler-45eb3fbc-4971-4e54-80ff-50699c9b5191-373ac50b-51ce-4264-b0bf-0954f8ad465b","isLeader":false,"memberAssignment":{"prod-chaperone-task-verify-missedscan-input":[3],"prod-chaperone-task-changefeed":[1,15]},"groupProtocol":"RoundRobinAssigner","duration":2749}
{"time":"2022-12-23T03:23:21.688Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka.kafka-v2-shared1-prod.scus.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T03:23:21.713Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-11-1196072524.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T03:23:24.471Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-13-1196072530.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T03:23:24.490Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-9-1196072518.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T03:33:21.721Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-4-1196072503.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T03:38:21.731Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-13-1196072530.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T03:53:21.755Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-4-1196072503.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T04:03:21.773Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-9-1196072518.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T04:13:21.800Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-4-1196072503.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T04:23:21.819Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-13-1196072530.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"time":"2022-12-23T04:38:21.856Z","level":"DBUG","ctx":"tooling.task-handler.2_0_1","fields":{"msg":["Kafkajs Network Request Queue Size"],"broker":["kafka-420262885-13-1196072530.scus.kafka-v2-shared1-prod.ms-df-messaging.prod-az-southcentralus-2.prod.us.walmart.net:9092"],"clientId":["task-handler-verify-trxn-ms-795c6657f5-ztf5t-task-handler-c06698a8-1ff5-4638-9586-e2df3e9c0247"],"queueSize":[0]}}
{"tim