search

OS2iot / OS2iot-backend

This repository contains the backend to the project OS2iot.
Mozilla Public License 2.0
10 stars 7 forks source link

BUG: Lost data, "Connection error: write after end" error from kafkajs #208

Closed bkd231 closed 1 year ago

bkd231 commented 1 year ago

Data are not correctly processed by the OS2IoT. The log shows that the message is received by the OS2IoT, but then cannot be sent to kafka for further processing as the opening connection to kafka fails (os2iot opens a new connection every time a message need to be sent to kafka). The longer OS2IoT works, the issue is more common. After a few hours of operation, almost all packages are lost during processing. It works fine again after the OS2IoT backend restart.

{"level":"ERROR","timestamp":"2023-03-15T06:00:04.6630000Z","logger":"kafkajs","message":"[Connection] Connection error: write after end","broker":"kafka-cluster-kafka-bootstrap:9092","clientId":"os2iot-client","stack":"Error [ERR_STREAM_WRITE_AFTER_END]: write after end\n at writeAfterEnd (_stream_writable.js:266:14)\n at Socket.Writable.write (_stream_writable.js:315:5)\n at Object.sendRequest (/tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:312:27)\n at SocketRequest.send [as sendRequest] (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/index.js:139:23)\n at SocketRequest.send (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:85:10)\n at RequestQueue.sendSocketRequest (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/index.js:168:19)\n at RequestQueue.push (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/index.js:148:12)\n at /tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:307:29\n at new Promise (<anonymous>)\n at sendRequest (/tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:302:14)\n at async Connection.send (/tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:321:53)\n at async Broker.[private:Broker:sendRequest] (/tmp/os2iot/backend/node_modules/kafkajs/src/broker/index.js:890:14)\n at async Broker.metadata (/tmp/os2iot/backend/node_modules/kafkajs/src/broker/index.js:191:12)\n at async /tmp/os2iot/backend/node_modules/kafkajs/src/cluster/brokerPool.js:155:25\n at async Cluster.refreshMetadata (/tmp/os2iot/backend/node_modules/kafkajs/src/cluster/index.js:134:5)\n at async Cluster.addMultipleTargetTopics (/tmp/os2iot/backend/node_modules/kafkajs/src/cluster/index.js:192:11)"}

We have tested it with kafka versions: 3.2.0, 3.3.2, 2.7.0 with the very same results.

ramogens commented 1 year ago

Hi @bkdkmd Thanks for reporting. I'm sorry to hear about this bug. We'll look into it and get back. Best, Mogens, OS2iot product koordinator