peterprib / node-red-contrib-kafka-manager

Implement
GNU General Public License v3.0
22 stars 12 forks source link

This flow causes Node-Red to crash. #67

Open deanjott opened 1 year ago

deanjott commented 1 year ago

The code in the 'function' node is

msg.topic ="listTopics";
return msg;

image

The flow as taken from test/generalTest.json, then just adding the HTTP request and response nodes with the function node only supplying the value for msg.topic which normally comes from the 'inject' node in this example. If this flow is triggered from the 'inject' node then it operates correctly. It's something in the 'msg' from the HTTP node that's killing Node-Red from within the 'Kafka Admin' node. Any ideas on what I can try or supply to diagnose further?

peterprib commented 1 year ago

If you post the error message in nodered log may be able to help. I am in process of major rebuild to make it more resilient.

deanjott commented 1 year ago

Hi Peter. Here's the logging for the crash you requested

18 Aug 16:06:35 - [debug] Kafka Broker {"label":"Metadata refresh","node":"cff5e121c002955d","connected":true} 18 Aug 16:06:35 - [debug] Kafka Broker {"label":"adminRequest","action":"listTopics","properties":["action","callback","error"]} 18 Aug 16:06:35 - [debug] Kafka Broker {"label":"adminRequestSend","action":"listTopics","properties":["action","callback","error"]} 18 Aug 16:06:35 - [info] [Kafka Admin:getTopicNames] connected and processing 1 messages 18 Aug 16:06:35 - [red] Uncaught Exception: 18 Aug 16:06:35 - [error] TypeError: Converting circular structure to JSON --> starting at object with constructor 'Socket' | property 'parser' -> object with constructor 'HTTPParser' --- property 'socket' closes the circle at JSON.stringify () at Logger.send (C:\Windows\System32\config\systemprofile.node-red\node_modules\node-red-contrib-logger\logger.js:21:62) at KafkaAdminNode.processInput (C:\Windows\System32\config\systemprofile.node-red\node_modules\node-red-contrib-kafka-manager\kafkaManager\kafkaAdmin.js:65:27) at Admin. (C:\Windows\System32\config\systemprofile.node-red\node_modules\node-red-contrib-kafka-manager\kafkaManager\kafkaBroker.js:232:53) at Admin.emit (node:events:513:28) at KafkaClient. (C:\Windows\System32\config\systemprofile.node-red\node_modules\kafka-node\lib\admin.js:23:10) at Object.onceWrapper (node:events:627:28) at KafkaClient.emit (node:events:513:28) at C:\Windows\System32\config\systemprofile.node-red\node_modules\kafka-node\lib\kafkaClient.js:754:14 at C:\Windows\System32\config\systemprofile.node-red\node_modules\kafka-node\lib\kafkaClient.js:661:7

peterprib commented 1 year ago

looks like there may be an error being raised or the debugger message has been past a recursive JSON property. My guess is the payload from http has been past onto to kafkAdmin node. If you rename property payload, it may get over the issue. I will make sure such is not an issue with the new version .

peterprib commented 1 year ago

I assume you may be attempting to store entire http payload. Basically you have an object that needs to be converted to serializable form or stringify. In either case you need to appreciate in consuming message it will not have access to all properties of original json object.

deanjott commented 1 year ago

Hi Peter.
If I use the 'inject' node to start the flow as in the example supplied with the code, then the 'msg' object has only this: _msgid: "0413d338d3d9f28c" payload: 1692632762207 topic: "listTopics"

If the flow is started with the 'http' node followed by 'function' node to set 'topic' value then it's the same as when it's triggered by by the 'inject' node, except there are the additional elements of 'req' and 'res' in the 'msg' structure related to the http request.

The cause of this issue is the node-red-contrib-logger code. The 'msg.req' part of the http request is by definition a circular object. The node-red-contrib-logger is using JSON.stringify(message) which will throw on this circular reference because JSON.stringify doesn't handle circular references (though, there are libraries that can). I know you're aware of the possibility because you commented in the code about circular references in other parts of the logger code. Perhaps in your rebuild efforts you can address this line of code.

For my use case, I'm not so concerned with this logging so I just commented out the offending line of code in my local copy (node-red-contrib-logger logger.js line 21)

peterprib commented 1 year ago

Thanks for detail, will fix issue by fixing logger as this clearly should be better at handling situation.