Cannot access a HTTP GET After distributing a flow to Node-RED instances

What are the steps to reproduce?

Here is the corresponding JSON of this flow:

[{"id":"23b7a311.c2d91c","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test2","method":"get","upload":false,"swaggerDoc":"","x":120,"y":960,"wires":[["f62b16d5.355ec8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"aa54ed4a.26107","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":630,"y":960,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"f62b16d5.355ec8","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":340,"y":860,"wires":[["db91eed6.8e26"]],"constraints":{"connstraint2(device2-port12002)":{"id":"connstraint2(device2-port12002)","deviceName":"device2-port12002","fill":"#4286f4","text":"connstraint2(device2-port12002)"}}},{"id":"db91eed6.8e26","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":440,"y":960,"wires":[["aa54ed4a.26107"]],"constraints":{}}]

What happens?

When I visit http://localhost:12001/test2, the server (run on port 12001) does not reply a HTTP response.

There is a error 5 Aug 14:14:55 - [error] [dnr:ba71fbe6.1be358] TypeError: Converting circular structure to JSON shown on Node-RED instance 1 (run on port 12001).

What do you expect to happen?

When I visit http://localhost:12001/test2, the server replies a HTTP response (with 200 HTTP state code)
Node-RED instance 2 (run on port 12002) prints "log1" on its logs

Please tell us about your environment:

[x] Node-RED version:0.18.7
[ ] node.js version: v8.11.2
[ ] npm version:6.1.0
[ ] Platform/OS: macOS 10.13.6 (17G65)
[ ] Browser: Google Chrome Version 67.0.3396.99 (Official Build) (64-bit)

Hi,

Thanks for your bug report. Basically, there are two things that DNR has not been supporting correctly and I'm working on a solution right now:

Support non-serializable objects: since DNR has to send the message to an external devices, it has to serialize the object before sending out. This is not always possible for all existing Node RED nodes as many of them use non-serializable object and passing the pointer between nodes. For a local, single device deployment this is ok as the object can be retrieved using its pointer. But not if it has to be sent to an external devices.
Support stateful nodes: some Node RED nodes are stateful, they keep their state within the system and the nodes share the state by keeping a pointers to it. In a DNR setting, when a message is sent to an external device, the state of the sending node is not sent along with it. When the message arrived back to the device, it does not know the existing state of the system.

Http nodes are a typical example of these requirements. 1) http nodes keep a circular object with their message (req, res objects) so that it can be used by other local http nodes. 2) http nodes keep the state about the current connecting sockets so that they can send the response back to the right clients.

Basically these use cases are not quite aligned with the data flow paradigm where we have a one way flow of the data. Yes, HTTP is kind of a request/response model, not a data flow primitives. That's why it's not fully compatible to the DNR system.

I'll let you know when I have a proper solution to this.

Perfect! And thanks for your explanation.

Thanks for your fix.

It works in some cases. For example, it works when visiting "http://127.0.0.1:12001/test1" and "http://127.0.0.1:12001/test4", but it does not work for the "http://127.0.0.1:12001/test2" case.

[{"id":"505317ca.0dc098","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test1","method":"get","upload":false,"swaggerDoc":"","x":100,"y":320,"wires":[["e6ff056c.d83758"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"1000a757.b1c9e9","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":710,"y":320,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"e6ff056c.d83758","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":300,"y":320,"wires":[["44285ee1.91796"]],"constraints":{}},{"id":"44285ee1.91796","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":320,"wires":[["1000a757.b1c9e9"]],"constraints":{}},{"id":"8039886f.d2ce78","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test2","method":"get","upload":false,"swaggerDoc":"","x":100,"y":540,"wires":[["91e5b87b.7bd6f8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"d82ea071.d0dab","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":650,"y":540,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"91e5b87b.7bd6f8","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":280,"y":540,"wires":[["25077d69.c26ef2"]],"constraints":{"connstraint2(device2-port12002)":{"id":"connstraint2(device2-port12002)","deviceName":"device2-port12002","fill":"#4286f4","text":"connstraint2(device2-port12002)"}}},{"id":"25077d69.c26ef2","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":460,"y":540,"wires":[["d82ea071.d0dab"]],"constraints":{}},{"id":"722b21f.d17bfe","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test4","method":"get","upload":false,"swaggerDoc":"","x":120,"y":700,"wires":[["1e7ab81b.56a2c8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"2fa7f88e.566ab8","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":710,"y":700,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"1e7ab81b.56a2c8","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":300,"y":700,"wires":[["f5318bf9.e1c378"]],"constraints":{"connstraint2(device2-port12002)":{"id":"connstraint2(device2-port12002)","deviceName":"device2-port12002","fill":"#4286f4","text":"connstraint2(device2-port12002)"}}},{"id":"f5318bf9.e1c378","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":700,"wires":[["2fa7f88e.566ab8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}}]

Hi Wei,

(remedies at the end)

This is an interesting known concept (wire fragmentation) that I haven't implemented yet. Basically there is nothing wrong with the flow design, it is just the wires are not smart enough as yet to realise this. You can read more about this in my paper at the last IEEE Edge conference about link fragmentation. An example of it is shown here:

Some more explanations:

Since there is no constraint on the LogOnly2 node, the device #2 thinks that it can certainly run this node.

Thus it just forwards the message from the remote LogOnly1 to it's local LogOnly2.

When it continues, it finds that it cannot run the last http node, it then looks for who else is waiting for this data. I cannot find anyone.

Nobody asks for it!

Yes, let's take a look at device #2, from the http node's point of view, it sees that it can still get data from the local LogOnly2 at device #1 because LogOnly2 doesn't have any constraints, meaning any device will run it.

So http out of device #1 never asks for remote data. LogOnly2 of device #1 asks for data, but it never receives it because LogOnly1 of device #2 just send the data locally.

The LogOnly2 of device #2 then just drops the message completely because nobody was asking for it.

Remedy:

To overcome this at the moment, I think the only way is to make it a bit more explicit about the wires. For example, putting all the constraints onto the flow like test4 did.

I'll work more on this, but an ultimate fix might be a bit longer than expected to solve such fundamental problem.

Thanks.

PS/ I'm working on the DNR experiment as well, hopefully during the weekend I'll got something interesting for you.

On Wed, Aug 15, 2018 at 3:55 AM, swsmile notifications@github.com wrote:

Thanks for your fix.

It works in some cases. For example, it works when visiting " http://127.0.0.1:12001/test1" and "http://127.0.0.1:12001/test4", but it does not work for the "http://127.0.0.1:12001/test2" case.

[image: image] https://user-images.githubusercontent.com/15687188/44145039-ee0643d4-a081-11e8-98b7-7b6abae7eb69.png

[{"id":"505317ca.0dc098","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test1","method":"get","upload":false,"swaggerDoc":"","x":100,"y":320,"wires":[["e6ff056c.d83758"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"1000a757.b1c9e9","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":710,"y":320,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"e6ff056c.d83758","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":300,"y":320,"wires":[["44285ee1.91796"]],"constraints":{}},{"id":"44285ee1.91796","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":320,"wires":[["1000a757.b1c9e9"]],"constraints":{}},{"id":"8039886f.d2ce78","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test2","method":"get","upload":false,"swaggerDoc":"","x":100,"y":540,"wires":[["91e5b87b.7bd6f8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"d82ea071.d0dab","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":650,"y":540,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"91e5b87b.7bd6f8","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":280,"y":540,"wires":[["25077d69.c26ef2"]],"constraints":{"connstraint2(device2-port12002)":{"id":"connstraint2(device2-port12002)","deviceName":"device2-port12002","fill":"#4286f4","text":"connstraint2(device2-port12002)"}}},{"id":"25077d69.c26ef2","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":460,"y":540,"wires":[["d82ea071.d0dab"]],"constraints":{}},{"id":"722b21f.d17bfe","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test4","method":"get","upload":false,"swaggerDoc":"","x":120,"y":700,"wires":[["1e7ab81b.56a2c8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"2fa7f88e.566ab8","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":710,"y":700,"wires":[],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}},{"id":"1e7ab81b.56a2c8","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":300,"y":700,"wires":[["f5318bf9.e1c378"]],"constraints":{"connstraint2(device2-port12002)":{"id":"connstraint2(device2-port12002)","deviceName":"device2-port12002","fill":"#4286f4","text":"connstraint2(device2-port12002)"}}},{"id":"f5318bf9.e1c378","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":700,"wires":[["2fa7f88e.566ab8"]],"constraints":{"connstraint1(device1-port12001)":{"id":"connstraint1(device1-port12001)","deviceName":"device1-port12001","fill":"#407266","text":"connstraint1(device1-port12001)"}}}]

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/namgk/dnr-editor/issues/12#issuecomment-413163303, or mute the thread https://github.com/notifications/unsubscribe-auth/AA9PoDg8wyb9zZoSdX6OB-GVM9HXbb2jks5uQ_4OgaJpZM4VvYNW .

Support non-serializable objects: since DNR has to send the message to an external devices, it has to serialize the object before sending out. This is not always possible for all existing Node RED nodes as many of them use non-serializable object and passing the pointer between nodes. For a local, single device deployment this is ok as the object can be retrieved using its pointer. But not if it has to be sent to an external devices.

It seems every Node-RED instance is a single individual (as they don't share any memory space), whatever they are run on the same machine or different machines.

Is it right to think that in original Node-RED, it doesn't need to serialize an object before passing the corresponding message of that object (as there is exactly one Node-RED instance)? However, in DNR, every object has to be serialized when passing the message of that object as that message may be passed to another Node-RED instance (in this case, simply using a pointer will lose that object). Or maybe we can add a condition if easy - if a message will be passed to another Node-RED instance, we just serialize all objects in that message, otherwise we simply pass the pointer.

Thanks

you're right, for single instance of Node-RED, pointer passing is enough for inter-node communication. But for inter-device communication in DNR, you do need to serialize your message.

On Mon, Aug 20, 2018 at 4:41 AM, swsmile notifications@github.com wrote:

Support non-serializable objects: since DNR has to send the message to an external devices, it has to serialize the object before sending out. This is not always possible for all existing Node RED nodes as many of them use non-serializable object and passing the pointer between nodes. For a local, single device deployment this is ok as the object can be retrieved using its pointer. But not if it has to be sent to an external devices.

It seems every Node-RED instance is a single individual (as they don't share any memory space), whatever they are run on the same machine or different machines.

Is it right to think that in Node-RED, it doesn't need to serialize a object before passing the corresponding message of that object, however, in DNR, every object has to be serialized as the message of that object may be passed to another Node-RED instance (in this case, simply using a pointer will lose that object). Or we can add a condition if easy - if a message will be passed to another Node-RED instance, we just serialize all objects in that message, otherwise we pass the pointer.

Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/namgk/dnr-editor/issues/12#issuecomment-414286985, or mute the thread https://github.com/notifications/unsubscribe-auth/AA9PoOiv0IpF_JiSr00qZCftOIFkYgDxks5uSqBfgaJpZM4VvYNW .

Thanks for your reply, Nam. That is very clear which helps me a lot:).

BTW, may I ask how it is going when you are playing with the DNR experiment and any idea on how to set the node constraints for the flow in our experiment so as to improve performance as much as possible?

My idea is like use a Node-RED instance (device 3) as a server (provide a POST endpoint) and use other instances (in this case, device1 and device2) to do something.

Thanks

Hi,

I cannot reproduce your problem. From my side it seems running correctly. Can you post version of the node-red-contrib-dnr, dnr-editor, as well as flatted?

From here:

/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/flatted

You can type in > grep ver package.json

Thanks

On Sat, Aug 25, 2018 at 2:49 PM, swsmile notifications@github.com wrote:

Another strange case I have met: The Flow is like:

[image: image] https://user-images.githubusercontent.com/15687188/44622789-39aeb580-a8b8-11e8-8eed-a5071d2feece.png

Note that each "new-constraintX" constraint includes the deviceX only. JSON for the flow:

[{"id":"722b21f.d17bfe","type":"http in","z":"1b53be18.b0a122","name":"","url":"/test3","method":"get","upload":false,"swaggerDoc":"","x":100,"y":640,"wires":[["1e7ab81b.56a2c8"]],"constraints":{"new-connstraint1":{"id":"new-connstraint1","deviceName":"device1","fill":"#187f67","text":"new-connstraint1"}}},{"id":"2fa7f88e.566ab8","type":"http response","z":"1b53be18.b0a122","name":"","statusCode":"","headers":{},"x":650,"y":640,"wires":[],"constraints":{"new-connstraint1":{"id":"new-connstraint1","deviceName":"device1","fill":"#187f67","text":"new-connstraint1"}}},{"id":"1e7ab81b.56a2c8","type":"function","z":"1b53be18.b0a122","name":"LogOnly1","func":"// msg.payload = {\"a\":\"b\"};\nconsole.log(\"log1\");\nreturn msg;","outputs":1,"noerr":0,"x":280,"y":640,"wires":[["f5318bf9.e1c378"]],"constraints":{"new-connstraint2":{"id":"new-connstraint2","deviceName":"device2","fill":"#4286f4","text":"new-connstraint2"}}},{"id":"f5318bf9.e1c378","type":"function","z":"1b53be18.b0a122","name":"LogOnly2","func":"msg.payload = {\"a\":\"b\"};\nconsole.log(\"log2\");\nreturn msg;","outputs":1,"noerr":0,"x":460,"y":640,"wires":[["2fa7f88e.566ab8"]],"constraints":{"new-constraint3":{"id":"new-constraint3","deviceName":"device3","fill":"#bf337b","text":"new-constraint3"}}}]

After visiting http://127.0.0.1:12001/test3:

shows a error on device1's terminal below

prints "log1" on device2's terminal

prints "log2" on device3's terminal

no HTTP response back to the browser

The error on device1's terminal

TypeError: JSON.parse(...).map is not a function at parse (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/flatted/cjs/index.js:24:48) at MqttClient. (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/node-red-contrib-dnr/dnr/broker.js:31:26) at emitThree (events.js:136:13) at MqttClient.emit (events.js:217:7) at MqttClient._handlePublish (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/node-red-contrib-dnr/node_modules/mqtt/lib/client.js:851:12) at MqttClient._handlePacket (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/node-red-contrib-dnr/node_modules/mqtt/lib/client.js:292:12) at process (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/node-red-contrib-dnr/node_modules/mqtt/lib/client.js:248:12) at Writable.writable._write (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/node-red-contrib-dnr/node_modules/mqtt/lib/client.js:258:5) at doWrite (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/readable-stream/lib/_stream_writable.js:428:64) at writeOrBuffer (/Users/weishi/Desktop/OneDrive/Dissertation/Project/NodeRed-subnode/node-red/node_modules/readable-stream/lib/_stream_writable.js:417:5)

Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/namgk/dnr-editor/issues/12#issuecomment-415998729, or mute the thread https://github.com/notifications/unsubscribe-auth/AA9PoKpOSa6eU6sOR857y-QkARJO0oxhks5uUcZsgaJpZM4VvYNW .

namgk / dnr-editor