Jacques44 / node-red-contrib-bigfile

node-red contribution. This node is for managing big files in a blue way !
Apache License 2.0
2 stars 3 forks source link

streaming control #7

Open berry4u opened 5 years ago

berry4u commented 5 years ago

Hi I'm trying to use this node to read big files, passing them to an http streamer (as blocks). Is there a way to manage the flow control in order to wait the reception confirmation of a block to proceed in reading the next one.

Thanks

Jacques44 commented 5 years ago

Hello,

here is a solution:

every node I know including node-red-contrib-traffic I’ve made or others use memory to store flow messages. This may be not acceptable for very very big files.

The solution here relies on a function which stacks messages en send them only if it receives a « go » control message

You just have to fill your filename in this flow in the node named « filename"

[{"id":"e00a8e65.41d33","type":"bigfile","z":"81e867ca.e36108","name":"","filename":"","nopayload":false,"flow":"blocks","highWaterMark":16,"encoding":"utf8","format":"utf8","keepEmptyLines":false,"x":338,"y":227,"wires":[["35e3993.af63466"],[]]},{"id":"de189b80.166b98","type":"inject","z":"81e867ca.e36108","name":"filename","topic":"","payload":"fill here your filename","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":226,"wires":[["e00a8e65.41d33"]]},{"id":"62ef45a4.78087c","type":"debug","z":"81e867ca.e36108","name":"","active":true,"console":"false","complete":"false","x":1030,"y":221,"wires":[]},{"id":"372697ee.a46be8","type":"function","z":"81e867ca.e36108","name":"next","func":"msg.payload = \"go\";\nmsg.control = true;\nreturn msg;\n","outputs":1,"noerr":0,"x":574,"y":121,"wires":[["35e3993.af63466"]]},{"id":"d106356f.8d6008","type":"inject","z":"81e867ca.e36108","name":"","topic":"","payload":"next","payloadType":"str","repeat":"","crontab":"","once":false,"x":283,"y":122,"wires":[["372697ee.a46be8"]]},{"id":"35e3993.af63466","type":"function","z":"81e867ca.e36108","name":"control","func":"// Get stack from context\nvar stack = flow.get('stat');\nstack = stack || [];\n\n// If data message, store it in stack\nif (!msg.control) stack.push(msg);\n\n// If \"next\" control message\nif (msg.control && msg.payload === \"go\") {\n msg = stack.pop();\n} else {\n msg = undefined;\n}\n\n// Store stack in node context\nflow.set('stat', stack);\n\n// Node status\nif (stack.length) {\n node.status({fill:\"blue\",shape:\"dot\",text:\"has \" + stack.length + \" messages\"});\n} else {\n node.status({fill:\"green\",shape:\"dot\",text:\"has not any messages\"});\n}\n\nif (msg) return msg;\n","outputs":1,"noerr":0,"x":734.5,"y":219,"wires":[["62ef45a4.78087c"]]},{"id":"4882fe.18c85d04","type":"inject","z":"81e867ca.e36108","name":"","topic":"","payload":"clear","payloadType":"str","repeat":"","crontab":"","once":false,"x":281,"y":66,"wires":[["f43a9c08.bbffe"]]},{"id":"f43a9c08.bbffe","type":"function","z":"81e867ca.e36108","name":"clear","func":"flow.set('stat', []);\nreturn msg;","outputs":1,"noerr":0,"x":425,"y":66,"wires":[["372697ee.a46be8"]]}]

Le 25 août 2019 à 17:13, berry4u notifications@github.com a écrit :

Hi I'm trying to use this node to read big files, passing them to an http streamer (as blocks). Is there a way to manage the flow control in order to wait the reception confirmation of a block to proceed in reading the next one.

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Jacques44/node-red-contrib-bigfile/issues/7?email_source=notifications&email_token=AEKS6M5ETKWYMIBBSDFSUHDQGKOQ7A5CNFSM4IPJK322YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HHHUFTQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AEKS6M5KWR2MPKC2DNYGW3TQGKOQ7ANCNFSM4IPJK32Q.

berry4u commented 5 years ago

Thank you for answering. Yes, your hint works fine and as you pointed out it is suitable for big but not huge files. For those, I'm wondering if it's feasible to build a loop with "big file reader node" by setting the start and end reading positions in order to load only a couple of blocks per time in memory.

Jacques44 commented 5 years ago

Hello,

I thought of offering a way to flow blocks through the big file node. In fact it relies ont .fs.createReadStream which streams block and I’ve no control on it.

Le 26 août 2019 à 19:58, berry4u notifications@github.com a écrit :

Thank you for answering. Yes, your hint works fine and as you pointed out it is suitable for big but not huge files. For those, I'm wondering if it's feasible to build a loop with "big file reader node" by setting the start and end reading positions in order to load only a couple of blocks per time in memory.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Jacques44/node-red-contrib-bigfile/issues/7?email_source=notifications&email_token=AEKS6M4UIDBC2WKKGP5WRWLQGQKVVA5CNFSM4IPJK322YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FEUEY#issuecomment-524962323, or mute the thread https://github.com/notifications/unsubscribe-auth/AEKS6M6N6XYTC3P6DBWN7A3QGQKVVANCNFSM4IPJK32Q.