Throttle / frame discarding doesn't seem to work

[Semmu]

Hey there,

First of all, thank you for these modules (this one and the decoder), they are really great! I use them to proxy my camera feeds via Node-Red so I can display and access them more easily.

I have been playing around with this and I think I found an issue. This encoder node specifically has an option to discard messages when the stream is overloaded, but it does not seem to work.

My testing setup is the following:

0. (I have a USB camera attached to my machine and I create an MJPEG stream from it using motion, and the stream is accessible at localhost:8080)

1. I created a simple flow to connect to that stream, decode it, then re-encode as MJPEG and to base64 as well.

2. I display both types of streams on a dashboard.

[{"id":"4905c520.ea9574","type":"tab","label":"MJPEG proxy","disabled":false,"info":""},{"id":"db5630e7.83cdc","type":"multipart-decoder","z":"4905c520.ea9574","name":"decode","ret":"bin","url":"http://localhost:8080","tls":"","delay":0,"maximum":"10000000","blockSize":"1","x":240,"y":480,"wires":[["6535feb.cbf33","53928960.69c7c"]]},{"id":"dfcc9a31.860948","type":"inject","z":"4905c520.ea9574","name":"start","props":[],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":89.83334350585938,"y":480.00003814697266,"wires":[["db5630e7.83cdc"]]},{"id":"fb64a032.e945b","type":"ui_template","z":"4905c520.ea9574","group":"b183ee4c.db238","name":"Display image","order":2,"width":"12","height":"8","format":"<img width=\"16\" height=\"16\" src=\"data:image/jpg;base64,{{msg.payload}}\" />\n","storeOutMessages":true,"fwdInMessages":true,"resendOnRefresh":false,"templateScope":"local","x":760,"y":480,"wires":[[]]},{"id":"6535feb.cbf33","type":"base64","z":"4905c520.ea9574","name":"Encode","action":"","property":"payload","x":580,"y":480,"wires":[["fb64a032.e945b"]]},{"id":"53928960.69c7c","type":"multipart-encoder","z":"4905c520.ea9574","name":"","statusCode":"","ignoreMessages":true,"outputOneNew":false,"outputIfSingle":false,"outputIfAll":false,"globalHeaders":{"Content-Type":"multipart/x-mixed-replace;boundary=--myboundary","Connection":"keep-alive","Expires":"Fri, 01 Jan 1990 00:00:00 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Pragma":"no-cache"},"partHeaders":{"Content-Type":"image/jpeg"},"destination":"all","highWaterMark":"1048576","x":580,"y":560,"wires":[[]]},{"id":"29aef3b0.842bf4","type":"http in","z":"4905c520.ea9574","name":"","url":"/stream","method":"get","upload":false,"swaggerDoc":"","x":410,"y":560,"wires":[["53928960.69c7c"]]},{"id":"285b6b09.8cea8c","type":"ui_text","z":"4905c520.ea9574","group":"b183ee4c.db238","order":1,"width":"0","height":"0","name":"","label":"Base64","format":"{{msg.payload}}","layout":"row-spread","x":741.7430763244629,"y":439.5833339691162,"wires":[]},{"id":"7a734c00.208d8c","type":"ui_template","z":"4905c520.ea9574","group":"b183ee4c.db238","name":"Display image","order":4,"width":"12","height":"8","format":"<img width=\"16\" height=\"16\" src=\"http://localhost:1880/stream\" />\n","storeOutMessages":true,"fwdInMessages":true,"resendOnRefresh":false,"templateScope":"local","x":761.7430763244629,"y":559.5833339691162,"wires":[[]]},{"id":"f2a94d55.591108","type":"inject","z":"4905c520.ea9574","name":"stop","props":[{"p":"stop","v":"true","vt":"bool"}],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":90,"y":520,"wires":[["db5630e7.83cdc"]]},{"id":"ca4b6caa.933348","type":"delay","z":"4905c520.ea9574","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"5","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":430,"y":840,"wires":[[]]},{"id":"35068c04.1a1454","type":"ui_text","z":"4905c520.ea9574","group":"b183ee4c.db238","order":3,"width":"0","height":"0","name":"","label":"RTSP","format":"{{msg.payload}}","layout":"row-spread","x":730,"y":520,"wires":[]},{"id":"b183ee4c.db238","type":"ui_group","z":"","name":"Default","tab":"c622d5eb.eed528","order":1,"disp":true,"width":"12","collapse":true},{"id":"c622d5eb.eed528","type":"ui_tab","z":"","name":"UI things","icon":"home","order":3,"disabled":false,"hidden":false}]

(You may need to modify the dashboard elements to assign them to existing groups on your dashboard.)

My issue is that both types of streams seem to slow down and lag behind, and the delay seems to steadily increase as the streams go on, even when the option to "ignore messages when the stream is overloaded" is checked in the encoder node.

I managed to fix the issue by inserting a rate limiter node into the flow, which seems to correctly discard frames, so the stream delay does not seem to build up.

I also played around with the memory limit, hoping it would force the system to have a smaller queue of frames, but it didn't help much either.

[{"id":"4905c520.ea9574","type":"tab","label":"MJPEG proxy","disabled":false,"info":""},{"id":"db5630e7.83cdc","type":"multipart-decoder","z":"4905c520.ea9574","name":"decode","ret":"bin","url":"http://localhost:8080","tls":"","delay":0,"maximum":"10000000","blockSize":"1","x":240,"y":480,"wires":[["ca4b6caa.933348"]]},{"id":"dfcc9a31.860948","type":"inject","z":"4905c520.ea9574","name":"start","props":[],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":89.83334350585938,"y":480.00003814697266,"wires":[["db5630e7.83cdc"]]},{"id":"fb64a032.e945b","type":"ui_template","z":"4905c520.ea9574","group":"b183ee4c.db238","name":"Display image","order":2,"width":"12","height":"8","format":"<img width=\"16\" height=\"16\" src=\"data:image/jpg;base64,{{msg.payload}}\" />\n","storeOutMessages":true,"fwdInMessages":true,"resendOnRefresh":false,"templateScope":"local","x":760,"y":480,"wires":[[]]},{"id":"6535feb.cbf33","type":"base64","z":"4905c520.ea9574","name":"Encode","action":"","property":"payload","x":580,"y":480,"wires":[["fb64a032.e945b"]]},{"id":"53928960.69c7c","type":"multipart-encoder","z":"4905c520.ea9574","name":"","statusCode":"","ignoreMessages":true,"outputOneNew":false,"outputIfSingle":false,"outputIfAll":false,"globalHeaders":{"Content-Type":"multipart/x-mixed-replace;boundary=--myboundary","Connection":"keep-alive","Expires":"Fri, 01 Jan 1990 00:00:00 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Pragma":"no-cache"},"partHeaders":{"Content-Type":"image/jpeg"},"destination":"all","highWaterMark":"1048576","x":580,"y":560,"wires":[[]]},{"id":"29aef3b0.842bf4","type":"http in","z":"4905c520.ea9574","name":"","url":"/stream","method":"get","upload":false,"swaggerDoc":"","x":410,"y":560,"wires":[["53928960.69c7c"]]},{"id":"285b6b09.8cea8c","type":"ui_text","z":"4905c520.ea9574","group":"b183ee4c.db238","order":1,"width":"0","height":"0","name":"","label":"Base64","format":"{{msg.payload}}","layout":"row-spread","x":741.7430763244629,"y":439.5833339691162,"wires":[]},{"id":"7a734c00.208d8c","type":"ui_template","z":"4905c520.ea9574","group":"b183ee4c.db238","name":"Display image","order":4,"width":"12","height":"8","format":"<img width=\"16\" height=\"16\" src=\"http://localhost:1880/stream\" />\n","storeOutMessages":true,"fwdInMessages":true,"resendOnRefresh":false,"templateScope":"local","x":761.7430763244629,"y":559.5833339691162,"wires":[[]]},{"id":"f2a94d55.591108","type":"inject","z":"4905c520.ea9574","name":"stop","props":[{"p":"stop","v":"true","vt":"bool"}],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":90,"y":520,"wires":[["db5630e7.83cdc"]]},{"id":"ca4b6caa.933348","type":"delay","z":"4905c520.ea9574","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"5","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":410,"y":480,"wires":[["6535feb.cbf33","53928960.69c7c"]]},{"id":"35068c04.1a1454","type":"ui_text","z":"4905c520.ea9574","group":"b183ee4c.db238","order":3,"width":"0","height":"0","name":"","label":"RTSP","format":"{{msg.payload}}","layout":"row-spread","x":730,"y":520,"wires":[]},{"id":"b183ee4c.db238","type":"ui_group","z":"","name":"Default","tab":"c622d5eb.eed528","order":1,"disp":true,"width":"12","collapse":true},{"id":"c622d5eb.eed528","type":"ui_tab","z":"","name":"UI things","icon":"home","order":3,"disabled":false,"hidden":false}]

So is this a bug in the node itself or am I missing something? What could be causing the queue to build up and why is the whole flow blocking and queing the frames?

Thanks in advance!

[bartbutenaers]

Hi Laszlo,

This must be the best documented issue that I have ever received! Thanks for taking your time to explain it!

Is there any reason why you show both streams at the same time? I have created this node at the time being because I found no way to get the base64 stream working:

1. Start sending messages (containing images) to the template node
2. The template node will push those images via the dashboard's (socketio) websocket channel to the dashboard.
3. The template node frontend side will display the images

But for some reason the dashboard becomes very slow, even for only few images per second. I have always thought that the websocket channel (socketio library) couldn't handle this amount of data. So I developed the multipart-encoder node to solve that problem: let the dashboard fetch images, instead of pushing the images toward the dashboard.

However recently Kevin Godell has joined the Node-RED community, after I had asked him some questions about video and ffmpeg. He is really very experienced with video. And he uses socketio (outside of Node-RED!) without problems, as you can read in our discussion:

So there must be something wrong with the integration of socketio in the Node-RED dashboard. But I have not enough free time to investigate that...

Anyway: you should really remove that base64 stream and only use the multipart-encoder node. Hopefully it works better then... Please let me know the results!

P.S. If you are interested in video in Node-RED you should follow Kevin Godell on the Node-RED forum. I had created an experimental hls player to get him started with Node-RED development, and now he has created already some quite amazing nodes (as you can read about here). Still in beta phase, but the testers are VERY enthousiastic. So nice stuff is coming in the near future ...

Bart

[Semmu]

I display both types of streams because I want to measure which one is faster and also which causes a smaller system load. I guess the MJPEG stream should be less resource-intensive (on the server-side, that is), but interestingly enough the base64 stream seems to be faster, i.e. it builds up the lag slower and in general it is more responsive, closer to real-time. (Although without throttling it builds up seconds of delay as well after a short period of time.)

One more advantage of the base64 stream seems to be that it is more responsive, e.g. if I stop the stream and then restart it, because of that solution pushing the changes to the dashboard, I don't have to refresh the browser to see the newly started stream. And this is a very important feature if I want to display the streams on a machine I don't wan't to touch regularly.

Currently, I am testing these settings while being 250kms away from the source machine, as I rent a flat in a different city and I use SSH to port-forward Node-RED to my local machine temporarily.

I think I will stick with the base64-based solution and I will see how my server can handle the load of multiple cameras, but I thought I would report this throttling issue I found, as my solution/workaround may be useful to others as well.