bartbutenaers / node-red-contrib-multipart-stream-encoder

Node Red node for encoding multipart streams over http
Apache License 2.0
9 stars 1 forks source link

node-red-contrib-multipart-stream-encoder

Node-Red node for encoding multipart streams over http

Note about version 0.0.2: Thanks to Simon Hailes, who has been testing version 0.0.1 thoroughly. As a result, a number of new features have been added to increase the interaction between this node and the other nodes in the flow.

Install

Run the following npm command in your Node-RED user directory (typically ~/.node-red):

npm install node-red-contrib-multipart-stream-encoder

Usage

The goal is to setup a stream over http, to create a continous sequence of data (text, images, ...). One of the most known examples is an MJPEG stream, to send continously JPEG images (like a video stream).

The encoder converts (payloads from) separate messages into a continuous stream. For example to convert images into a continous MJPEG stream, so your Node-Red flow behaves like an IP camera that can offer live video:

Stream encoder

This node will work closely together with the HttpIn node, as can be seen in the next flow:

Basic flow

[{"id":"561b7ff7.7caf3","type":"http in","z":"15244fe6.9ae87","name":"","url":"/xxxx","method":"get","upload":false,"swaggerDoc":"","x":1120,"y":260,"wires":[["cc1feca1.b909b"]]},{"id":"cc1feca1.b909b","type":"multipart-encoder","z":"15244fe6.9ae87","name":"","statusCode":"","ignoreMessages":true,"outputIfSingle":true,"outputIfAll":false,"globalHeaders":{"Content-Type":"multipart/x-mixed-replace;boundary=--myboundary","Connection":"keep-alive","Expires":"Fri, 01 Jan 1990 00:00:00 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Pragma":"no-cache"},"partHeaders":{"Content-Type":"image/jpeg"},"destination":"all","x":1300,"y":200,"wires":[[]]},{"id":"bd955688.623878","type":"http request","z":"15244fe6.9ae87","name":"HttpRequest to get image","method":"GET","ret":"bin","url":"","tls":"","x":1070,"y":200,"wires":[["cc1feca1.b909b"]]},{"id":"da39d7.4e70f628","type":"function","z":"15244fe6.9ae87","name":"Next image url","func":"var counter = global.get(\"image_counter\") || 0; \ncounter++;\nglobal.set(\"image_counter\",counter);\n\nmsg.url = 'https://dummyimage.com/400x200/fff/000&text=PNG+' + counter;\n\nreturn msg;","outputs":1,"noerr":0,"x":845,"y":200,"wires":[["bd955688.623878"]]},{"id":"435929ff.b35c18","type":"inject","z":"15244fe6.9ae87","name":"Every second","topic":"","payload":"","payloadType":"date","repeat":"1","crontab":"","once":false,"x":637.9999885559082,"y":199.99999904632568,"wires":[["da39d7.4e70f628"]]}]

The above flow captures images (e.g. from an IP camera, from disc, ...) and encodes those images into a live stream. When a browser is used to navigate to the URL specified in the HttpIn node, the HttpIn node will pass this request to the encoder node. As a result, the encoder node will send the video stream to your browser. Multiple browser windows can be opened simultaneously, to display the same stream multiple times. For this simple test Chrome is being used, because some other browsers require the stream to be called from within an \<img> or \<video> tag (which means it cannot simply display the stream, when a stream URL is entered).

We will use MJPEG streams in the remainder of this page, however it is also possible to stream other data types.

Controlling a stream

The streams can be controlled in multiple ways:

The following flow offers buttons to pause/resume/stop a stream:

Control stream

[{"id":"561b7ff7.7caf3","type":"http in","z":"15244fe6.9ae87","name":"","url":"/xxxx","method":"get","upload":false,"swaggerDoc":"","x":1660,"y":240,"wires":[["cc1feca1.b909b"]]},{"id":"cc1feca1.b909b","type":"multipart-encoder","z":"15244fe6.9ae87","name":"","statusCode":"","ignoreMessages":true,"outputIfSingle":true,"outputIfAll":false,"globalHeaders":{"Content-Type":"multipart/x-mixed-replace;boundary=--myboundary","Connection":"keep-alive","Expires":"Fri, 01 Jan 1990 00:00:00 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Pragma":"no-cache"},"partHeaders":{"Content-Type":"image/jpeg"},"destination":"all","x":1840,"y":300,"wires":[[]]},{"id":"bd955688.623878","type":"http request","z":"15244fe6.9ae87","name":"HttpRequest to get image","method":"GET","ret":"bin","url":"","tls":"","x":1610,"y":300,"wires":[["cc1feca1.b909b"]]},{"id":"da39d7.4e70f628","type":"function","z":"15244fe6.9ae87","name":"Next image url","func":"var counter = global.get(\"image_counter\") || 0; \ncounter++;\nglobal.set(\"image_counter\",counter);\n\nmsg.url = 'https://dummyimage.com/400x200/fff/000&text=PNG+' + counter;\n\nreturn msg;","outputs":1,"noerr":0,"x":1380,"y":300,"wires":[["bd955688.623878"]]},{"id":"435929ff.b35c18","type":"inject","z":"15244fe6.9ae87","name":"Every second","topic":"","payload":"","payloadType":"date","repeat":"1","crontab":"","once":false,"x":1037.9999885559082,"y":299.9999990463257,"wires":[["58d2c398.976efc"]]},{"id":"5a5de18d.2ef8a","type":"inject","z":"15244fe6.9ae87","name":"Stop stream","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":"","x":1450,"y":360,"wires":[["bca25c2b.4079f"]]},{"id":"bca25c2b.4079f","type":"change","z":"15244fe6.9ae87","name":"","rules":[{"t":"set","p":"stop","pt":"msg","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":1650,"y":360,"wires":[["cc1feca1.b909b"]]},{"id":"a39192ce.80e4b","type":"inject","z":"15244fe6.9ae87","name":"Pause stream","topic":"","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":"","x":1450,"y":400,"wires":[["d1bced7a.5790c"]]},{"id":"d84c646e.088fd8","type":"inject","z":"15244fe6.9ae87","name":"Resume stream","topic":"","payload":"false","payloadType":"bool","repeat":"","crontab":"","once":true,"onceDelay":"","x":1460,"y":440,"wires":[["d1bced7a.5790c"]]},{"id":"58d2c398.976efc","type":"switch","z":"15244fe6.9ae87","name":"","property":"streamPaused","propertyType":"flow","rules":[{"t":"false"}],"checkall":"true","repair":false,"outputs":1,"x":1210,"y":300,"wires":[["da39d7.4e70f628"]]},{"id":"d1bced7a.5790c","type":"change","z":"15244fe6.9ae87","name":"","rules":[{"t":"set","p":"streamPaused","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1680,"y":400,"wires":[[]]}]

Same stream to 'all clients'

Watching camera images from a Node-Red flow, is an example of an infinite stream. The flow generates an endless stream of data, and all clients hook in to the same existing stream: clients are not interested in the previous old data (they only want to see the current camera images).

Stream all

[{"id":"7f2b6b5c.691b64","type":"http in","z":"15244fe6.9ae87","name":"","url":"/infinite","method":"get","upload":false,"swaggerDoc":"","x":570,"y":120,"wires":[["5b3e4039.df33b"]]},{"id":"5b3e4039.df33b","type":"multipart-encoder","z":"15244fe6.9ae87","name":"","statusCode":"","ignoreHeaders":false,"ignoreMessages":true,"outputIfSingle":true,"outputIfAll":true,"globalHeaders":{"Content-Type":"multipart/x-mixed-replace;boundary=--myboundary","Connection":"keep-alive","Expires":"Fri, 01 Jan 1990 00:00:00 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Pragma":"no-cache"},"partHeaders":{"Content-Type":"image/jpeg"},"destination":"all","x":781,"y":120,"wires":[["ca54ad09.d6904"]]},{"id":"fb72a642.df0eb8","type":"http request","z":"15244fe6.9ae87","name":"Get image by url","method":"GET","ret":"bin","url":"","tls":"","x":560,"y":180,"wires":[["5b3e4039.df33b"]]},{"id":"f8b8895a.77a838","type":"interval","z":"15244fe6.9ae87","name":"Every second","interval":"1","onstart":false,"msg":"ping","showstatus":false,"unit":"seconds","statusformat":"YYYY-MM-D HH:mm:ss","x":150,"y":180,"wires":[["59a9f54a.322d2c"]]},{"id":"59a9f54a.322d2c","type":"function","z":"15244fe6.9ae87","name":"Next image url","func":"var counter = global.get(\"image_counter\") || 0; \ncounter++;\nglobal.set(\"image_counter\",counter);\n\nmsg.url = 'https://dummyimage.com/400x200/fff/000&text=PNG+' + counter;\n\nreturn msg;","outputs":1,"noerr":0,"x":340,"y":180,"wires":[["fb72a642.df0eb8"]]},{"id":"3ffa9ebc.86a872","type":"comment","z":"15244fe6.9ae87","name":"Stream to all clients","info":"","x":170,"y":120,"wires":[]},{"id":"ca54ad09.d6904","type":"debug","z":"15244fe6.9ae87","name":"Encoder output","active":true,"console":"false","complete":"res","x":954,"y":120,"wires":[]}]

The clients are connecting at different times, but they all receive the same data at a particular moment in time:

Stream all result

Make sure that messages containing data (images), don't have a msg.res field ! When the message contains a msg.res field, the encoder assumes the message is sended by the HttpIn node. And the msg.payload from those (HttpIn) messages won't be streamed, because it will contain all kind of information about the http request (instead of a real image). Otherwise the stream would become corrupt...

Separate stream for every 'Single client'

This option has been added, based on the feedback of Nick O’Leary.

In some cases each client wants to have its own individual stream. E.g. when camera images have been stored on disc, and that recorded video footage needs to be replayed afterwards in the dashboard. In that case every client wants to see the entire video from the start: each client wants to get the entire stream from the beginning, independent of other clients.

This means that the image processing needs to be started (by the flow), as soon as a new client request arrives.

Stream single

[{"id":"bc87df5c.bc323","type":"http in","z":"15244fe6.9ae87","name":"","url":"/finite","method":"get","upload":false,"swaggerDoc":"","x":160,"y":660,"wires":[["218fb2a5.4ba8ae"]]},{"id":"bff8aa99.c07c98","type":"http request","z":"15244fe6.9ae87","name":"Get image by url","method":"GET","ret":"bin","url":"","tls":"","x":518,"y":660,"wires":[["4dc14794.f344e8"]]},{"id":"422deea9.f5539","type":"comment","z":"15244fe6.9ae87","name":"Stream per client","info":"","x":180,"y":620,"wires":[]},{"id":"4dc14794.f344e8","type":"multipart-encoder","z":"15244fe6.9ae87","name":"","statusCode":"","ignoreHeaders":true,"ignoreMessages":true,"outputIfSingle":true,"outputIfAll":false,"globalHeaders":{"Content-Type":"multipart/x-mixed-replace;boundary=--myboundary","Connection":"keep-alive","Expires":"Fri, 01 Jan 1990 00:00:00 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Pragma":"no-cache"},"partHeaders":{"Content-Type":"image/jpeg"},"destination":"single","x":713,"y":660,"wires":[["2216a759.8dc718"]]},{"id":"2216a759.8dc718","type":"debug","z":"15244fe6.9ae87","name":"Encoder output","active":true,"console":"false","complete":"true","x":899,"y":660,"wires":[]},{"id":"218fb2a5.4ba8ae","type":"function","z":"15244fe6.9ae87","name":"Msg factory","func":"// Repeat the msg every second\nvar repeatInterval = 1000;\n\ncontext.set('counter', 0);\n\nvar interval = setInterval(function() {\n    var counter = context.get('counter') || 0;\n    counter = counter + 1;\n    context.set('counter', counter);\n    \n    msg.url = 'https://dummyimage.com/400x200/fff/000&text=PNG+' + counter;\n    \n    if(counter >= 15) {\n        msg.stop = true;\n        clearInterval(interval);\n    }\n    \n\tnode.send(msg);\n}, repeatInterval); \n\nreturn null;","outputs":1,"noerr":0,"x":328,"y":660,"wires":[["bff8aa99.c07c98"]]}]

The clients are connecting at different times, but they all get the entire stream from the start:

Stream single result

Make sure that all messages have msg.res field (containing the HttpIn node's response)! Reason is that the encoder should know to which client the image (in the msg.payload) needs to be send. This can be accomplished by passing the response object from the HttpIn node.

Keeping track of active connections

When no connections are currently active, this means that no clients are requesting data (i.e. no active client sessions). In that case the encoder node will ignore all data from input messages. However it is adviced to stop sending message/data to the encoder, since that data won't be used anyway.

However it is not easy to keep track of active requests from within a Node-Red flow, since the ExpressJs webserver closes the client connections without notifying the Node-Red flow.

To solve this, the encoder node can generate following output message types (if specified in the node's config screen):

In all these output messages, the msg.payload field contains the number of current connections.

By keeping track of the active connections, the flow can decide whether is it useful to process data (images). Make sure you don't execute useless processing in the flow:

Adaptive streaming

When we are sending lots of messages to the encoder node (containing lots of data in the payloads), we would run into troubles if the stream cannot handle all that data. For example if we have a slow network or a slow client system. In that case the stream buffer would start growing until all memory would be consumed. At the end our system would stop functioning correctly.

This can be solved by enabling the 'Ignore messages if stream is overloaded' checkbox on the config screen. When the stream cannot process all messages, the encoder node will start ignoring input messages. As soon as the stream is again ready to process new data, the encoder node will again start processing input messages. For example for MJPEG streaming, it is better to send less images than to have a malfunctioning system...

This option can be disabled if there is a short burst of images, and all of those images should be sended. But make sure that you don't overload your system.

By default the stream buffer size is 16 Kbyte in NodeJs, and is called the high water mark. This default memory limit can be changed in the encoder's config screen. E.g. when working with images, the 16 Kbyte would be exceeded all the time (since a single image will already exceed 16 Kbyte).

Node status

The node status displays the number of active client connections:

Stream tabsheets

The status text can have different colors:

Streaming basics

Sometimes data need to be received at high rates. For example get N camera images per second, to be able to display fluent video.

Such high data rates cannot be reached by sending a request for every image:

  1. Request image 1
  2. Wait for response image 1
  3. Request image 2
  4. Wait for response image 2
  5. ...

Indeed this would result in too much overhead: we would have to wait all the time. Moreover some devices cannot handle the overflow of http requests, ...

In this case (http) streaming is preferred. We send a single (http) request, and the response will be an (in)finite stream of images. A boundary string will be used as a separator between the images:

Remark: this has nothing to do with mp4 streaming. In a MJPEG stream each image is compressed (as jpeg), but an mp4 stream also compresses the entire stream (by only sending differences between the images).

Http headers

At the start of the stream, global http headers will be send. Afterwards each part of the multipart stream will contain part http headers, followed by the real data (e.g. an image).

To make sure the client understands that a multipart stream has been setup, both type of headers need some minimal entries:

By default, all required http headers (on both levels) will be available in the config screen. You can always replace them or add new ones, but make sure you don't remove mandatory headers (or the stream will fail)!!

None of both headers can be specified via the msg.headers field!

Comparison to alternative solutions

To explain why this encoder node might be handy, let's go through all the available solutions to display video streams in your dashboard:

Dashboard gets MJPEG stream directly from IP camera

In this case the images don't pass through the Node-Red flow. This can be implemented easily with a simple flow that only contains a single Template node, which puts an \<img> element on the dashboard (with the camera URL as source):

Stream directly flow

[{"id":"4e44e10.85d262","type":"ui_template","z":"47b91ceb.38a754","group":"16a1f12d.07c69f","name":"Display image","order":1,"width":"6","height":"6","format":"<img width=\"16\" height=\"16\" src=\"http://200.36.58.250/mjpg/video.mjpg?resolution=640x480\" />\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":920,"y":1360,"wires":[[]]},{"id":"16a1f12d.07c69f","type":"ui_group","z":"","name":"Default","tab":"f136a522.adc2a8","order":1,"disp":true,"width":"6"},{"id":"f136a522.adc2a8","type":"ui_tab","z":"","name":"Home","icon":"home","order":1}]

As a result the dashboard html page will contain an \<img> element, which gets its images directly from the IP camera MJPEG stream:

Stream directly dashboard

  1. The \<img> element requests an MJPEG stream from the IP camera.
  2. The IP camera responds by sending an endless stream of images (i.e. an MJPEG stream).
  3. The browser will render the images (i.e. the images are displayed in the dashboard).

That is simple and works fine. However suppose the Node-Red flow needs to receive the images directly from the camera, do some image processing and display the manipulated images on your dashboard. For example a rectangle should be drawn around every human face. Then this solution won't be sufficient ...

Push the images to the dashboard

The flow could get the images from the camera (using a HttpRequest or MultipartStreamDecoder node), do some image processing and afterwards push the manipulated images (via websocket) to the dashboard:

Stream push websocket

[{"id":"db5630e7.83cdc","type":"multipart-decoder","z":"47b91ceb.38a754","name":"","ret":"bin","url":"http://200.36.58.250/mjpg/video.mjpg?resolution=640x480","tls":"","delay":0,"maximum":"10000000","x":590,"y":1240,"wires":[["6535feb.cbf33"]]},{"id":"dfcc9a31.860948","type":"inject","z":"47b91ceb.38a754","name":"Start stream","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":"","x":389.8333435058594,"y":1240.0000381469727,"wires":[["db5630e7.83cdc"]]},{"id":"6535feb.cbf33","type":"base64","z":"47b91ceb.38a754","name":"Encode","x":780,"y":1240,"wires":[["fb64a032.e945b"]]},{"id":"fb64a032.e945b","type":"ui_template","z":"47b91ceb.38a754","group":"16a1f12d.07c69f","name":"Display image","order":1,"width":"6","height":"6","format":"<img width=\"16\" height=\"16\" src=\"data:image/jpg;base64,{{msg.payload}}\" />\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":958.2569236755371,"y":1240.4166660308838,"wires":[[]]},{"id":"16a1f12d.07c69f","type":"ui_group","z":"","name":"Default","tab":"f136a522.adc2a8","order":1,"disp":true,"width":"6"},{"id":"f136a522.adc2a8","type":"ui_tab","z":"","name":"Home","icon":"home","order":1}]
  1. The flow requests an MJPEG stream from the IP camera.
  2. The IP camera responds by sending an endless stream of images (i.e. an MJPEG stream).
  3. The flow will do some image processing and send the message (image in msg.payload) to the template node.
  4. The template node will push the message - via a websocket channel - to the dashboard.
  5. The source of the \<img> element in the html will (constantly) being updated to refer to the latest image.
  6. The browser will render the images (i.e. the images are displayed in the dashboard).

That is again simple and works fine. However if you want to display multiple cameras (with higher frame rates) simultaneously, keep in mind that all the updated data (graphs, node statusses, debug panel messages, images ...) will be pushed through a single websocket channel to the dashboard! When too much data is being pushed through that single websocket channel, the browser will start freezing. Then this solution won't be sufficient ...

Dashboard gets MJPEG stream via the Node-Red flow

The flow could get the images from the camera (using a HttpRequest or MultipartStreamDecoder node), do some image processing and afterwards the dashboard will request the manipulated images stream from the flow.

This means that the dashboard should be setup with a single template node, similar to the first alternative. However the image needs to be requested from the Node-Red flow (instead of from the IP camera), so the Node-Red flow IP address should be specified:

Stream via NR dashboard

The encoder node allows us to create a Node-Red flow that behaves like an IP camera, which means you can 'request' a live video stream from your flow:*

Stream via NR flow

  1. The flow requests an MJPEG stream from the IP camera.
  2. The IP camera responds by sending an endless stream of images (i.e. an MJPEG stream).
  3. The flow will do some image processing and send the message (image in msg.payload) to the encoder node.
  4. The \<img> element in the html will request an MJPEG stream from the flow. The ExpressJs webserver will send the request to the HttpIn node, since the URL contains '/videostream'.
  5. The HttpIn node forwards the request (wrapped in a message) to the encoder node.
  6. The encoder node creates an MJPEG stream from all the images, and sends that stream to the dashboard.
  7. The browser will render the images (i.e. the images are displayed in the dashboard).

Remark: The node-red-contrib-multipart-stream-decoder is used to decode the MJPEG stream from the IP camera, and convert it to separate images.

Request/Response objects (Advanced)

As said before, the encoder node works closely together with the HttpIn node. The communication between the HttpIn node and the encoder node will be setup in a number of steps:

  1. The ExpressJs will create two related objects for each http request: a Request object and a Response object.
  2. The HttpIn node will create an output message (request in msg.payload and responsein msg.res).
  3. The encoder node stores the response object for using it later on.
  4. The encoder node will send images (that arrive on its input) to those response objects.

This way a single request will result in an endless response, i.e. a multipart stream ...