bartbutenaers / node-red-contrib-http-proxy

A Node-Red node to use Node-RED as a http proxy server, for redirecting http(s) requests
Apache License 2.0
8 stars 3 forks source link

node-red-contrib-http-proxy

A Node-Red node to use Node-RED as a http proxy server, for redirecting http(s) requests

Install

Run the following npm command in your Node-RED user directory (typically ~/.node-red):

npm install node-red-contrib-http-proxy

Introduction

This is not a full blown http proxy, but a lightweight one which I developed for personal use. The node-http-proxy library is used under the cover, which contains some extra funtionality. Pull requests are always welcome, to implement these extra features into this node! When you need an advanced http proxy, you can also use a third party free http proxy (e.g. Nginx) ...

This node can be used for all kind of http(s) request, independent whether the http response is finite or infinite (as explained below). But since I developed it to get an mjpeg stream from my IP camera, most examples will be about IP cameras ...

What is a http proxy?

A http proxy is a server that is put between a client and one or more other servers. The http proxy intercepts all http(s) requests from that client, and decides to which server the request needs to be forwarded. As soon as the server has responded, the http proxy will return the response to the client. For the client it appears as if the http proxy itself is the origin of the response, i.e. the client is not aware that his request has been forwarded to other servers ...

httproxy

Why using a http proxy?

There are a number of advantages for adding a http proxy in between:

The solution without proxy

It is very easy to show camera images/streams in the dashboard, by getting the data directly from the the target (e.g. an IP camera):

hacker

It is very easy to create such a dashboard. For example using a dashboard template node with an <img> or <video> tag, whose source (src) attribute referring directly to your IP camera URL:

<img src="https://<ip cam hostname>/<some path>">

Although such a dashboard can be created very easily, it has a number of disadvantages:

As a result, this kind of setup needs to be AVOIDED!!!

The standard Node-RED solution

Node-RED provides some nodes out-of-the-box, which can be used to turn your flow into a http proxy:

standard solution

  1. The client (e.g. dashboard) sends a http request to the http-in node, e.g. by pointing the <img> or <video> tag element's source (src) to your Node-RED flow (instead of directly to e.g. your IP camera):

    <img src="https://<node-red hostname>:1880/<some path>">
  2. A message is send to the http-request node.

  3. The http-request node sends a new http request to the real target (e.g. your IP camera), to get the data (e.g. image).

  4. The target returns the requested data.

  5. A message - containing the data - will be send to the http-out node.

  6. The http-out node returns the data (e.g. image) to the client.

This setup works fine. And it is secure, since the credentials (username/password) can be stored safely inside Node-RED and used by the http-request node.

However it works only for finite http responses, i.e. responses with data of a fixed length (e.g. a single image). But this solution cannot be used for responses with infinite length (e.g. an mjpeg stream) because:

I first tried to add (infinite) streaming functionality to the http-request and http-out nodes. But meanwhile it became clear that creating a dedicated http-proxy node was a much better solution ...

The decoding & encoding solution

Specific for infinite multipart http streams (like mjpeg) stream, you could solve it using my node-red-contrib-multipart-stream-decoder and node-red-contrib-multipart-stream-encoder nodes:

encode_decode

  1. The client (e.g. dashboard) sends a request to the Node-RED flow, instead of directly to the IP camera.
  2. The http-in node captures the request, and sends a message to the multipart-decoder node.
  3. The multipart-decoder sends a new request to the IP camera.
  4. The IP camera will return an infinite stream of data chunks.
  5. The multipart-decoder node will collect the data chunks that arrive, and detect images in those chunks. Each output message will contain a single image!
  6. The multipart-encoder node will setup an mjpeg stream, and it will write the arriving images into that stream. So the dashboard will decode the mjpeg stream again, and show the images ...

This works again fine and is secure, since the multipart-decoder node can use credentials that are safely stored inside Node-RED. And it is very usefull when you want to do some image processing! Indeed the images can be processed (e.g. face detection) before sending them to the client.

However decoding and encoding will use a lot of system resources (both memory and cpu), which is rather useless if you don't want to do any image processing. If you just want to forward the original data chunks to the client, then you will need the http-proxy solution ...

The http-proxy solution

Since the standard Node-RED setup isn't able to deal with infinite http responses (see previous paragraph), I decided to develop this node. It can be used as a safe way to get infinite response (e.g. mjpeg stream from an IP camera):

safe

  1. Create a dashboard template node, containing an image that will get its data from Node-RED (instead of directly from your IP camera):
    <img src="https://<node-red hostname>:1880/<some path>">
  2. Via a http-in node, this request will be captured and send to the http-proxy node.
  3. The http-proxy node will add your credentials (which are stored securely in Node-RED) and forward/redirect your request to your IP camera.
  4. The client will get the camera image and display it (e.g. via the <img> element).

Node Usage

The following example flow explains how this node works closely together with Node-RED's httpin node. Instead of navigating directly to some public url (in this case an mjpeg camera stream from https://webcam1.lpl.org/axis-cgi/mjpg/video.cgi), we will navigate to our Node-RED flow and Node-RED will forward the request to that target:

Stream example

   [{"id":"cf75b05d.199df","type":"http in","z":"8bb35f74.82618","name":"","url":"/mjpeg_test","method":"get","upload":false,"swaggerDoc":"","x":600,"y":480,"wires":[["fc540b94.6dd968"]]},{"id":"fc540b94.6dd968","type":"http-proxy","z":"8bb35f74.82618","name":"","url":"https://webcam1.lpl.org/axis-cgi/mjpg/video.cgi","events":[],"headers":{},"proxy":"","restart":false,"timeout":1,"x":820,"y":480,"wires":[]}]

This is what happens:

  1. Capture all requests for http(s)://:1880/mjpeg_test via the http-in node.
  2. Forward all requests from the httpin node to the target host (https://webcam1.lpl.org/axis-cgi/mjpg/video.cgi).
  3. The response from the target host will be passed back to your dashboard.

This way, it looks like Node-RED is providing your mjpeg camera stream...

http_proxy

Node configuration

It is easy to simulate the effect of every individual setting in the config screen, by using following test flow:

test flow

[{"id":"cf75b05d.199df","type":"http in","z":"8bb35f74.82618","name":"","url":"/show_request","method":"get","upload":false,"swaggerDoc":"","x":430,"y":480,"wires":[["8f1c4103.5481e"]]},{"id":"8f1c4103.5481e","type":"http-proxy","z":"8bb35f74.82618","name":"","url":"https://httpbin.org/get","incomingTimeout":0,"outgoingTimeout":0,"changeOrigin":false,"preserveHeaderKeyCase":false,"verifyCertificates":false,"followRedirects":false,"toProxy":false,"xfwd":false,"x":660,"y":480,"wires":[]}]
  1. Change a single setting in the node's config screen.
  2. Navigate with your browser to <http or https>://<node-red hostname>:1880/show_request.
  3. The target host (in this case httpbin will return your http request.
  4. The request (that has been send to the target host) will be displayed in the browser. For example:

    {
     "args": {}, 
     "headers": {
       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
       "Accept-Encoding": "gzip, deflate, br", 
       "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
       "Cache-Control": "max-age=0", 
       "Cookie": "io=wgWEKuHzooHPp2eDAAAA", 
       "Dnt": "1", 
       "Host": "<ip address from your original http request>", 
       "Upgrade-Insecure-Requests": "1", 
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36 Avast/75.0.1447.80"
     }, 
     "origin": "84.195.139.40, 84.195.139.40", 
     "url": "https://<ip address from your original http request>/get"
    }

    Since the httpbin is an external endpoint, my WAN IP address will be inside the "origin" field ...

URL

This URL refers to the target host system, to which the http(s) requests need to be forwarded.

When the URL is not specified in the config screen, it needs to be specified in the input message as msg.url:

msg.url

[{"id":"28c11aa6.a36ac6","type":"http in","z":"8bb35f74.82618","name":"","url":"/url_test","method":"get","upload":false,"swaggerDoc":"","x":470,"y":1020,"wires":[["f1d3e8e3.21c4e8"]]},{"id":"f1d3e8e3.21c4e8","type":"change","z":"8bb35f74.82618","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"https://webcam1.lpl.org/axis-cgi/mjpg/video.cgi","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":670,"y":1020,"wires":[["471f0190.38cac"]]},{"id":"471f0190.38cac","type":"http-proxy","z":"8bb35f74.82618","name":"Http proxy without URL","url":"","incomingTimeout":0,"outgoingTimeout":0,"changeOrigin":false,"preserveHeaderKeyCase":false,"verifyCertificates":false,"followRedirects":false,"toProxy":false,"xfwd":false,"x":920,"y":1020,"wires":[]}]

Timeout in

The timeout (in milliseconds) for the incoming connection. A timeout of 0 means no timeout, i.e. the proxy will keep waiting.

Timeout out

The timeout (in milliseconds) for the outgoing connection. A timeout of 0 means no timeout, i.e. the proxy will keep waiting.

Add authentication credentials

When this option is selected, the basic authentication credentials can be entered:

credentials

As a result, an 'Authorization' http header will be added to the target request. This header field will contain the literal "Basic " followed by the (base64 encoded) username and password:

{
  "args": {}, 
  "headers": {
    ... 
    "Authorization": "Basic bXlfdXNlcl9uYW1lOm15X3Bhc3N3b3Jk", 
    ...
}

Change origin of the host header to the target URL

The target server should use the same hostname as the http-proxy (when available to the public web), otherwise the hostname/ipaddress of the target server might become visible to the enduser. For example when somebody requests a non-existing page, the target server will return a error page containing the real ip address of the resource:

image

This option is deselected by default, which means both 'host' and 'url' contain the hostname/ipaddress of the original http request (i.e. the IP address of your Node-RED host):

{
  "args": {}, 
  "headers": {
    ... 
    "Host": "<ip address from your original http request>", 
    ...
  }, 
  ...
  "url": "https://<ip address from your original http request>/get"
}

When this option is selected, both 'host' and 'url' will contain the hostname/ipaddress of the target host:

{
  "args": {}, 
  "headers": {
    ...
    "Host": "httpbin.org", 
    ...
  }, 
  ...
  "url": "https://httpbin.org/get"
}

Remark: The 'host' header variable tells the webserver which virtual host to use. Activate this option for name-based virtual hosted sites, which use multiple host names for the same IP address. When one of those hostnames is included in the http request, the target server (which hosts multiple sites) will be able to respond the corresponding content.

Keep letter case of response header key

If not selected, the keys of the http header variables will be converted to lowercase.

Verify SSL certificates

Enable validation of the SSL certificate chain that will be received from the target host.

Caution: Normally this option won't be selected, because self-signed certificates will be rejected by this test! In that case you would receive an UNABLE_TO_VERIFY_LEAF_SIGNATURE error ...

Follow redirects

When selected, the request will be redirected in case the response has status 3xx. A 'location' header is required, otherwise no redirection is possible...

Pass the absolute URL as path (proxying to proxy)

Normally the http request 'path' contains the relative path from the URL. So it does not include any query/URL parameters, in contradiction to the http request 'uri' field ( which contains the full absolute URL).

E.g. when "url" is "http://domain.com/foo/bar", then the proxy will fill the http request "path" with "foo/bar"

When this option is selected, the absolute URL (part of the 'url' field) will be used as the path.

E.g. when "url" is "http://domain.com/foo/bar", then the proxy will fill the http request "path" with "/http://domain.com/foo/bar"

This is useful for proxying to proxies.

Add X-FORWARD headers

Some target hosts might require an XFH header to be send in the request, to determine which host originally has send the request.

{
  "args": {}, 
  "headers": {
    ...
    "X-Forwarded-Host": "<ip address from original request>:1880",
    ...
  }, 
}

Indeed when a request is forwarded by one or more proxy servers, the target host can only see (via the 'origin') the last server hostname/ipaddress. The XFH header contains a (comma separated list) of all hostnames/ipaddresses, which allows the target host to determine the route that the request has traversed: ", <proxy 1 ip address>, <proxy 2 ip address> ..."

Remark: always be aware that the content of this field might be incomplete or incorrect ...

Error handling

In this example flow, the http-proxy node doesn't have an URL:

error flow

[{"id":"10e09af3.174bf5","type":"http in","z":"8bb35f74.82618","name":"","url":"/no_url_test","method":"get","upload":false,"swaggerDoc":"","x":560,"y":1160,"wires":[["2c4712b9.794fee"]]},{"id":"95353a94.3e3ac8","type":"catch","z":"8bb35f74.82618","name":"Catch http proxy errors","scope":["471f0190.38cac"],"x":530,"y":1220,"wires":[["1bab7886.288597"]]},{"id":"1bab7886.288597","type":"debug","z":"8bb35f74.82618","name":"Http proxy errors","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":800,"y":1220,"wires":[]},{"id":"2c4712b9.794fee","type":"http-proxy","z":"8bb35f74.82618","name":"Http proxy without URL","url":"","incomingTimeout":0,"outgoingTimeout":0,"changeOrigin":false,"preserveHeaderKeyCase":false,"verifyCertificates":false,"followRedirects":false,"toProxy":false,"xfwd":false,"x":820,"y":1160,"wires":[]}]

An unexpected error will occur, resulting in following actions:

  1. A Node-RED error will be raised, which can be intercepted using the Catch-node:

    catch error

  2. An internal server error (status 500) will be returned to the client:

    server error

  3. The connection will be closed, to avoid that the client keeps waiting for an answer.

Performance

All the data chunks in the http response (arriving from the target server), need to be passed via the proxy to original response (handled by the http-in node). This means a lot of data needs to be handled, for example all images in an Mjpeg steam.

Let's use the Mjpeg stream (see example flow in the "Node Usage" paragraph), to get a basic idea of the performance. The stream contains about 11 images per second at a resolution of 640x480. As soon as the proxy starts handling the stream, there is almost no extra CPU being used on my Raspberry PI 3:

image

Remarks:

In depth explanation

This section explains in more detail how Node-RED handles the http requests and corresponding http responses.

The standard Node-RED solution in detail

Standard nodes

  1. Navigate to an URL (e.g. via a browser), which refers to a Node-RED instance.
  2. A http(s) request will be send to Node-RED.
  3. The http-in node listens for all requests for sub-path 'show_request', and it creates both a request and response object.
  4. The output message contains the request object in msg.req and the corresponding response in msg.res.
  5. The http-request node creates a new http request, which will be send to the target host.
  6. The target host will answer with a http response.
  7. The http-request node will copy the new http response content into the output message, while the original request and response will be left untouched (in msg.req and msg.res).
  8. The http-out node will fill the original http response (via msg.res) with data from the input message:
    • Fill response body with msg.payload.
    • Fill response status code with msg.statusCode.
    • Fill response headers with msg.headers.
    • Fill response cookies with msg.cookies.

At the end the browser will receive the response ...

The http-proxy contribution solution in detail

Proxy in Node-RED

  1. Navigate to an URL (e.g. via a browser), which refers to a Node-RED instance.
  2. A http(s) request will be send to Node-RED.
  3. The http-in node listens for all requests for sub-path 'show_request', and it creates both a request and response object.
  4. The output message contains the request object in msg.req and the corresponding response in msg.res.
  5. The http-proxy will redirect the original http request to the target host.
  6. The target host will answer with a http response.
  7. The http proxy node node will fill the original http response (via msg.res) with data received from the target system.

At the end the browser will receive the response ...