proxy for outgoing requests

[5im-0n]

Is there a way to specify a proxy for outgoing requests? At work we have to pass requests that go to the internet trough a proxy. I have http_proxy and https_proxy environment variables set, and most command line utilities like curl, git and so on use those env vars, but when I do something like

--rewrite '/api/* -> https://external-service.example.com/api/$1'

the https://external-service.example.com/api/$1 requests do not use the proxy for the outgoing connection. Am I doing something wrong, or is a proxy just not supported?

[75lb]

Hi, sorry I was on holiday.. requests rewritten to a remote server using --rewrite do not currently respect http_proxy env variables, I will look into this..

[5im-0n]

No problem @75lb, thanks for the answer and thanks for putting this awesome web server together. Hope you had a nice holiday!

[5im-0n]

I tried to create a patch for this in req-then, but the proxy connection is rather complex for https endpoint (a CONNECT has to be issued first). Would it be ok for you if I mess around in req-then to add the proxy support? Or would you rather use some other library that can handle a proxy to make the outgoing request and add that as a dependency?

[5im-0n]

local-web-server could also use global-tunnel-ng, just by adding require('global-tunnel-ng').initialize() in the ws script, or sometime on startup. But the globalAgent override in rewrite would have to go, and setSocketKeepAlive has to be used instead.

I would be happy to make a pull request. Let me know @75lb.

[5im-0n]

Here is my proposal: https://github.com/lwsjs/rewrite/pull/2 https://github.com/lwsjs/local-web-server/pull/116

[75lb]

Hi, thanks again for the contribution.. this is high on my list to look at, will get back to you ASAP 👍

[75lb]

hmm, global-tunnel does not support node v11 and above - it recommends using global-agent but that doesn't support node v8 (but that's not a blocker).

We need node v12 support at the least. If your patch works using global-agent, let me know. I'm looking for the most minimal solution possible to achieve http_proxy support.

Btw, the require('global-tunnel-ng').initialize() line you added to local-web-server should instead be added to the top of the lws-rewrite module. Keeps all the changes in one place and avoids tightly coupling local-web-server to its plugins. 👍

[5im-0n]

Here you go: https://github.com/lwsjs/rewrite/pull/3

[75lb]

If you run this script using your own local proxy host and port details does it correctly print the facebook response?

const http = require('http')

const reqOptions = {
  host: '134.209.178.181', // replace with your proxy host
  port: 3128, // proxy port
  path: 'https://www.facebook.com',
  headers: {
    host: 'www.facebook.com',
    'user-agent': 'nodejs'
  }
}
http.get(reqOptions, res => {
  console.log(res.statusCode, res.headers)
  res.pipe(process.stdout)
})

[5im-0n]

No, it does not work. The proxy is configured to use the CONNECT method for TLS connections.

You can reproduce the problem like this: https://31337.it/ is configured to allow connections on port 80 and 443. If a connection on port 80 is made, a 302 redirect to https is returned:

$ npx anyproxy &

$ cat | node <<EOF
const http = require('http')

const reqOptions = {
  host: '127.0.0.1', // replace with your proxy host
  port: 8001, // proxy port
  path: 'https://31337.it/',
  headers: {
    host: '31337.it',
    'user-agent': 'nodejs'
  }
}
http.get(reqOptions, res => {
  console.log(res.statusCode, res.headers)
  res.pipe(process.stdout)
})
EOF

$ npx anyproxy &
[1] 1198
[AnyProxy Log][2019-05-22 08:55:05]: Http proxy started on port 8001
[AnyProxy Log][2019-05-22 08:55:05]: web interface started on port 8002
cat | node <<EOF
> const http = require('http')
>
> const reqOptions = {
>   host: '127.0.0.1', // replace with your proxy host
>   port: 8001, // proxy port
>   path: 'https://31337.it/',
>   headers: {
>     host: '31337.it',
>     'user-agent': 'nodejs'
>   }
> }
> http.get(reqOptions, res => {
>   console.log(res.statusCode, res.headers)
>   res.pipe(process.stdout)
> })
> EOF
[AnyProxy Log][2019-05-22 08:55:10]: received request to: GET 31337.it/
301 { date: 'Wed, 22 May 2019 06:55:11 GMT',
  server: 'Apache/2.4.29 (Ubuntu)',
  location: 'https://31337.it/',
  'content-type': 'text/html; charset=iso-8859-1',
  'x-anyproxy-origin-content-length': '299',
  'x-anyproxy-origin-connection': 'close',
  'content-length': '299',
  connection: 'close' }
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://31337.it/">here</a>.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 31337.it Port 80</address>
</body></html>

As you can see, the request to 31337.it from the proxy is made on port 80, not 443, and an http 302 is returned. Same happens with facebook.com or stackoverflow.com.

[75lb]

Hi, thanks for that reproduction example - i'll try that in a min.. in the meantime, could you test this example please. Here are the commands to run the gist:

git clone https://gist.github.com/7e00e6c6acebb1e9da85e5b5521a2a26.git tunnel
cd tunnel/
npm install 
node index.js 

The script will attempt to load https://www.google.com via your configured http_proxy variable (which must be in http://x.x.x.x:xxxx format).

[5im-0n]

jup: works.

[5im-0n]

But it does not work with http://www.nossl.net/

[75lb]

yeah, the example only works with HTTPS - for HTTP I would need to use a different module (http-proxy-agent).

I'm looking for a reliable, non-intrusive solution which ideally works with node 8,10 and 12, handles Proxy Auth and both HTTP proxying and CONNECT-based tunnelling. Thanks for helping me so far!

Could you try my example using the request module please. It will automatically pick up your http_proxy env variable.

git clone https://gist.github.com/c3a2da31bbec2711074f7fcacf4c6859.git request
cd request/
npm install 
node index.js <url to request>

If HTTP requests don't work for you, try forcing the request module to always use CONNECT-tunnelling by setting tunnel: true in line 4.

[5im-0n]

request works, but it's deprecated: https://github.com/request/request/issues/3142

[75lb]

yeah, personally I never use it but in this situation it ticks the most boxes. I'm not using global-agent, it is massive (includes the 5mb core-js library), seems incomplete (it's less than one month old and a WIP) and is completely unused in the wild.

[5im-0n]

I understand. Then maybe request is the best option. It will work for a lot of years to come for sure.

[75lb]

This is fixed and released in lws-rewrite v1.0.0. The new version uses the latest version of path-to-regexp which has some breaking changes since the last version.. basically, if your "from" route contains a wildcard, e.g. /api/* it should be changed to /api/(.*). Let me know if you have any issues.

To test the latest lws-rewrite, install it locally to your project then run a custom stack e.g.

$ npm install lws-rewrite
$ ws --stack ./node_modules/lws-rewrite/ lws-static

[5im-0n]

Works like a charm!

[5im-0n]

@75lb, may I ask when you plan to release this new version?

[75lb]

I'm still working on local-web-server v3 but i've published a prerelease, just for you.

$ npm install local-web-server@next

Obviously this is still in beta and unstable! Let me know if you have any issues.

[75lb]

Fixed and released in v3.0.0