Feature idea: nghttpx for HTTP/2 proxy

[wzyboy]

Hi,

I am both a big fan of Ansible and all kinds of VPN / proxy software. So I am thrilled to find such a awesome, detailed documented project like Streisand.

I am thinking about contributing a new Ansible role to Streisand that sets up an nghttpx HTTP/2 proxy.

HTTP/2 proxy is, in fact, a secured forward proxy that can only be used natively by modern browsers like Chromium / Google Chrome and new versions of Firefox. Using the proxy in other software is possible with additional client software.

Plain HTTP forward proxy has been blocked decades ago in China, but by wrapping an HTTP proxy in a TLS connection you get an HTTPS proxy. There are many ways to do it:

• Setting up a plain HTTP forward proxy like Squid and wrap the connection with stunnel;
• Compile Squid with --enable-ssl (non-default for most distros) and it can serve HTTPS proxy natively;
• Setting up an nghttpx daemon that translates between HTTP/2 and HTTP/1.1, acting as a frontend to an HTTP/1.1 forward proxy like Squid. Since modern browsers support only encrypted HTTP/2, this method is secure by default.

The last method has the best performance thanks to HTTP/2.

A few considerations:

• Authentication could be done during TLS handshake, or via HTTP basic auth (encrypted in TLS), or both. I prefer the former one but this requires manually importing a client certificate to the browser. What's your opinions?
• The biggest disadvantage of this method is that you need a valid HTTPS server certificate for your HTTPS proxy, just like any HTTPS website. The difference is that there is no "Proceed to XXX (unsafe)" link to click on and ignore the security warning. There is only an ERR_PROXY_CERTIFICATE_INVALID error which cannot be dismissed. So self-signed certificate simply does not work. Should the playbook prompt for a valid HTTPS server certificate to copy to the remote server?

[cpu]

Hi @wzyboy,

I am thinking about contributing a new Ansible role to Streisand that sets up an nghttpx HTTP/2 proxy.

I'm not familiar with nghttpx - is that a separate daemon or a configuration possible with nginx? My biggest point of feedback would be to make sure the role is modular if you do start on a new role.

Authentication could be done during TLS handshake, or via HTTP basic auth (encrypted in TLS), or both. I prefer the former one but this requires manually importing a client certificate to the browser. What's your opinions?

In general I recommend submitting a minimal viable proof of concept first and iterating on that with subsequent smaller PRs. Doing TLS client certificate authentication will likely complicate the role considerably and I'd rather review that as a follow-up to a more simplified HTTP basic auth implementation first.

The biggest disadvantage of this method is that you need a valid HTTPS server certificate for your HTTPS proxy, just like any HTTPS website. The difference is that there is no "Proceed to XXX (unsafe)" link to click on and ignore the security warning. There is only an ERR_PROXY_CERTIFICATE_INVALID error which cannot be dismissed. So self-signed certificate simply does not work. Should the playbook prompt for a valid HTTPS server certificate to copy to the remote server?

I'd like to get to the point where the gateway certificate can be a Let's Encrypt issued certificate when the Streisand instance has a domain name. I think that discussing an approach to accomplish this should be a separate prerequisite to supporting this proxy since it sounds rather critical. I think prompting for a certificate path is perhaps the simplest approach but not likely to be a great long term solution. It puts a lot of manual work onto the user which is counter to the intentions of Streisand. Perhaps you can add your thoughts on this to https://github.com/jlund/streisand/issues/639?

[wzyboy]

Hi @cpu,

I'm not familiar with nghttpx - is that a separate daemon or a configuration possible with nginx? My biggest point of feedback would be to make sure the role is modular if you do start on a new role.

nghttpx is a part of nghttp2 (BTW the author of nghttp2 also writes the famous aria2 project), which is a complete different project with nginx. They just share a similar name. An nghttpx installation consists of only three essential files:

• nghttpx (main executable)
• libnghttp2.so (a shared library with other components of nghttp2)
• nghttpx.conf (main configuration file)

In general I recommend submitting a minimal viable proof of concept first and iterating on that with subsequent smaller PRs. Doing TLS client certificate authentication will likely complicate the role considerably and I'd rather review that as a follow-up to a more simplified HTTP basic auth implementation first.

I agree. I will submit a quick-and-dirty role that deploys an nghttpx instance without any authentication (anyone know the domain name -- IP address does not work beacuse of invalid certifications -- and port could use it) as a POC.

I'd like to get to the point where the gateway certificate can be a Let's Encrypt issued certificate when the Streisand instance has a domain name. I think that discussing an approach to accomplish this should be a separate prerequisite to supporting this proxy since it sounds rather critical. I think prompting for a certificate path is perhaps the simplest approach but not likely to be a great long term solution. It puts a lot of manual work onto the user which is counter to the intentions of Streisand. Perhaps you can add your thoughts on this to jlund/streisand#639?

The idea of issuing valid certificates with Let's Encrypt is tempting. But I have never used Let's Encrypt before. As of nghttpx role, a valid certificate is required (you cannot dismiss the security error just like when you browse a simple HTTPS webpage with self-signed certificates). Currently I think a roadmap would be:

• A certificate path must be provided by user to run nghttpx role;
• A Let's Encrypt role is implemented and Streisand gateway now gets valid certificates automatically;
• The Let's Encrypt role is re-used in nghttpx role. A ceritifcate path is now optional to deploy nghttpx.

[wzyboy]

One fun fact I observed in China: many users of nghttpx simply do not use any authencation. They employ "security by obscurity" instead of "security by design".

For example, one could get a single-domain certificate for $9/y for some-long-and-hard-to-guess-subdomain.example.org and serve an nghttpx instance on a non-default port. The domain name and the port serves as a poor man's username and password, though they can be sniffed out easily as domain name is sent in plain text during TLS handshake if SNI is enabled (which is true for modern browsers).

However, if you are not a paranoid and you are not facing a state-sponsered hacker, an nghttpx instance without authencation is probably not a big problem. According to my own experiment, such a instance simply does not get any bot / crawler traffic being exposed in public Internet for months if you configure it to allow only TLS 1.2 + HTTP/2. Most bots and crawlers use old technology. They do not support TLS 1.2, not to say an HTTP/2 only server :-)

[wzyboy]

@cpu I have made a proof of concept PR: https://github.com/jlund/streisand/pull/885 :-)

[cpu]

I agree. I will submit a quick-and-dirty role that deploys an nghttpx instance without any authentication (anyone know the domain name -- IP address does not work beacuse of invalid certifications -- and port could use it) as a POC.

Hmmm. A PR that leaves a service unauthenticated like this wouldn't be mergeable.

Currently I think a roadmap would be: A certificate path must be provided by user to run nghttpx role;

I'm pretty adverse to merging anything that requires the user provision a certificate by hand and provide it. I think supporting the node getting its own certificate is a much more palatable way forward. If you're interested in working on this we could talk about details in another issue.

The domain name and the port serves as a poor man's username and password, However, if you are not a paranoid and you are not facing a state-sponsered hacker

This isn't congruent. It's extremely easy to do IPv4 wide Internet scanning to find instances like this for less than $10 and a Sunday (speaking from experience!). I don't think this is paranoia or specific to an extreme threat model, it's very baseline.

@wzyboy I don't want to tamper your enthusiasm to contribute (It's definitely welcome!) but I feel like this particular service isn't well suited to Streisand as it exists today. Perhaps it's something we could revisit in the future but there's some more foundational work to be done first.

[wzyboy]

Don't get me wrong. I was not saying I would like to merge an unauthenticated service to master, or running unauthenticated service in production. I myself is a full-time sysadmin for a financial company so I take security seriously. English is not my native language so maybe I express my ideas in a wrong way. All I want to do is to submit a POC first and let others test if it works in their country / region / firewall zone. And I want to discuss about how to authenticate the users before writing the authentication code.

I'm pretty adverse to merging anything that requires the user provision a certificate by hand and provide it. I think supporting the node getting its own certificate is a much more palatable way forward. If you're interested in working on this we could talk about details in another issue.

Hmm... so user-provided server certificate is not an option. Maybe it's time for me to take look at Let's Encyrpt...

I don't want to tamper your enthusiasm to contribute (It's definitely welcome!) but I feel like this particular service isn't well suited to Streisand as it exists today. Perhaps it's something we could revisit in the future but there's some more foundational work to be done first.

Okay. We may revisit this role after the role of Let's Encrypt is done.

[cpu]

Don't get me wrong. I was not saying I would like to merge an unauthenticated service to master, or running unauthenticated service in production. I myself is a full-time sysadmin for a financial company so I take security seriously. English is not my native language so maybe I express my ideas in a wrong way.

Ok! I apologize for getting the wrong idea :-) I think we're both understanding each other now.

All I want to do is to submit a POC first and let others test if it works in their country / region / firewall zone. And I want to discuss about how to authenticate the users before writing the authentication code.

:+1: - That makes sense!

Hmm... so user-provided server certificate is not an option. Maybe it's time for me to take look at Let's Encyrpt... Okay. We may revisit this role after the role of Let's Encrypt is done.

Yeah. Full disclosure: I work for Let's Encrypt so I'm obviously not a neutral party here :laughing:

I think native Let's Encrypt support is the best path forward for Streisand because: a) It's free and won't add an additional cost to the user on top of hosting the instance b) It's automated, meaning that Streisand can get the certificate & install it in a generic way without requiring any manual administrator steps (beyond the initial DNS configuration for having an external hostname pointing at the IP).

Experienced sysadmins like yourself generally can get their own certificate with another CA and provide it to Streisand but it's a difficult process for most users. Accepting a cert from the user would also require accepting an intermediate certificate and a private key. We'd have to add code that checked the private key matched the leaf certificate's public key and that the chain validates leaf to root with the provided intermediate. Otherwise that will be a lot of chances for a user to get something wrong! :-)

[wzyboy]

Yeah. Full disclosure: I work for Let's Encrypt so I'm obviously not a neutral party here

Nice!

I think native Let's Encrypt support is the best path forward for Streisand

I searched both streisand and streisand-discussions issues but could not find more related topics other than https://github.com/jlund/streisand/issues/639. Is there any work in progress that I could take a look at? If there is none, do you have some suggestions on the choice of implement of Let's Encrypt? I heard there are many software that implements the ACME. Which one do you think suits for Streisand?

I'll read the docs this weekend if I have time. But I'd appreciate it if you could give some suggestions :-)

[cpu]

Is there any work in progress that I could take a look at?

I don't believe there is. I started working on replacing the OpenSSL based gateway TLS certificate/generation with a small tool called minica my colleague wrote but haven't moved beyond that to getting a public CA issued certificate. It looks like I haven't pushed the branch and it isn't on this computer. Remind me later and I will make it available! It's significantly behind master because I decided to stop working on that to prioritize the modular roles work.

If there is none, do you have some suggestions on the choice of implement of Let's Encrypt? I heard there are many software that implements the ACME. Which one do you think suits for Streisand?

I think there are two ways to go. My initial inclination is to use Certbot because it's the ~standard ACME client that Let's Encrypt recommends. It's a little bit heavyweight and it can be somewhat of a challenge to get the latest version in 16.04 from package. Personally I like acme-tool.

Presently only port 443 is externally accessible, I believe 80 is being used for the tor hidden service or something (I'd have to verify). If this is the case only the TLS-SNI-01 challenge will work and it might be a little tricky to integrate with Nginx/sslh. HTTP-01 will not work without some tweaking. DNS-01 is probably not a good fit for Streisand because of the complexity & number of different DNS providers that could be involved.

[wzyboy]

I think there are two ways to go. My initial inclination is to use Certbot because it's the ~standard ACME client that Let's Encrypt recommends. It's a little bit heavyweight and it can be somewhat of a challenge to get the latest version in 16.04 from package. Personally I like acme-tool.

I took a quick look at acme-tool and I think I like its single-file dependency-free nature and idempotent design. "Unlike the official Let's Encrypt client, this doesn't modify your web server configuration." (quoted from acme-tool README) This is great for Streisand because it seems a lot of headache to be idempotent if two programs (Ansible and ACME official client) wants to modify a same file...

I'll try to write something with acme-tool later.

[cpu]

This is great for Streisand because it seems a lot of headache to be idempotent if two programs (Ansible and ACME official client) wants to modify a same file...

Agreed. For reference Certbot can also operate in a mode like acmetool where it only generates the cert but doesn't modify your webserver config. Certbot calls it "certonly" mode.

[wzyboy]

Though the feature idea itself (nghttpx) has been rejected. There is a side product of this discussion: Let's Encrypt role. I'll leave the link here and close this issue: https://github.com/jlund/streisand/pull/911