Reverse proxy to HTTPS backend web server

[GloktarFR]

Hello,

I'm trying to use OOD to connect to a custom HTTPS server using the reverse proxy feature. While it works very well for unsecure web servers (HTTP), with the SSL layer I'm getting the following error:

Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request

Reason: Error reading from remote server

I'm not sure if it's currently possible with OOD to do such thing because all other interactive apps (like Jupyter Notebooks or RStudio) works without the SSL layer.

Can you please enlighten me?

[johrstrom]

We currently only support plain http. Though someday we'd like to support ssl transports here.

How are your clients able to use certificates? Do you just have them readable on the cluster so that apps can directly use them?

[GloktarFR]

Thank you for the answer. That's what I suspected.

I suppose that we could use something like this yes. The problem is more like we have some web-based applications that do not support plain http, so we can't use OOD as a proxy to access them. So it’s more like a functional problem than a security problem (for us at least).

[johrstrom]

Can you provide details on how you would give access to certificate root chains? While I've thought about this feature, that's the one thing I'm stuck on - allowing users to create or use system certificates that would be recognized by the apache.

[GloktarFR]

I'm not sure I follow you, I naively thought it would work the same way as a regular NGINX reverse proxy. Can you give me more details?

[johrstrom]

Sorry - it could be that I'm confused. I've only ever thought of what technical hurdles there are, not actually investigated any. That and my understanding of SSL is very basic.

One technical hurdle I thought (think) would exist when we do enable ssl connections is the ability for an unprivileged user to use and possibly create ssl certificates that they then boot their application with.

For example - I (as an unprivileged user) cannot even read the ssl certificates that my centers' OOD apache uses, let alone create one that certifies <compute node hostname> urls.

The certificate that the application uses on the origin server (the compute node) has to work with the certificate chain that apache uses for it to secure the connection. As an example, self signed certificates wouldn't work here. Unprivileged users would need access to (I believe) the root certificate which (I suspect) could be a bit troublesome.

[GloktarFR]

I’m not that into SSL myself, so I’m trying to keep up here. But I was not aware of any issue that we could have using self-signed certificates from an unprivileged user perspective. This basically means (forgive me if I’m completly wrong here) that as an unprivileged user, I cannot start a basic web server with self-signed SSL certificate and establish a secure connection with a reverse proxy on the other side?

My original question regarding this was for a complete different use case, where the web server is installed on the « compute node » by an administrator with proper self-signed or trusted certificates.

[johrstrom]

This basically means (forgive me if I’m completly wrong here) that as an unprivileged user, I cannot start a basic web server with self-signed SSL certificate and establish a secure connection with a reverse proxy on the other side?

When you create a self signed certificate you're saying that you are a certificate authority and the certificate you've generated is OK. Only problem is, nobody else thinks you're an authority, so how can they trust your certificates? You'd have to get your apache instance to recognize you (and everyone else) as certificate authorities.

Again, that's from my very basic understanding of this all - so there's likely more to it and/or nuance. But that's basically the gist. You have to generate certificates from some 3rd party (like LetsEncrypt) who is an authority and is recognized as such by the first (apache) and second (your app) parties.

My original question regarding this was for a complete different use case, where the web server is installed on the « compute node » by an administrator with proper self-signed or trusted certificates.

Even so, an unprivileged user has to boot the app up. The app will in turn have to read these certificates. so they'd need 644 permissions, which means the same user can generate more certificates? I think that's the vulnerable spot.

[GloktarFR]

I understand your point, though maybe these kind of security concerns do not apply to the majority of clusters. For example right now, when you access Jupyter Notebook, RStudio, or Matlab, one can snif packets and retrieve sensitive informations, but… it’s fine for everyone because we (kind of) trust all the users who have access to the cluster.

One way of improving this, is to secure the communications between the reverse proxy and the compute node running the application. Using SSL here can help and could be a first step. It has the advantage of preventing these type of attacks. But you would have to implement a ‘bypass’ function in OOD to be able to skip the CA verification. I’m not sure if that’s enough to do that, but I assume that’s the purpose of the ‘proxy_ssl_verify’ option in NGINX. This way, we would be able to use self-signed certificates for these kind of web app, in order to secure the link, but without having the CA verification concern.

I don’t know if that’s enough in terms of security, nor if it is technically possible to use SSL like this, maybe I’m completely wrong here :’( But having SSL certificates with proper CA recognition seems difficult to implement in a cluster environment.