Mixing of HTTPS proxying and SNI+TCP on a single port

[far-blue]

In my setup the majority of HTTPS handling is done at the edge of the cluster by Fabio with internal services running only HTTP. However, BitBucket is being a complete pain and, thinking the request is on HTTP, generating 302 redirects to HTTPS because it doesn't realise the proxy is handling this.

It would be very useful to be able to mix SNI+TCP and HTTPS proxying on the same port (443) based on the incoming request hostname. If the incoming request matches a hostname for SNI+TCP then SNI+TCP will be used, otherwise HTTPS proxying down to HTTP would be used.

[magiconair]

Is there another way to convince BitBucket that the connection is on HTTPS? Some way that fabio isn't supporting yet?

[far-blue]

I have no idea what it is using to check. TBH, it's really an issue for Atlassian because BitBucket really shouldn't care. It should just use '//' on the protocol for ajax calls etc. I've raised a support ticket for them.

However, in general, I could see how a setup with fabio being used as edge proxies for a cluster running multiple separate and distinct services could have other needs for mixing HTTPS and SNI+TCP style proxying on the same port.

I figure the easiest way to achieve that would be to check an incoming request against a regex of hostnames that should be using SNI+TCP and, if it doesn't match then fall back to using HTTPS

[far-blue]

Does Fabio add the proxy headers in the same way Apache does? Specifically:

X-Forwarded-For: The IP address of the client. X-Forwarded-Host: The original host requested by the client in the Host HTTP request header. X-Forwarded-Server: The hostname of the proxy server.

And does Fabio support CONNECT style tunnelling?

[magiconair]

This is the code that handles the headers in addition to what the golang http library is doing. https://github.com/eBay/fabio/blob/master/proxy/util.go#L22-L94

Unfortunately, the tests are only unit tests and not full integration tests which tests the combination of both. Might be a good idea to refactor them.

I think the X-Forwarded-For header is set by the go lib and fabio only adds this for websocket connections. I don't see code for either the X-Forwarded-Host or the X-Forwarded-Server headers in either fabio or go. CONNECT may work but I would need to test that. Could you please open separate issues for the headers and CONNECT?

[deuch]

Hello, i've a use case for that too :)

I've setuped a fabio and try to have SNI and HTTPS on the same port.

The idea is that some micro services are served only with FQDN (myservice.domain.com) so SNI is OK for that, and some have path based routing (api.gw.com/locate/v1 or api.gw.cm/deal/v1.2 etc ...)

I can not use both of those system with fabio because SNI take the lead above HTTPS. So i need 2 ports to do that and it's not so easy because developpers need to know the ports of the fabio. I would prefer to use the 443 for the both (like a nginx or haproxy).

Is it possible to fabio to check the table a little differently : If fabio has a request, check the SNI hostname, if you find a exact match, do with SNI and if not, check the global url to find match

example :

src : tomcat.demo.com/ dest : http://192.168.1.2:32225 --> with SNI src : tomcat.demo.com/book/v2 dest : http://192.168.1.2:32225 --> Classical https src : api.demo.com/locate/v1 : http://192.168.4.2:17554 --> Classical https

One other thing, if i use the same port for HTTPS and SNI and set proto=https and tlsskipverify=true, SNI is used rather that HTTPS and HTTPS upstream ...

[deuch]

An other thought about it and maybe a start for an algorithm ...

Imagine to have a setup like this:

proxy.addr=:443;proto=tcp+nsi,:443;proto=https;cs=mysslcs

So we have multiple scenarios with our url-prefix and tags :

1) urlprefix = mysite.domain.com/ 2) urlprefix = mysite.domain.com/ proto=https 3) urlprefix = mysite.domain.com/v1/prod/myapi 4) urlprefix = mysite.domain.com/v1/prod/myapi proto=https

For 3) and 4) it's pretty obvious ... 3) Retrieve the certificate, do the SSL termination and call the backend with HTTP 4) Retrieve the certificate, do the SSL termination and call the backend with HTTPS (for full SSL end to end)

2) Retrieve the certificate, do the SSL termination and call the backend with HTTPS (for full SSL end to end)

For the 1) it can be a little more complex, but in fact you have just to check if you have a certificate in Fabio. If it's the case, do the SSL Termination. If not use SNI instead.

Or if you do not have SNI information, check the certificate and do the SSL termination.

What do you think ?

[magiconair]

As for your first suggestion to check from least to most specific routes I think this isn't the right approach. You'd expect the more specific routes to be matched first.

Also, this isn't a problem of wanting to provide this but the two proxy implementations are quite different and I have a hard time coming up with a solution that could work. What you are trying to do here smells a lot like a refactoring issue that you should solve in the app instead of abusing the LB to do this. You could either introduce another ip address or just a different hostname for these endpoints. In the end you want very different behavior: either full end-to-end encryption for the entire conversation or man-in-the-middle decryption by the LB with more dynamic routing. I know that other LBs may be able to do this but I usually encourage people to try to solve this in the app itself since the solution is then independent of the LB.

[deuch]

Our fabio will be mutualized (at a certain level) between application, and we have 1500 applications to serve.

For me applications do not have to understand how it works behind the scenes. Put a certificate in Vault and an urlprefix and they do not need to do more. Everything else is not the responsability of the service.

We have automation for certificate, dns record, fabio installation and configuration. But some applications will handle their certificates out of a vault, and we need to adress those needs.

My proposal is just to be able to use the same port for tcp+sni and https. The order to check the rules is not from least to more specific, its just examples of each examples can be treated.

Only the scenario 1) can have 2 possibilities. And the decision to do sni is based of a presence of certificates in fabio. if not, do sni, if you have a certificate, terminate ssl ... or maybe you can add a tag to help for deciding which method to use. Like terminateSSL=true or false ?

My concern is to have to use other port than 443. It start to be complicated to have 2 port for SSL or different hostnames to the same thing (for a developer point of view).

Anyway i can use man in the middle for both but it seems just not the best approach for performance. and websocket seems to not work with https upstream :)

[magiconair]

If performance is your main concern and not full end-to-end encryption the I suggest to benchmark first. Users haven't complaining that fabio can't handle their workload. Also, you can always spin up more instances. I'll look at the websocket issue in the meantime

[taemon1337]

I think this is a valid use case, for me as well.

Some clients/services that are using the fabio load balancing may want to use its HTTPS termination for free while others may want TCP+SNI for end-to-end encryption with their own certificate.

p.s. great work on fabio!

[taemon1337]

Duplicate of https://github.com/fabiolb/fabio/issues/169

The tcp+sni is an awesome feature which is one of the reasons we are looking to use fabio for client load balancing, and would definitely like the flexibilty to choose tcp+sni or https termination.

[magiconair]

413 could become relevant for this. Feel free to comment on that issue if that is a viable path for you.

[shantanugadgil]

Does this look fixed in version 1.5.14 with https://github.com/fabiolb/fabio/pull/784

[nathanejohnson]

@shantanugadgil this feature was rolled out as of 1.5.14 several months ago. Thanks for pointing this out, I will close this here.