HTTPS termination at external load balancer results in http scheme

[jaredtbates]

Motivation

I am using an AWS Network Load Balancer (layer 4) in front of Spring Cloud Gateway, but I think that this feature request belongs in the Reactor project instead (but correct me if I'm wrong). I have proxy protocol v2 enabled and the load balancer is doing TLS/HTTPS termination at the edge. However, when the requests come into Reactor/SCG, the scheme is always http instead of https. I realize that this is a limitation of using a layer 4 load balancer in front (which lacks the ability to add X-Forwarded-Proto to the headers), however I feel like there should be a way to manually configure Reactor to set the scheme to https instead in this case.

Desired solution

A way to set the scheme to https if the incoming proxy protocol port is 443, or some other approach.

Considered alternatives

Currently the alternative would be to switch to an Application Load Balancer which is layer 7. We'd like to stick with the NLB if at all possible to keep it simple and defer all gateway/routing/load balancing decisions to SCG.

Additional context

It's possible that I'm missing an easy way to extend Reactor to support this use case, or that this is out of scope for the project. Please point it out to me if I'm missing something! If I get time, I may experiment with this in code, but I'm not sure how you would suggest approaching it, which is the reason for me creating this issue.

[violetagg]

@jaredtbates Did you try to configure Spring Cloud Gateway that there will be proxy protocol? You need to configure the server with https://projectreactor.io/docs/netty/release/api/reactor/netty/http/server/HttpServer.html#proxyProtocol-reactor.netty.http.server.ProxyProtocolSupportType-

[jaredtbates]

Thanks @violetagg! I did, here was my configuration class:

import org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactory;
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.stereotype.Component;
import reactor.netty.http.server.ProxyProtocolSupportType;

@Component
public class ReactorCustomizer implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {

    @Override
    public void customize(NettyReactiveWebServerFactory factory) {
        // enable NLB proxy protocol support
        factory.addServerCustomizers(httpServer -> httpServer.proxyProtocol(ProxyProtocolSupportType.AUTO));
    }

}

I also added the io.netty:netty-codec-haproxy dependency.

[violetagg]

@jaredtbates Can you verify that the load balancer sends the necessary data? We support proxy protocol v1 and v2.

[jaredtbates]

From what I can tell the AWS NLB supports sending proxy protocol v2, and when I set it up, I see my source IPs showing up in logs appropriately. (more documentation is here: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#proxy-protocol)

I may be missing something (this is honestly one of my first projects with reactor, so it is very likely!), but from my very brief digging in the code, I only see the source IP and port being set in the channel: https://github.com/reactor/reactor-netty/blob/911d0b687825351d05b53d42d2f1b8b5e682ac7f/reactor-netty-http/src/main/java/reactor/netty/http/server/HAProxyMessageReader.java#L70-L76

I see there is a tlvs() method available in HAProxyMessage... maybe that can be used to set the X-Forwarded-Proto/scheme downstream to https?

[violetagg]

@jaredtbates I think I know what's going on ... let me see whether we can fix it.

[violetagg]

One last question: do you have server.forward-headers-strategy configured? If yes - is it with native or framework?

[jaredtbates]

I did not have that enabled. I will try that tomorrow to see if it makes a difference!

[violetagg]

yes please do so server.forward-headers-strategy=native

[jaredtbates]

@violetagg I tried that today and I seem to see the same thing happening. I did set up a layer 7 load balancer that sends the X-Forwarded-Proto header and pointed it at the same Spring Cloud Gateway instance, and it passed along the https proto just fine to the application. It's just something with the layer 4 load balancer and proxy protocol that isn't picking it up.

[jaredtbates]

I did some more digging and I'm not entirely sure if the proxy protocol can forward if the request was made with SSL or not. I'll leave that to you all, the experts :) I appreciate your help with looking at this! I understand if it's an edge case enough that this issue ends up being closed - no worries at all.

As a workaround - instead of switching to a layer 7 LB - I added the following to my Spring Cloud Gateway config to essentially "force" the X-Forwarded-Proto: https header to be added to the downstream requests. It seems to work well enough for my use case.

spring:
  cloud:
    gateway:
      default-filters:
        - AddRequestHeader=X-Forwarded-Proto, https
      x-forwarded:
        proto-enabled: false

[violetagg]

@jaredtbates I'm closing this issue with status has-workaround.