ssl-passthrough ingress breaks others

[tumbl3w33d]

Might be related to #492 but in this case it's not a missing feature but it actually breaks my setup when I add the ssl-passthrough ingress to other ones which "merge" properly.

My ingresses

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myproject-haproxy-soap-http
  namespace: myproject
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/hsts: "false"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: myproject-service
              servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myproject-haproxy-management
  namespace: myproject
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/hsts: "false"
    ingress.kubernetes.io/backend-protocol: "h1-ssl"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /management
            backend:
              serviceName: myproject-service
              servicePort: 8500

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myproject-haproxy-soap-tls
  namespace: myproject
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
      - example.com
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              serviceName: myproject-service
              servicePort: 443

Service

apiVersion: v1
kind: Service
metadata:
  name: myproject-service
  namespace: myproject
  annotations:
    haproxy.org/check: "disabled"
    haproxy.org/forwarded-for: "disabled"
spec:
  selector:
    customer: myproject
  ports:
    - name: soap-https
      protocol: TCP
      port: 443
      targetPort: 8443
    - name: soap-http
      protocol: TCP
      port: 80
      targetPort: 8081
    - name: management-http
      protocol: TCP
      port: 8500
      targetPort: 8500

Explanation:

• ingress 1 exposes simple http of service port 80
• ingress 2 exposes service port 8500 which itself terminates SSL, hence the backend-protocol annotation
• this both works fine
  • I can see in the config that the ssl-redirect configuration has effect, i.e. the request reaches the right backends, ssl-redirect and hsts is suppressed
• when I create the third ingress which is supposed to work on port 443 and pass through to the backend service 8443, everything breaks, i.e. the former working http frontend on port 80 gets overwritten
  • the service backends 80 and 8500 are unreachable
  • ssl-redirect and hsts suppression is gone

With the first 2 ingresses the /etc/haproxy/maps/_front001_host.map looks like this

example.com/management myproject1_myproject-service_8500
example.com/ myproject1_myproject-service_8081

When the third ingress comes in, such file doesn't exist anymore. Instead, the port 80 frontend loads /etc/haproxy/maps/_global_http_front.map which is empty. So the effect is, that the formerly working http frontends are gone.

[jcmoraisjr]

Hi, ssl-passthrough was designed in such a way that, if configured, everything else is ignored due to the fact that a path cannot be extracted from the encrypted data. But http/:80 doesn't have this issue and could be configured if ssl-redirect is false. I'll have a look on this and perhaps merge to at least v0.10 depending on the amount of changes. Thanks for reporting.

[tumbl3w33d]

Edit: ignore this comment. Explanation in the next. ;)

Hi @jcmoraisjr , thank you for your quick response. I understand that the current ssl-passthrough approach comes with these limitations, but reading the explanation I'd like to extend the example to make sure we're on the same page about what HAProxy can achieve and it would be great if the ingress controller would strive for the same.

For the sake of completeness I'll post all 3 ingresses again with a modified version of the second

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myproject-haproxy-soap-http
  namespace: myproject
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/hsts: "false"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: myproject-service
              servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myproject-haproxy-management
  namespace: myproject
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/hsts: "false"
    ingress.kubernetes.io/backend-protocol: "h1-ssl"
spec:
  tls:
    - hosts:
        - example.com
      secretName: example-cert
  rules:
    - host: example.com
      http:
        paths:
          - path: /management
            backend:
              serviceName: myproject-service
              servicePort: 8500

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myproject-haproxy-soap-tls
  namespace: myproject
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
      - example.com
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              serviceName: myproject-service
              servicePort: 443

In this scenario there are pure http hosts (1st ingress), ssl hosts terminated by HAProxy (ingress 2) and ssl hosts passed through (ingress 3). And the currently generated configuration is not even far from delivering that.

• the /etc/haproxy/maps/_global_sslpassthrough.map looks wrong. It actually only contains the pure http service backend at 8081 instead of the one at 8443 which handles ssl on its own
• the frontend that now listens on the unix socket which actually is supposed to handle those hosts that don't do their own ssl looks not bad, apart from the /etc/haproxy/maps/_front001_host.map being completely empty although it should contain the backend for my service host 8500.
• the unix socket frontend certificate list at /etc/haproxy/maps/_front001_bind_crt.list only contains the default fake certificate instead of the one provided by ingress 2

Again, like in the first scenario, the setup works as expected as long as the third ingress with ssl-passthrough doesn't come in.

Since I'm new to k8s and the haproxy ingress controller - there's no way of working around this until it's handled properly, is there?

[tumbl3w33d]

I just realized that it cannot work to distinguish on the same port with the same hostname depending on the path because that information isn't part of the SNI header. My bad. In the setup from which I'm porting this I used a different port for that reason.