Ambassador returns http 404 when request url doesn't include a port in host header

[rootrahulagr]

As per the following release note, if a request doesn’t specify a port, port 443 is assumed by ambassador when it comes through. So, ambassador should handle both the requests with or without port with common host configuration.

https://www.getambassador.io/docs/emissary/latest/release-notes#3.5.0

However, this theory is not working in our case. Ambassador is able to translate only one type of request at a time, depending on its host configuration. If host.hostname is configurated with ":port", the incoming request with a port in host header would be accepted (the request without port would get 404) and vice versa.

We are using Ambassador (Emissary Ingress) v3.6.0, chart version emissary-ingress-8.6.0, on EKS v1.24 cluster.

Configuration of ambassador related resources look like this (have masked some fields):

Host -

apiVersion: getambassador.io/v3alpha1 kind: Host metadata: name: demo-host spec: hostname: "*.xxx.yyy.abc.com.cd:443" tlsSecret: name: ambassador-certs requestPolicy: insecure: action: Redirect

Listeners -

apiVersion: getambassador.io/v3alpha1 kind: Listener metadata: name: https-listener spec: port: 8443 protocol: HTTPS securityModel: XFP hostBinding: namespace: from: ALL

apiVersion: getambassador.io/v3alpha1 kind: Listener metadata: name: http-listener spec: port: 8080 protocol: HTTP securityModel: XFP hostBinding: namespace: from: ALL

Mapping (Note: This mapping is defined as annotation of application's service object) -

apiVersion: ambassador/v3alpha1 kind: Mapping name: myapp_mapping service: https://kubernetes-dashboard.kube-system host: "myapp\..*\.abc\.com\.cd" host_regex: true prefix: /dashboard/

[rootrahulagr]

We were not using TLS context, so it was not part of the setup originally. However, I have created TLS Context now with following definition, as suggested by @cindymullins-dw, tested again and it didn't make any difference. I am not sure either what purpose this TLS context is serving here.

---

apiVersion: getambassador.io/v3alpha1 kind: TLSContext metadata: name: example-host-context namespace: default spec: hosts: [".xxx.yyy.abc.com.cd", ".xxx.yyy.abc.com.cd:443"] secret: ambassador-certs

Have also linked this tls context to the host by adding tlsContext.name in Host resource.

With this setup, currently on the request without port is working.

[cindymullins-dw]

Thanks for trying that. I think you can remove the TLSContext. The original issue was fixed in 3.5 so your config should be working for you and I was hoping the TLSContext was the missing piece. However since that isn't working, we do have a manual workaround as below which we've tested.

8443 is special in that it is our default listener for 443 (or "port-less" requests). Can you use a port like 10443? If so:

• create listener for 10443 and expose that on the EdgeStack service
• add match Labels to say which Host can attach to the Listener
• add host with hostname: example.com:8443
• include the same match labels as listener
• add mappingSelectors (hostname example.com:443) so that Mappings will only be attached to this Host

Then request to https://example.com:10443/backend/would have its own Listener/FilterChain/Vhost and routes and would not collide with the other routes that are on the standard 8443 listener. The config would look like this:

apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: edge-stack-listener-10443
  namespace: ambassador
spec:
  hostBinding:
    namespace:
      from: ALL
    selector:
      matchLabels:
        "aes-listener": "10443"
  port: 10443
  protocol: HTTPS
  securityModel: XFP

Second, create a Host that has a wild-card hostname ( "*") but leverages the a TLS Secret that works for the hostname without the port (i.e v3test.mstanfield.k736.net). In this sample, I just use the one Acme created for the other host.

Add Labels to the metadata so that it will bind to the Listener listening on 10443. Add Mapping selectors too, which will ensure only the routes you want exposed on this listener are added and not the routes that attach to the other Host.

---
apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: 10443-wildcard-host
  labels:
    "aes-listener": "10443"
spec:
  hostname: "*"
  tlsSecret:
    name: v3test.mstanfield.k736.net
  mappingSelector:
    matchLabels:
      "aes-listener": "10443"

Add the mapping to back end service, ensuring you have the correct Labels added so that it can be added to the Host

---

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: quote-backend3
  namespace: default
  labels:
    "aes-listener": "10443"
spec:
  prefix: /backend3/
  service: quote2

Finally, expose it externally on the Edge Stack LoadBalancer Service. You can expose 8443 externally while forwarding it to the 10443 where our Listener is configured with the new Host and Mapping:

ports:
- name: http
   protocol: TCP
   port: 80
   targetPort: 8080
- name: https
   protocol: TCP
   port: 443
   targetPort: 8443
 - name: https2
   protocol: TCP
   port: 8443
   targetPort: 10443

To ensure that this Host and Mapping do not attach to the other listeners (8443, 8080) then we need to set selectors and labels so something like this on those Listeners with corresponding labels on the Host:

# On Listener in hostbinding
selector:
  matchLabels:
    aes-listener: 8443-8080

 # on Host in metadata as Labels
labels:
  aes-listener: 8443-8080

This isolates the Host, and Routes into their own Listener and FilterChain in Envoy.

[rootrahulagr]

@cindymullins-dw Thanks a lot for your reply. Actually, the requirement from application team here is to use port 443 only. Can we please make it work based on fix in v3.5? If any further information is required from my side, please let me know.

Nevertheless, I tried the steps you have suggested (just in case we have a requirement to use some non-default port in future) but request is not getting through ambassador. FYI - we are using mapping through service annotation.

[rootrahulagr]

@cindymullins-dw : I also realized that in v3.5, it was only fixed for custom ports (other than 80 and 443), eg- 8550. I don't see anything fixed/mentioned there for port 443. Please let me know if I am missing something. Thanks.

In our case, the requirement at the moment is to use port 443, not custom ports.

[cindymullins-dw]

On a fresh install of 3.6 and 3.2 with requests including and not including the port, we get 200s on curl request. What does your curl request output look like? You're correct that this should be working on 443 and shouldn't require special configuration. Do you have anything in front of Emissary perhaps interfering with your 443 traffic? Are you getting traffic on port 80 ok?

[rootrahulagr]

Hey @cindymullins-dw :

I have tested via postman and the response is something like this -

HTTP/1.1 404 Not Found date: xxxx server: envoy content-length: 0

As we have deployed Emissary on EKS cluster, ambassador service is pointing a Amazon Network Load balancer. I don't think there is anything else in front of Emissary. We get 404 for http as well.

From our previous discussion over slack, I feel like this is something to do with host configuration itself as one kind of service always works at any point in time, depending on host configuration.

So, if host.hostname has port (443) defined, the service with port works fine (and service without port doesn't) if host.hostname has no port (443) defined, the service without port works fine (and service with port doesn't)

Looks like there is a tiny bit we are missing here somewhere around host configuration, just can't figure it out quickly and that's where need your expertise.

Thanks!

[cindymullins-dw]

Can you try implementing this protocolStack on the load balancer instead of using protocol: TCP only? protocolStack: [ "PROXY", "TLS", "HTTP", "TCP" ]

[rootrahulagr]

@cindymullins-dw : I tried to update ambassador service with suggested change, but it was throwing following error. Maybe I got something wrong. Could you share any example?

services "ambassador" was not valid:

: Invalid value: "The edited file failed validation": ValidationError(Service.spec.ports[1]): unknown field "protocolStack" in io.k8s.api.core.v1.ServicePort

[rootrahulagr]

Hey @cindymullins-dw : Please advise further on this. Thanks.

[cindymullins-dw]

So basically on port 443 this should be working, meaning your traffic should be routed whether the header includes the port or not. If its not working (or you're using a port other than 443) then a Lua script to strip the port is the suggestion, example here.

[cindymullins-dw]

Could we see this in action? Would you be able to join a help session with us on Thursday at 2:30pm ET?