Listener not accepting HTTPS without a `*` host being defined, but with a `*` host, `mappingSelector` do not work

[dmaclaury]

Describe the bug Through testing locally and on EKS 1.29 I have run into this same issue.

• Without a Host defined for the hostname *, a listener defined with protocol: HTTPS does not accept HTTPS/TLS connections
• If I create a host with hostname: "*", mappingSelector does not work on more specific hosts.

To Reproduce Steps to reproduce the behavior:

1. Deploy Emissary following the getting started guide, and these values:

   namespace:
   name: emissary
   ingressClassResource:
   name: emissary
   service:
   type: ClusterIP
   replicaCount: 1
   resources:
   limits:
   # cpu: 100m
   memory: 6000Mi
   requests:
   # cpu: 100m
   memory: 3000Mi

helm install emissary-ingress --namespace emissary datawire/emissary-ingress --version 8.9.1 -f emissary-values.yaml 
NAME: emissary-ingress
LAST DEPLOYED: Mon Apr  8 11:02:05 2024
NAMESPACE: emissary
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
-------------------------------------------------------------------------------
  Congratulations! You've successfully installed Emissary Ingress!

-------------------------------------------------------------------------------
To get the IP address of Emissary, run the following commands:
  export POD_NAME=$(kubectl get pods --namespace emissary -l "app=emissary-ingress,release=emissary-ingress" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:80

For help, visit our Slack at http://a8r.io/Slack or view the documentation online at https://www.getambassador.io.

2. Apply the qotm test service from getting started guide

   
   kubectl apply -f https://app.getambassador.io/yaml/v2-docs/3.9.1/quickstart/qotm.yaml

deployment.apps/quote created service/quote created

3. Create a self-signed cert following the steps from https://www.getambassador.io/docs/emissary/latest/howtos/tls-termination
```console
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -subj '/CN=ambassador-cert' -nodes

kubectl create secret tls tls-cert --cert=cert.pem --key=key.pem

secret/tls-cert created

4. Create the following listener configs (and port forward 443 on the service to 10443 locally)

   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Listener
   metadata:
   name: emissary-ingress-listener-8080
   namespace: emissary
   spec:
   port: 8080
   protocol: HTTP
   securityModel: XFP
   hostBinding:
   namespace:
     from: ALL
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Listener
   metadata:
   name: emissary-ingress-listener-8443
   namespace: emissary
   spec:
   port: 8443
   protocol: HTTPS
   securityModel: XFP
   hostBinding:
   namespace:
     from: ALL

5. Create these hosts

   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Host
   metadata:
   name: wildcard-host
   spec:
   hostname: "*"
   acmeProvider:
   authority: none
   tlsSecret:
   name: tls-cert
   mappingSelector:
   matchLabels:
     hostKind: wildcard-host
   requestPolicy:
   insecure:
     action: Route
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Host
   metadata:
   name: localhost2
   spec:
   hostname: localhost2
   tlsSecret:
   name: tls-cert
   mappingSelector:
   matchLabels:
     hostKind: localhost2
   requestPolicy:
   insecure:
     action: Route
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Host
   metadata:
   name: localhost-splat
   spec:
   hostname: "*.localhost"
   tlsSecret:
   name: tls-cert
   mappingSelector:
   matchLabels:
     hostKind: localhost-splat
   requestPolicy:
   insecure:
     action: Route

6. Create these mappings

   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Mapping
   metadata:
   labels:
   hostKind: wildcard-host
   name: quote-backend-wildcard
   namespace: default
   spec:
   docs:
   path: /.ambassador-internal/openapi-docs
   prefix: /backend/
   service: quote
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Mapping
   metadata:
   labels:
   hostKind: localhost2
   name: quote-backend
   namespace: default
   spec:
   docs:
   path: /.ambassador-internal/openapi-docs
   prefix: /backend/
   service: quote
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Mapping
   metadata:
   name: quote-backend-host
   labels:
   hostKind: localhost2
   spec:
   prefix: /
   service: quote
   docs:
   path: "/.ambassador-internal/openapi-docs"
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Mapping
   metadata:
   name: quote-backend-host-splat
   labels:
   hostKind: localhost-splat
   spec:
   prefix: /splat-only/
   service: quote
   docs:
   path: "/.ambassador-internal/openapi-docs"

7. Check which mappings work - results attached https_with_wildcard_test.txt But only the quote-backend-wildcard mapping is effective

   curl -kvv https://localhost:10443/backend/
   curl -kvv https://localhost:10443/ -H "Host:localhost2"
   curl -kvv https://localhost:10443/backend/ -H "Host:localhost2"
   curl -kvv https://localhost:10443/backend/ -H "Host:test.localhost"
   curl -kvv https://localhost:10443/splat-only/ -H "Host:test.localhost"

8. Delete host/wildcard-host

   k delete host/wildcard-host                                           
   host.getambassador.io "wildcard-host" deleted

9. Re-test, and we receive SSL errors on the mappings that should complete

   curl -kvv https://localhost:10443/ -H "Host:localhost2"
   *   Trying [::1]:10443...
   * Connected to localhost (::1) port 10443
   * ALPN: curl offers h2,http/1.1
   * (304) (OUT), TLS handshake, Client hello (1):
   * LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
   * Closing connection
   curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

10. Try the same curl with http on the https listener and we see a 200 response

    curl -kvv http://localhost:10443/ -H "Host:localhost2"
    *   Trying [::1]:10443...
    * Connected to localhost (::1) port 10443
    > GET / HTTP/1.1
    > Host:localhost2
    > User-Agent: curl/8.4.0
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < content-type: application/json
    < date: Mon, 08 Apr 2024 21:57:21 GMT
    < content-length: 131
    < x-envoy-upstream-service-time: 0
    < server: envoy
    <
    {
    "server": "chubby-kiwi-t82xhysy",
    "quote": "668: The Neighbor of the Beast.",
    "time": "2024-04-08T21:57:21.903616Z"
    * Connection #0 to host localhost left intact
    }%

Expected behavior Either, the listener accepts HTTPS connections without a * host being created, or more specific routes can use mappingSelector when a * host is provided.

In Step 7, all CURLs should have been 200 response. In step 9, we would expect a 200 response, but instead get ssl errors In step 10, we would not expect 200 response on http when the host has a TLS-secret and we're using the HTTPS listener

Versions (please complete the following information):

• Emissary: 3.9.1, Helm version 8.9.1
• Kubernetes environment: Rancher Desktop Kubernetes 1.29.3 and EKS 1.29

Additional context Add any other context about the problem here.

[cindymullins-dw]

We notice you're using Rancher and there's a possibility might be altering the YAML, so that's something we'd like to check. Can you run a kubectl get mapping for one of your Mappings here so we can take a look at that?

[dmaclaury]

We see this in EKS as well, but here is the requested output from my Rancher test environment:

k get mapping -A
NAMESPACE   NAME                       SOURCE HOST                      SOURCE PREFIX   DEST SERVICE   STATE   REASON
default     quote-backend-wildcard     _skip_mapping_with_empty_host_   /backend/       quote
default     quote-backend              _skip_mapping_with_empty_host_   /backend/       quote
default     quote-backend-host         _skip_mapping_with_empty_host_   /               quote
default     quote-backend-host-splat   _skip_mapping_with_empty_host_   /splat-only/    quote

k describe mapping -A
Name:         quote-backend-wildcard
Namespace:    default
Labels:       hostKind=wildcard-host
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:45Z
  Generation:          1
  Resource Version:    152376
  UID:                 405ec9da-6446-463c-bfdb-351aa40fb27f
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /backend/
  Service:  quote
Events:     <none>

Name:         quote-backend
Namespace:    default
Labels:       hostKind=localhost2
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:45Z
  Generation:          1
  Resource Version:    152377
  UID:                 f1e4d716-3ee7-4fdd-b983-707f0913813b
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /backend/
  Service:  quote
Events:     <none>

Name:         quote-backend-host
Namespace:    default
Labels:       hostKind=localhost2
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:45Z
  Generation:          1
  Resource Version:    152378
  UID:                 5afd128d-ea07-4e91-860e-2a53ae367e31
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /
  Service:  quote
Events:     <none>

Name:         quote-backend-host-splat
Namespace:    default
Labels:       hostKind=localhost-splat
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:46Z
  Generation:          1
  Resource Version:    152379
  UID:                 7feb8cc0-53dc-4930-9ad9-22151ee90e24
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /splat-only/
  Service:  quote
Events:     <none>

[cindymullins-dw]

Thanks, I think that looks ok. Can you try running this as well? kubectl get host wildcard-host -n ambassador -o yaml

[dmaclaury]

Here are all the hosts, they are in default namespace, but listeners are configured for ALL

k get hosts.getambassador.io -o yaml
apiVersion: v1
items:
- apiVersion: getambassador.io/v2
  kind: Host
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v3alpha1","kind":"Host","metadata":{"annotations":{},"name":"localhost2","namespace":"default"},"spec":{"hostname":"localhost2","mappingSelector":{"matchLabels":{"hostKind":"localhost2"}},"requestPolicy":{"insecure":{"action":"Route"}},"tlsSecret":{"name":"tls-cert"}}}
    creationTimestamp: "2024-04-17T20:49:20Z"
    generation: 1
    name: localhost2
    namespace: default
    resourceVersion: "152364"
    uid: 2d038cbe-6334-414d-928c-db845f18272a
  spec:
    ambassador_id:
    - --apiVersion-v3alpha1-only--default
    hostname: localhost2
    requestPolicy:
      insecure:
        action: Route
    selector:
      matchLabels:
        hostKind: localhost2
    tlsSecret:
      name: tls-cert
  status: {}
- apiVersion: getambassador.io/v2
  kind: Host
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v3alpha1","kind":"Host","metadata":{"annotations":{},"name":"localhost-splat","namespace":"default"},"spec":{"hostname":"*.localhost","mappingSelector":{"matchLabels":{"hostKind":"localhost-splat"}},"requestPolicy":{"insecure":{"action":"Route"}},"tlsSecret":{"name":"tls-cert"}}}
    creationTimestamp: "2024-04-17T20:49:21Z"
    generation: 1
    name: localhost-splat
    namespace: default
    resourceVersion: "152365"
    uid: 4f8ca933-1a20-43cb-baf2-22ceb7b89a6e
  spec:
    ambassador_id:
    - --apiVersion-v3alpha1-only--default
    hostname: '*.localhost'
    requestPolicy:
      insecure:
        action: Route
    selector:
      matchLabels:
        hostKind: localhost-splat
    tlsSecret:
      name: tls-cert
  status: {}
- apiVersion: getambassador.io/v2
  kind: Host
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v3alpha1","kind":"Host","metadata":{"annotations":{},"name":"wildcard-host","namespace":"default"},"spec":{"acmeProvider":{"authority":"none"},"hostname":"*","mappingSelector":{"matchLabels":{"hostKind":"wildcard-host"}},"requestPolicy":{"insecure":{"action":"Route"}},"tlsSecret":{"name":"tls-cert"}}}
    creationTimestamp: "2024-04-18T21:40:14Z"
    generation: 1
    name: wildcard-host
    namespace: default
    resourceVersion: "153787"
    uid: bf52ee0e-418d-4ef7-86f4-446f97fb8ca6
  spec:
    acmeProvider:
      authority: none
    ambassador_id:
    - --apiVersion-v3alpha1-only--default
    hostname: '*'
    requestPolicy:
      insecure:
        action: Route
    selector:
      matchLabels:
        hostKind: wildcard-host
    tlsSecret:
      name: tls-cert
  status: {}
kind: List
metadata:
  resourceVersion: ""

[cindymullins-dw]

Thanks for that. I did some research, and this seems to be a known issue: when setting mappingSelector on v3alpha1 CRDs, apiext(an Ambassador extension which converts resources to the v2 storage version) incorrectly handles the translation and stores the resource with an invalid Selector: field rather than mappingSelector. I see this in your yaml output as well.

Our recommendations for now are

1. Continue to use Selector: on the v2 resources -- Do not attempt to upgrade to v3alpha1 until the issue has been resolved with mappingSelector:
2. Manually edit the Selector: field on the v3alpha1 resource to be a mappingSelector: field. This breaks CI/CD pieplines when you introduce manual edits, and is highly discouraged as a best practice.
3. Ignore using Selector: / mappingSelector: altogether for now, and associate Mappings and Hosts by specifying Hostname: fields on the Mapping resources.