Emissary uses wrong TLSContext with identical name in different namespace

[dschuldt]

Describe the bug If two TLSContexts are present with an identical name but in different namespaces, Emissary does not use the one which resides in the same namespace as the Host resource.

To Reproduce

1. Create two namespaces
2. Create a TLSContexts and Hosts with the same names in these namespaces
3. Create two different certificates with different hostnames and configure the ambassador resources accordingly
4. You'll experience a host mismatch error and the wrong certificate is presented to user requests

Expected behavior Emissary should use the TLSContext within the same namespace as the Host

Versions (please complete the following information):

• docker.io/emissaryingress/emissary:3.6.0
• Rancher RKE Cluster v1.24.4

Additional context I'm sorry if I misunderstood the TLSContext resource. Is it not bound to a specific namespace?

Best Denis

[cindymullins-dw]

HI @dschuldt , are you linking the Host to the TLSContext you want it to use? You can configure that explicitly, just make sure the The Host and the TLSContext refer to the same Kubernetes Secret.

[dschuldt]

Hi @cindymullins-dw,

yes, the Host/TLSContext are linked. The secret is the same in both resources. The secret also exists in both namespaces with an identical name. I'll provide a redigated code example.

[dschuldt]

Here we go.

These are the TLSContexts (Same name, but different namespace and hostname):

apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
  name: my-tlscontext
  namespace: my-namespace-1
spec:
  alpn_protocols: h2,http/1.1
  hosts:
  - my-fqdn-1
  min_tls_version: v1.2
  secret: my-secret
---
apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
  name: my-tlscontext
  namespace: my-namespace-2
spec:
  alpn_protocols: h2,http/1.1
  hosts:
  - my-fqdn-2
  min_tls_version: v1.2
  secret: my-secret

The Hosts (again, same name but different namespace and different hostname):

apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: my-host
  namespace: my-namespace-1
spec:
  hostname: my-fqdn-1
  requestPolicy:
    insecure:
      action: Redirect
  tlsContext:
    name: my-tlscontext
  tlsSecret:
    name: my-secret
---
apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: my-host
  namespace: my-namespace-2
spec:
  hostname: my-fqdn-2
  requestPolicy:
    insecure:
      action: Redirect
  tlsContext:
    name: my-tlscontext
  tlsSecret:
    name: my-secret

Here are the used Secrets (same name but different namespace and content:

apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
  name: my-secret
  namespace: my-namespace-1
data:
  tls.crt: ...
  tls.key: ...
---
apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
  name: my-secret
  namespace: my-namespace-2
data:
  tls.crt: ...
  tls.key: ...

With this configuration, Emissary links the TLSContext from my-namespace-1 to the Host from my-namespace-2 (or vice versa). If you rename a TLSContext and reconfigure any host to use the new name, everything works fine.

Hope this helps.

[dschuldt]

Hi @cindymullins-dw, anything I can provide additionally? This issue makes our live harder than it needs to be :-)

Best Denis

[cindymullins-dw]

It should look for the secret that's in the same namespace as the Host by default. But as a test can you try specifying the namespace you want under the Host like the docs example?

apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: example-host
spec:
  hostname: host.example.com
  acmeProvider:
    authority: none
  tlsSecret:
    name: host-secret
    **namespace: example**

[dschuldt]

Hi @cindymullins-dw,

thank you but I'm afraid the behaviour remains the same; a TLSContect from a different Namespace is used..

[eevdev]

Someone on my team noticed this problem. They were using the same name (ie. metadata.name) of Host and TLSContext resources in different namespaces, causing conflicts for both. Not only was the wrong cert served but traffic was also routed to both services backends for both hosts.

In a multi-tenancy environment where multiple services/teams share the same Emissary, the prospect of others messing up your service is of course very bad.

This was observed on the latest version 3.8.1 (chart 8.8.1), as well as on 3.3.1.

[eevdev]

FYI, this was seen in envoy.json:

            "name": "httpshost-<host.metadata.name>.<namespace A>",
                  "tls_certificates": [
                    {
                      "certificate_chain": {
                        "filename": "/ambassador/snapshots/<namespace B>/secrets-decoded/api-xyz-tls/abc.crt"
                      },
                      "private_key": {
                        "filename": "/ambassador/snapshots/<namespace B>/secrets-decoded/api-xyz-tls/abc.key"
                      }
                    }
                  ],

[martysweet]

I've just run into this issue as well. The TLSContext from the incorrect namespace is being used.

<RichStatus BAD error=\"Hosts mismatch between Host web.stg (accepted hosts: ['app.stg.cluster.io', 'web.stg']) and TLSContext web (hosts: ['app.int.cluster.io'])\" hostname='emissary-ingress-...' version='3.8.2'>"}   

Both TLSContext objects exist correctly in snapshots/snapshot.yaml, with the same metadata.name, but different metadata.namespace values, as expected.

Both Secret objects exist correctly in snapshots/snapshot.yaml.

I'm not sure if this is the correct codepath, but it seems (at least this) TLSContext logic is not namespace aware: https://github.com/emissary-ingress/emissary/blob/master/python/ambassador/ir/ir.py#L860-L864

Which is used here as part of Host resolution: https://github.com/emissary-ingress/emissary/blob/master/python/ambassador/ir/irhost.py#L364-L373

[rgs1]

For anyone else reading this, we bumped into a similar issue when setting up upstream TLSContexts with the same name in different namespaces.

In this scenario, we had parallel Emissary installations but we forgot to properly set up the corresponding ambassador_id on each TLSContext, so there was a collision. I didn't check the code, but I guess some resources are scoped by namespace, others by ambassador_id and others by both.