TCPMapping not applying host/tls configuration on specific port

[Mokto]

Describe the bug Hi! I've been struggling with this issue for the past afternoon, and after a lot of researches, I'm starting to suspect this is a bug. I can't apply a TCPMapping tls/host configuration. Without TLS I can connect to my rabbitmq server, but I can't use TLS termination. It is also impossible to use SNI (seems logical as tls doesn't work). By the way any webservice access works in HTTPS mode.

To Reproduce Steps to reproduce the behavior:

0. Install Ambassador with an additional port on the loadbalancer service :

     - name: rabbitmq
       port: 5671
       targetPort: 5671
       protocol: TCP

1. TLSContext configuration :

       apiVersion: ambassador/v1
       kind: TLSContext
       name: ocean-certs
       secret: ocean-prod-certs
       hosts: ["*.ocean.io"]
       redirect_cleartext_from: 8080
       alpn_protocols: h2,http/1.1

2. Apply a TCPMapping

   apiVersion: getambassador.io/v1
   kind: TCPMapping
   metadata:
   name: core-production-rabbitmq
   spec:
   port: 5671
   service: core-production-rabbitmq-rabbitmq-ha.namespace:5672
   host: rabbitmq.ocean.io

The TLSContext seems not to be applied to the 5671 listener. See my envoy configuration :

    "@type": "/envoy.config.bootstrap.v2.Bootstrap",
    "static_resources": {
        "clusters": [
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8877",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8877,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8877",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_core_production_rabbitmq_rabbitm-0",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "core-production-rabbitmq-rabbitmq-ha.core-production",
                                                "port_value": 5672,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_core_production_rabbitmq_rabbitm-0",
                "type": "STRICT_DNS"
            }
        ],
        "listeners": [
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 8443,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "config": {
                                    "access_log": [
                                        {
                                            "config": {
                                                "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
                                                "path": "/dev/fd/1"
                                            },
                                            "name": "envoy.file_access_log"
                                        }
                                    ],
                                    "http_filters": [
                                        {
                                            "name": "envoy.cors"
                                        },
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "http_protocol_options": {
                                        "accept_http_10": false
                                    },
                                    "normalize_path": true,
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "backend",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_ready",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877",
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_alive",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877",
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877",
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "server_name": "envoy",
                                    "stat_prefix": "ingress_http",
                                    "use_remote_address": true,
                                    "xff_num_trusted_hops": 0
                                },
                                "name": "envoy.http_connection_manager"
                            }
                        ],
                        "tls_context": {
                            "common_tls_context": {
                                "alpn_protocols": [
                                    "h2,http/1.1"
                                ],
                                "tls_certificates": [
                                    {
                                        "certificate_chain": {
                                            "filename": "/ambassador/snapshots/istio-system/secrets-decoded/ocean-prod-certs/XXXXX.crt"
                                        },
                                        "private_key": {
                                            "filename": "/ambassador/snapshots/istio-system/secrets-decoded/ocean-prod-certs/XXXXX.key"
                                        }
                                    }
                                ]
                            }
                        },
                        "use_proxy_proto": false
                    }
                ],
                "name": "ambassador-listener-8443"
            },
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 8080,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "config": {
                                    "access_log": null,
                                    "http_filters": [
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "http_protocol_options": {
                                        "accept_http_10": false
                                    },
                                    "normalize_path": true,
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "backend",
                                                "require_tls": "EXTERNAL_ONLY",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "prefix": "/"
                                                        },
                                                        "redirect": {
                                                            "https_redirect": true
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "server_name": "envoy",
                                    "stat_prefix": "ingress_http",
                                    "use_remote_address": true,
                                    "xff_num_trusted_hops": 0
                                },
                                "name": "envoy.http_connection_manager"
                            }
                        ],
                        "use_proxy_proto": false
                    }
                ],
                "name": "redirect_listener"
            },
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 5671,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "config": {
                                    "stat_prefix": "ingress_tcp_5671",
                                    "weighted_clusters": {
                                        "clusters": [
                                            {
                                                "name": "cluster_core_production_rabbitmq_rabbitm-0",
                                                "weight": 100
                                            }
                                        ]
                                    }
                                },
                                "name": "envoy.tcp_proxy"
                            }
                        ]
                    }
                ],
                "name": "listener-0.0.0.0-5671"
            }
        ]
    }
}

Versions (please complete the following information):

• Ambassador: 0.84.1
• Kubernetes environment: EKS
• Version: 1.14

Thanks for your help!

[ovk]

I encountered exactly the same issue on Ambassador 0.86.0 - TLS context is not applied for a TCP Mapping with host specified.

[ovk]

Okay, so I've done a little bit of digging and I think I found the problem.

For V2TCPListener TLS context looked up in the group and it's never there, so the following branch is never taken:

https://github.com/datawire/ambassador/blob/11754e7249ceb4d8de5b36def7fefbb6e0349395/python/ambassador/envoy/v2/v2listener.py#L412

In contrast, for V2Listener TLS context is taken from the config:

https://github.com/datawire/ambassador/blob/11754e7249ceb4d8de5b36def7fefbb6e0349395/python/ambassador/envoy/v2/v2listener.py#L749

I've tried doing a quick patch of v2listener.py to get TLS context from the config.ir.get_tls_contexts() for V2TCPListener and it seems to solve the issue: the TLS context now appears in the envoy config for the TCP mapping and the TCP mapping starts to work correctly (i.e. doing TLS termination as it supposed to).

Not sure what is the "proper" fix for this, but hopefully this info will help developers to address the issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[daddz]

I'm running into the same problem with the following setup:

• 1 TLSContext with wild-card certificate
• 2 TCPMappings with same port but different hosts

Error message:

listener '0.0.0.0:5672': multiple filter chains with the same matching rules are defined

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kflynn]

@ovk, I'm very sorry for missing your comment! Let me try your fix.

[biancajiang]

any updates @kflynn? We are running into exactly this issue too.

[alexmulholland]

Hi @kflynn had any luck with a fix? We are hitting this in our project. Thanks!

[Jethzabell]

Hello, @kflynn any update on this?

[weitzjdevk]

Any updates on this? We are trying amassador 1.5.5 @nbkrause ?

[weitzjdevk]

Just a hint for debugging purposes:

Don't set an ambassador_id on the pods or find a way how to specify the ambassador_id for a TCPMapping. This way it mostly works.

[zaroman]

Hi, is a fix for this issue planned in the near future? We are hitting this in our project. Thanks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ovk]

Not stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Mokto]

Not stale

[itay]

Also running into a similar issue

[mdekoster]

We are facing the same issue. Is there any update?

[bemipefe]

Same issue with version 1.14.2. I defined just one Hostobject and a TCPMappingobject. The correct certificate is returned but the HTTP response is 404 Not Found.

---
apiVersion: getambassador.io/v2
kind: TCPMapping
metadata:
  name:  ambassador-mapping
  namespace: myns
  labels:
    map: ambassador-mapping
spec:
  ambassador_id: [ "ambassador1" ]
  port: 8443
  host: test.ambassador
  host_regex: false
  service: test-deploy-service:8443
  tls: true

apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: ambassador-host
  namespace: myns
spec:
  ambassador_id: [ "ambassador1" ]
  hostname: test.ambassador
  tlsSecret:
    name: mysecret-tls

When the Hostobject is created this is added under spec:

  selector:
    matchLabels:
      hostname: test.ambassador

However there is no label nor field named "hostname" in the TCPMapping object. How is it supposed to work ?

[joecrop]

I am having this problem as well. Is anyone aware of a workaround?

[cindymullins-dw]

Has anyone had this experience on Emissary or AES 2.x or greater? Checking the latest docs here, @bemipefe 's config looks correct to me. First, Emissary-ingress checks for any Host resources with TLS configured whose Host.spec.hostname glob-matches the TCPMapping.spec.host; if such a Host exists, then its TLS certificate and configuration is used..

[gnanderson]

yes, we experience this trying to TLS terminate a TCP Mapping on Emissary v 3.4.1

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: arc-github-hook
  namespace: actions-runner-system
spec:
  hostname: "arc-github-hook.example.com"
  tlsSecret:
    name: webhook-tls-cert
  tls:
    min_tls_version: v1.2
    max_tls_version: v1.3
    alpn_protocols: h2[, http/1.1]
    ecdh_curves:
      - X25519
      - P-256
---
apiVersion: getambassador.io/v3alpha1
kind: TCPMapping
metadata:
  name:  arc-github-hook
  namespace: actions-runner-system
spec:
  port: 9443
  host: "arc-github-hook.example.com"
  service: http://actions-runner-controller-github-webhook-server:8080

Debug log output:

[2023-02-15 12:40:55.060][37][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:86] tls inspector: new connection accepted
[2023-02-15 12:40:55.060][37][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:117] tls:onServerName(), requestedServerName: arc-github-hook.example.com
[2023-02-15 12:40:55.060][37][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:199] [C148859] new tcp proxy session
[2023-02-15 12:40:55.061][37][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:371] [C148859] Creating connection to cluster cluster_actions_runner_controller_github-D3A891DE11DE7378-0
[2023-02-15 12:40:55.061][37][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1731] no healthy host for TCP connection pool
[2023-02-15 12:40:55.061][37][debug][connection] [source/common/network/connection_impl.cc:139] [C148859] closing data_to_write=0 type=1
[2023-02-15 12:40:55.061][37][debug][connection] [source/common/network/connection_impl.cc:250] [C148859] closing socket: 1

[cindymullins-dw]

Ok, thanks for the confirmation. I'll flag this as a community reported bug.