Handling wildcards in hostnames of called external services

[tomtomtomtom44]

Description

Hello, We need to call external services whose hostname is not known in advance (retrieved in base by a query engine), only the last part of the FQDN is static so URLs are of the type https://*.[static_part_of_the_fqdn].com. We want these external calls to be referenced in the mesh as ExternalService to be able to pass through the egress and also be able to apply policies to them Do not hesitate if you need more details, thank you!

[michaelbeaumont]

Triage: Envoy DNS may not be compatible with wildcards

This definitely makes sense. As a short term workaround, it might be possible to autogenerate the list of ExternalServices, if it's a queryable or closed list

[tomtomtomtom44]

Hello, first of all thank you for considering my request. On the other hand, the workaround is not possible for us because the list of hosts is too large and dynamic. It seems that the Istio product faced the same problem and found a solution if it can help (cf https://istio.io/v1.11/docs/tasks/traffic-management/egress/wildcard-egress-hosts/)

[tomtomtomtom44]

Hello, I was wondering if you've any news about this issue? Reading the Envoy 1.27 doc supported by Kuma, I said to myself that the "HTTP dynamic forward proxy" feature could help answer the problem (see https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http_proxy#arch- overview-http-dynamic-forward-proxy") , if we could apply it to ExternalServices with a subdomain. This would require rewriting the URL of the incoming service with only domain of the FQDN as hostname and passing [subdomain.domain] as a header (via host_rewrite_header parameter). I thought I could test it via a "ProxyTemplate" policy but this policy cannot be configured on the dataplane of the egress from what I saw. Thank you for your feedback

[tomtomtomtom44]

Hello,

We found a workaround that seems to resolve part of the issue for us. Although the modification is not currently used in production, I would like to share the information if it would be useful to anyone.

This bypass allows us to route calls to wildcard services through an egress and also to see their metrics appear in grafana cloud. The only problem left is that the policies must be redefined in the ProxyTemplate for certain of our policies since the ProxyTemplate is applied after the other policies, in our case we lose part of the information from the MeshAccessLog which we use to trace the logs, we redefine it in ProxyTemplate but some variables are not available since they seem only available when the policy is applied (%KUMA_...)

The idea is to add a "listener_filter" of type "envoy.filters.listener.tls_inspector" on "outbound:passthrough:ipv4" listener which allows TLS wildcard network calls to be filtered via the SNI protocol. We had to create an additional egress with Envoy to ensure not to create an edge effect on the existing Kuma egress, and used ProxyTemplate to customize the current mesh.

Also, to send the request to the original hostname, we used this Envoy functionality (https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/sni_dynamic_forward_proxy_filter) configured on our egress which allows the request to be sent to the original hostname (whatever the value of the subdomain corresponding to the wildcard)

Attached are pictures of before/after modification

Here are the ProxyTemplate modifications added :

For each wildcard service, we define an Envoy cluster which routes to the egress proxy, example:

cluster:
          operation: add
          value: |
            "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
            name: [wildcard-service-fqdn]
            altStatName: [wildcard-service-fqdn]
            type: STRICT_DNS
            connectTimeout: 10s
            loadAssignment:
              clusterName: [wildcard-service-name]
              endpoints:
              - lbEndpoints:
                - endpoint:
                    address:
                      socketAddress:
                        address: envoy-egress-proxy.[namespace_kuma]
                        portValue: 84XX
            dnsLookupFamily: V4_ONLY

The outbound:passthrough:ipv4 listener was redefined (operation: add), because patch mode, which was also possible, did not produce a result.

A tls_inspector object has been added to the listener and allows flows to be filtered by SNI

listener_filters:
              - name: "envoy.filters.listener.tls_inspector"
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector

For each wildcard service, we define a TCP filter which is applied based on the SNI (server_names variable) which is filtered by the TLS inspector, this filter redirects the flow to the previously defined cluster

- filter_chain_match:
                transport_protocol: tls
                server_names:
                - "*.[wildcard_service_fqdn]"
              filters:
                - name: envoy.filters.network.tcp_proxy
                  typed_config:
                    '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                    stat_prefix: [wildcard_proxy_name]
                    cluster: [wildcard_service_name]
                    accessLog:
                    - name: envoy.access_loggers.file
                      typedConfig:
                        "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                        path: "/dev/stdout"
                        logFormat:
                          textFormatSource:
                            inlineString: '[%START_TIME%] default "%REQ(:method)% %REQ(x-envoy-original-path?:path)%
                              %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT%
                              %DURATION% %RESP(x-envoy-upstream-service-time)% "%REQ(x-forwarded-for)%"
                              "%REQ(user-agent)%" traceID:%REQ(x-b3-traceid)% "%REQ(x-request-id)%" "%REQ(:authority)%"
                              "kuma_source_service" "external-[external-service_name]" "kuma_source_address_without_port" "%UPSTREAM_HOST%"

[github-actions[bot]]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[tomtomtomtom44]

Hello, I still find it relevant for this management of wildcards in external services to be natively supported by Kuma, the workaround found adds too much complexity to our infrastructure I find and a loss of information in the logs (rewriting access policies in a ProxyTemplate)

[github-actions[bot]]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[benjamindelannoy]

We are still in need of wildcard support.

[slonka]

Hi, we're going to be looking into this as part of new external service issue - https://github.com/kumahq/kuma/issues/6330 will probably update soon.

[slonka]

@tomtomtomtom44 can you try https://kuma.io/docs/2.8.x/policies/meshpassthrough/#meshpassthrough and let us know what you think?