IPv6 dual-stack not working on IPv6 first clusters due to IPv4 fixed listeners

alessiodionisi commented 1 day ago

Description:

I'm testing the latest release (v1.2.0-rc.1) that adds IPv6 and dual-stack support to Envoy Gateway. On my IPv6-first cluster the pods created for the gateway resource fail the startup probe because, by default, the listener is listening only on IPv4 instead of IPv6 or all interfaces.

https://github.com/envoyproxy/gateway/blob/19eb5f58c7d10228e5225ce8752e3c570c23e593/internal/xds/bootstrap/bootstrap.go#L29

https://github.com/envoyproxy/gateway/blob/19eb5f58c7d10228e5225ce8752e3c570c23e593/internal/xds/bootstrap/bootstrap.go#L42

These IPs are not customizable and are used directly in the bootstrap.yaml.tpl template, example: https://github.com/envoyproxy/gateway/blob/19eb5f58c7d10228e5225ce8752e3c570c23e593/internal/xds/bootstrap/bootstrap.yaml.tpl#L68

Repro steps:

Deploy Envoy Gateway on the cluster
Configure a resource of type Gateway

Environment:

Kubernetes cluster configured in dual-stack with IPv6 as primary:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
...
networking:
podSubnet: fd00:10:244::/48,10.244.0.0/16
serviceSubnet: fd00:10:96::/112,10.96.0.0/12

Envoy Gateway version: v1.2.0-rc.1

Logs:

Startup probe failed: Get "http://[fd00:10:244:2::5b]:19001/ready": dial tcp [fd00:10:244:2::5b]:19001: connect: connection refused

arkodg commented 1 day ago

thanks for testing this out @alessiodionisi ! we'll get this resolved by v1.2.0

arkodg commented 1 day ago

ptal @juwon8891

alessiodionisi commented 1 day ago

Quick update:

I've switched to v0.0.0-latest and patched the bootstrap configuration to listen on IPv6 instead of IPv4, and everything works correctly.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
...
spec:
  ipFamily: DualStack
  bootstrap:
    value: |
      yaml content below

Default values with 127.0.0.1 replaced to ::1 and 0.0.0.0 to :::

admin:
  accessLog:
    - name: envoy.access_loggers.file
      typedConfig:
        "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
        path: /dev/null
  address:
    socketAddress:
      address: ::1
      portValue: 19000
dynamicResources:
  adsConfig:
    apiType: DELTA_GRPC
    grpcServices:
      - envoyGrpc:
          clusterName: xds_cluster
    setNodeOnFirstMessageOnly: true
    transportApiVersion: V3
  cdsConfig:
    ads: {}
    resourceApiVersion: V3
  ldsConfig:
    ads: {}
    resourceApiVersion: V3
layeredRuntime:
  layers:
    - name: global_config
      staticLayer:
        envoy.restart_features.use_eds_cache_for_ads: true
        re2.max_program_size.error_level: 4294967295
        re2.max_program_size.warn_level: 1000
overloadManager:
  refreshInterval: 0.250s
  resourceMonitors:
    - name: envoy.resource_monitors.global_downstream_max_connections
      typedConfig:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
        maxActiveDownstreamConnections: "50000"
staticResources:
  clusters:
    - connectTimeout: 0.250s
      loadAssignment:
        clusterName: prometheus_stats
        endpoints:
          - lbEndpoints:
              - endpoint:
                  address:
                    socketAddress:
                      address: ::1
                      portValue: 19000
      name: prometheus_stats
      type: STATIC
    - connectTimeout: 10s
      loadAssignment:
        clusterName: xds_cluster
        endpoints:
          - lbEndpoints:
              - endpoint:
                  address:
                    socketAddress:
                      address: envoy-gateway
                      portValue: 18000
                loadBalancingWeight: 1
            loadBalancingWeight: 1
      name: xds_cluster
      transportSocket:
        name: envoy.transport_sockets.tls
        typedConfig:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          commonTlsContext:
            tlsCertificateSdsSecretConfigs:
              - name: xds_certificate
                sdsConfig:
                  pathConfigSource:
                    path: /sds/xds-certificate.json
                  resourceApiVersion: V3
            tlsParams:
              tlsMaximumProtocolVersion: TLSv1_3
            validationContextSdsSecretConfig:
              name: xds_trusted_ca
              sdsConfig:
                pathConfigSource:
                  path: /sds/xds-trusted-ca.json
                resourceApiVersion: V3
      type: STRICT_DNS
      typedExtensionProtocolOptions:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicitHttpConfig:
            http2ProtocolOptions:
              connectionKeepalive:
                interval: 30s
                timeout: 5s
    - connectTimeout: 10s
      loadAssignment:
        clusterName: wasm_cluster
        endpoints:
          - lbEndpoints:
              - endpoint:
                  address:
                    socketAddress:
                      address: envoy-gateway
                      portValue: 18002
                loadBalancingWeight: 1
            loadBalancingWeight: 1
      name: wasm_cluster
      transportSocket:
        name: envoy.transport_sockets.tls
        typedConfig:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          commonTlsContext:
            tlsCertificateSdsSecretConfigs:
              - name: xds_certificate
                sdsConfig:
                  pathConfigSource:
                    path: /sds/xds-certificate.json
                  resourceApiVersion: V3
            tlsParams:
              tlsMaximumProtocolVersion: TLSv1_3
            validationContextSdsSecretConfig:
              name: xds_trusted_ca
              sdsConfig:
                pathConfigSource:
                  path: /sds/xds-trusted-ca.json
                resourceApiVersion: V3
      type: STRICT_DNS
      typedExtensionProtocolOptions:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicitHttpConfig:
            http2ProtocolOptions: {}
  listeners:
    - address:
        socketAddress:
          address: "::"
          portValue: 19001
      filterChains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typedConfig:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                httpFilters:
                  - name: envoy.filters.http.health_check
                    typedConfig:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
                      headers:
                        - name: :path
                          stringMatch:
                            exact: /ready
                      passThroughMode: false
                  - name: envoy.filters.http.router
                    typedConfig:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                routeConfig:
                  name: local_route
                  virtualHosts:
                    - domains:
                        - "*"
                      name: prometheus_stats
                      routes:
                        - match:
                            prefix: /stats/prometheus
                          route:
                            cluster: prometheus_stats
                statPrefix: eg-ready-http
      name: envoy-gateway-proxy-ready-0.0.0.0-19001

zirain commented 1 day ago

https://github.com/envoyproxy/gateway/pull/4550 will fix some cases, but I'm stucked with some users case.

envoyproxy / gateway

IPv6 dual-stack not working on IPv6 first clusters due to IPv4 fixed listeners #4565