search

envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway
https://gateway.envoyproxy.io
Apache License 2.0
1.58k stars 344 forks source link

Envoy's port is not listening in HostNetwork #2813

Closed imroc closed 7 months ago

imroc commented 7 months ago

Description:

Envoy's port is not listening if gw pod is set to use HostNetwork, and also there are some error log in envoy pod.

Repro steps:

Use EnvoyProxy CR to declare gw pod using HostNetwork, like this:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg2
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: eg2
    namespace: envoy-gateway-system

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg2
  namespace: envoy-gateway-system
spec:
  shutdown:
    drainTimeout: "1s"
    minDrainDuration: "1s"
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
      envoyDeployment:
        replicas: 1
        container:
          resources: {}
        pod:
          hostNetwork: true

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg2
  listeners:
    - name: test
      protocol: HTTPS
      port: 8899
      tls:
        certificateRefs:
          - kind: Secret
            group: ""
            name: test-crt-secret
      allowedRoutes:
        namespaces:
          from: All

And the gw pod yaml:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/path: /stats/prometheus
    prometheus.io/port: "19001"
    prometheus.io/scrape: "true"
  creationTimestamp: "2024-03-07T03:12:19Z"
  generateName: envoy-envoy-gateway-system-test-c5654c8e-5788f75495-
  labels:
    app.kubernetes.io/component: proxy
    app.kubernetes.io/managed-by: envoy-gateway
    app.kubernetes.io/name: envoy
    gateway.envoyproxy.io/owning-gateway-name: test
    gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
    pod-template-hash: 5788f75495
  name: envoy-envoy-gateway-system-test-c5654c8e-5788f75495-qvjpm
  namespace: envoy-gateway-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: envoy-envoy-gateway-system-test-c5654c8e-5788f75495
    uid: 962cac76-47c1-4a30-9320-427225e593b5
  resourceVersion: "26347853154"
  uid: c636dda3-ff23-4026-893d-f0094f8a479e
spec:
  automountServiceAccountToken: false
  containers:
  - args:
    - --service-cluster envoy-gateway-system/test
    - --service-node $(ENVOY_POD_NAME)
    - |
      --config-yaml admin:
        access_log:
        - name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/null
        address:
          socket_address:
            address: 127.0.0.1
            port_value: 19000
      layered_runtime:
        layers:
        - name: global_config
          static_layer:
            envoy.restart_features.use_eds_cache_for_ads: true
            re2.max_program_size.error_level: 4294967295
            re2.max_program_size.warn_level: 1000
      dynamic_resources:
        ads_config:
          api_type: DELTA_GRPC
          transport_api_version: V3
          grpc_services:
          - envoy_grpc:
              cluster_name: xds_cluster
          set_node_on_first_message_only: true
        lds_config:
          ads: {}
          resource_api_version: V3
        cds_config:
          ads: {}
          resource_api_version: V3
      static_resources:
        listeners:
        - name: envoy-gateway-proxy-ready-0.0.0.0-19001
          address:
            socket_address:
              address: 0.0.0.0
              port_value: 19001
              protocol: TCP
          filter_chains:
          - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: eg-ready-http
                route_config:
                  name: local_route
                  virtual_hosts:
                  - name: prometheus_stats
                    domains:
                    - "*"
                    routes:
                    - match:
                        prefix: /stats/prometheus
                      route:
                        cluster: prometheus_stats
                http_filters:
                - name: envoy.filters.http.health_check
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
                    pass_through_mode: false
                    headers:
                    - name: ":path"
                      string_match:
                        exact: /ready
                - name: envoy.filters.http.router
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
        clusters:
        - name: prometheus_stats
          connect_timeout: 0.250s
          type: STATIC
          lb_policy: ROUND_ROBIN
          load_assignment:
            cluster_name: prometheus_stats
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 19000
        - connect_timeout: 10s
          load_assignment:
            cluster_name: xds_cluster
            endpoints:
            - load_balancing_weight: 1
              lb_endpoints:
              - load_balancing_weight: 1
                endpoint:
                  address:
                    socket_address:
                      address: envoy-gateway
                      port_value: 18000
          typed_extension_protocol_options:
            envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
              "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
              explicit_http_config:
                http2_protocol_options:
                  connection_keepalive:
                    interval: 30s
                    timeout: 5s
          name: xds_cluster
          type: STRICT_DNS
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
              common_tls_context:
                tls_params:
                  tls_maximum_protocol_version: TLSv1_3
                tls_certificate_sds_secret_configs:
                - name: xds_certificate
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-certificate.json"
                    resource_api_version: V3
                validation_context_sds_secret_config:
                  name: xds_trusted_ca
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-trusted-ca.json"
                    resource_api_version: V3
    - --log-level warn
    - --cpuset-threads
    command:
    - envoy
    env:
    - name: ENVOY_GATEWAY_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ENVOY_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: envoyproxy/envoy:distroless-v1.29-latest
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        httpGet:
          path: /shutdown/ready
          port: 19002
          scheme: HTTP
    name: envoy
    ports:
    - containerPort: 8899
      hostPort: 8899
      name: test-9f86d081
      protocol: TCP
    - containerPort: 19001
      hostPort: 19001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ready
        port: 19001
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /certs
      name: certs
      readOnly: true
    - mountPath: /sds
      name: sds
  - args:
    - envoy
    - shutdown-manager
    - --ready-timeout=11s
    command:
    - envoy-gateway
    env:
    - name: ENVOY_GATEWAY_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ENVOY_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: envoyproxy/gateway-dev:7d48480
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - envoy-gateway
          - envoy
          - shutdown
          - --drain-timeout=1s
          - --min-drain-duration=1s
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: 19002
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: shutdown-manager
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: 19002
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      requests:
        cpu: 10m
        memory: 32Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostNetwork: true
  nodeName: 10.10.11.17
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: envoy-envoy-gateway-system-test-c5654c8e
  serviceAccountName: envoy-envoy-gateway-system-test-c5654c8e
  terminationGracePeriodSeconds: 301
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: certs
    secret:
      defaultMode: 420
      secretName: envoy
  - configMap:
      defaultMode: 420
      items:
      - key: xds-trusted-ca.json
        path: xds-trusted-ca.json
      - key: xds-certificate.json
        path: xds-certificate.json
      name: envoy-envoy-gateway-system-test-c5654c8e
      optional: false
    name: sds
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-03-07T03:12:19Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-03-07T03:13:00Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-03-07T03:13:00Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-03-07T03:12:19Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://ed7660459ee578a8ae23754f1e040b344e51acd2e076c80ba173193451e0811e
    image: docker.io/envoyproxy/envoy:distroless-v1.29-latest
    imageID: docker.io/envoyproxy/envoy@sha256:76dcbc0509d51001da1c94937b4498bd30ae6eb074d98863336c522311b19fa4
    lastState: {}
    name: envoy
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-03-07T03:12:20Z"
  - containerID: containerd://d5f018ef18676d84fe526fe1a60498a579cd47beded39e3de1645eb55fbd4396
    image: docker.io/envoyproxy/gateway-dev:7d48480
    imageID: docker.io/envoyproxy/gateway-dev@sha256:4b217e42eceab975166ab9676f7ad524fa271abfa1a97fd30b01c87b77d8cbdb
    lastState: {}
    name: shutdown-manager
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-03-07T03:12:20Z"
  hostIP: 10.10.11.17
  phase: Running
  podIP: 10.10.11.17
  podIPs:
  - ip: 10.10.11.17
  qosClass: Burstable
  startTime: "2024-03-07T03:12:19Z"

The envoy should listen 8899 as it is declared in Gateway CR (envoy listens the same port if Gateway's port > 1024), but envoy did not.

Environment: envoygateway version: v1.0.0-rc.1 Tested both on single node k3s and the cloud hosted Tencent Kubenretes Engine, got the same result.

Logs: Logs in envoy container:

[2024-03-07 03:12:20.518][1][warning][main] [source/server/server.cc:910] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.res
ource_monitors.downstream_connections` resource monitor.
[2024-03-07 03:12:35.701][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.co
nfig.cluster.v3.Cluster
[2024-03-07 03:12:50.700][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.co
nfig.listener.v3.Listener
[2024-03-07 03:12:53.005][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 32s ago
: 14, no healthy upstream
[2024-03-07 03:12:58.650][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 37s ago
: 14, no healthy upstream
[2024-03-07 03:13:11.834][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 51s ago
: 14, no healthy upstream
[2024-03-07 03:13:21.367][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 60s ago
: 14, no healthy upstream
[2024-03-07 03:13:35.482][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 74s ago
: 14, no healthy upstream
[2024-03-07 03:13:50.977][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 90s ago
: 14, no healthy upstream
[2024-03-07 03:14:06.127][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 105s ag
o: 14, no healthy upstream
[2024-03-07 03:14:16.377][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 115s ag
o: 14, no healthy upstream
[2024-03-07 03:14:27.038][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 126s ag
o: 14, no healthy upstream
[2024-03-07 03:14:38.318][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 137s ag
o: 14, no healthy upstream
[2024-03-07 03:14:39.265][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 138s ag
o: 14, no healthy upstream
[2024-03-07 03:14:58.874][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 158s ag
o: 14, no healthy upstream
[2024-03-07 03:15:09.898][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 169s ag
o: 14, no healthy upstream
[2024-03-07 03:15:33.649][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 192s ag
o: 14, no healthy upstream
[2024-03-07 03:15:48.446][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 207s ag
o: 14, no healthy upstream
[2024-03-07 03:16:00.839][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 220s ag
o: 14, no healthy upstream
[2024-03-07 03:16:07.694][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 226s ag
o: 14, no healthy upstream
[2024-03-07 03:16:30.617][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 249s ag
o: 14, no healthy upstream
[2024-03-07 03:16:55.820][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 275s ag
o: 14, no healthy upstream
[2024-03-07 03:17:04.787][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 284s ag
o: 14, no healthy upstream
[2024-03-07 03:17:07.830][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 287s ag
o: 14, no healthy upstream
[2024-03-07 03:17:31.054][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 310s ag
o: 14, no healthy upstream
[2024-03-07 03:17:31.257][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 310s ag
o: 14, no healthy upstream
[2024-03-07 03:17:59.267][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 338s ag
o: 14, no healthy upstream
[2024-03-07 03:18:28.889][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 368s ag
o: 14, no healthy upstream
[2024-03-07 03:18:41.524][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 380s ag
o: 14, no healthy upstream
[2024-03-07 03:18:42.768][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 382s ag
o: 14, no healthy upstream
[2024-03-07 03:19:05.548][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 404s ag
o: 14, no healthy upstream
[2024-03-07 03:19:26.006][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 425s ago: 14, no healthy upstream
[2024-03-07 03:19:34.582][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 433s ago: 14, no healthy upstream
[2024-03-07 03:20:02.878][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 462s ago: 14, no healthy upstream
[2024-03-07 03:20:17.984][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 477s ago: 14, no healthy upstream
[2024-03-07 03:20:23.831][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 483s ago: 14, no healthy upstream
[2024-03-07 03:20:27.603][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 486s ago: 14, no healthy upstream
[2024-03-07 03:20:31.735][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 491s ago: 14, no healthy upstream
[2024-03-07 03:20:48.332][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 507s ago: 14, no healthy upstream
[2024-03-07 03:20:57.674][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 516s ago: 14, no healthy upstream
[2024-03-07 03:21:12.390][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 531s ago: 14, no healthy upstream
[2024-03-07 03:21:42.229][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 561s ago: 14, no healthy upstream
[2024-03-07 03:21:58.696][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 577s ago: 14, no healthy upstream
[2024-03-07 03:22:24.603][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 603s ago: 14, no healthy upstream
[2024-03-07 03:22:48.130][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 627s ago: 14, no healthy upstream
[2024-03-07 03:23:17.094][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 656s ago: 14, no healthy upstream
[2024-03-07 03:23:39.588][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 678s ago: 14, no healthy upstream
[2024-03-07 03:23:52.943][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 692s ago: 14, no healthy upstream
[2024-03-07 03:23:56.546][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 695s ago: 14, no healthy upstream
[2024-03-07 03:24:03.108][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 702s ago: 14, no healthy upstream
[2024-03-07 03:24:12.983][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 712s ago: 14, no healthy upstream
[2024-03-07 03:24:29.579][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 728s ago: 14, no healthy upstream
[2024-03-07 03:24:34.658][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 733s ago: 14, no healthy upstream
[2024-03-07 03:24:37.604][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 736s ago: 14, no healthy upstream
[2024-03-07 03:24:54.716][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 754s ago: 14, no healthy upstream
[2024-03-07 03:25:18.185][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 777s ago: 14, no healthy upstream
[2024-03-07 03:25:48.098][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 807s ago: 14, no healthy upstream
[2024-03-07 03:25:50.520][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 809s ago: 14, no healthy upstream
[2024-03-07 03:26:01.690][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 820s ago: 14, no healthy upstream
[2024-03-07 03:26:30.518][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 849s ago: 14, no healthy upstream
[2024-03-07 03:26:52.494][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 871s ago: 14, no healthy upstream
[2024-03-07 03:27:00.670][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 879s ago: 14, no healthy upstream
[2024-03-07 03:27:26.575][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 905s ago: 14, no healthy upstream
[2024-03-07 03:27:33.402][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 912s ago: 14, no healthy upstream
[2024-03-07 03:27:44.040][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 923s ago: 14, no healthy upstream
[2024-03-07 03:27:59.109][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 938s ago: 14, no healthy upstream
[2024-03-07 03:28:02.022][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 941s ago: 14, no healthy upstream
imroc commented 7 months ago

cc @Xunzhuo

Xunzhuo commented 7 months ago

You should set dnsPolicy: ClusterFirstWithHostNet too, the xds_clusters of envoyproxy used service name for looking up control plane:

          "cluster": {
            "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
            "connectTimeout": "10s",
            "loadAssignment": {
              "clusterName": "xds_cluster",
              "endpoints": [
                {
                  "lbEndpoints": [
                    {
                      "endpoint": {
                        "address": {
                          "socketAddress": {
                            "address": "envoy-gateway",
                            "portValue": 18000
                          }
                        }
                      },
                      "loadBalancingWeight": 1
                    }
                  ],
                  "loadBalancingWeight": 1
                }
              ]

When you are using hostNetwork, we need to set dnsPolicy to ClusterFirstWithHostNet too.

Refer to: https://github.com/envoyproxy/gateway/pull/2374#issue-2059138573

Xunzhuo commented 7 months ago

Since we have supported deployment/svc mergepatch, I think we need to remove hostnetwork support on Envoyproxy API or have a more complete implementation for it.

Any thoughts on it @arkodg ?

arkodg commented 7 months ago

Yes let's remove it

Xunzhuo commented 7 months ago

Let me handle it before GA.

imroc commented 7 months ago

Thanks, change to mergepatch solves this problem:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg2
  namespace: envoy-gateway-system
spec:
  shutdown:
    drainTimeout: "1s"
    minDrainDuration: "1s"
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  hostNetwork: true
                  dnsPolicy: ClusterFirstWithHostNet