search

kumahq / kuma

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.
https://kuma.io/install
Apache License 2.0
3.61k stars 331 forks source link

Wait For Dataplane Ready takes too long and times out at 3 minutes #7849

Closed cbugneac-nex closed 5 months ago

cbugneac-nex commented 12 months ago

What happened?

I have enabled the waitForDataplaneReady functionality and now starting a pod takes ~ 3 minutes due to postStart lifecycle hook timing out. It does restart the pod eventually but waiting 3 minutes is not acceptable. Are there any other changes needed to be done apart for setting in Helm chart ?

The config is pretty generic, just couple for externalServices. This is the config I have added in Helm chart values file:

  envVars:
    KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_WAIT_FOR_DATAPLANE_READY: true

Here are the pod events after enabling this feature:

Events:
  Type                    Reason               Age           From                           Message
  ----                    ------               ----          ----                           -------
  Normal                  Scheduled            3m46s         default-scheduler              Successfully assigned ***/****-5fb758f5b-vlwsx to ip-**-***-***-***.eu-west-1.compute.internal
  Warning                 FailedPostStartHook  45s           kubelet                        Exec lifecycle hook ([kuma-dp wait --url http://localhost:9901/ready]) for Container "kuma-sidecar" in Pod "****-5fb758f5b-vlwsx_nsl(5b505d6d-42cf-4138-aba7-8dba2a5bf87c)" failed - error: command 'kuma-dp wait --url http://localhost:9901/ready' exited with 1: 2023-09-25T12:02:01.897Z  INFO  kuma-dp.wait  waiting for data plane proxy to be ready  {"timeout": "3m0s"}
2023-09-25T12:02:02.898Z  INFO                 kuma-dp.wait  data plane proxy is not ready  {"err": "Get \"http://localhost:9901/ready\": dial tcp [::1]:9901: connect: connection refused"}
...
2023-09-25T12:05:01.898Z  INFO                 kuma-dp.wait  data plane proxy is not ready  {"err": "Get \"http://localhost:9901/ready\": dial tcp [::1]:9901: connect: connection refused"}
Error: timeout occurred while waiting for data plane proxy to be ready
, message: "2023-09-25T12:02:01.897Z\tINFO\tkuma-dp.wait\twaiting for data plane proxy to be ready\t{\"timeout\": \"3m0s\"}\n2023-09-25T12:02:02.898Z\tINFO\tkuma-dp.wait\tdata plane proxy is not ready\t{\"err\": \"Get \\\"http://localhost:9901/ready\\\": dial tcp [::1]:9901: connect: connection refused\"}\n2023-09-25T12:02:03.899Z\tINFO\tkuma-dp.wait\tdata plane proxy is not ready\t{\"err\": \"Get \\\"http://localhost:9901/ready\\\": dial tcp [::1]:9901: connect: connection refused\"}\n2023-09-25T12:02:04.898Z\tINFO\tkuma-dp.wait\tdata plane proxy is not ready
...
...
\\\"http://localhost:9901/ready\\\": dial tcp [::1]:9901: connect: connection refused\"}\nError: timeout occurred while waiting for data plane proxy to be ready\n"
  Normal   Killing               45s                  kubelet                          FailedPostStartHook
  Normal   Pulled                44s                  kubelet                          Container image "***************.dkr.ecr.eu-west-1.amazonaws.com/****************:0.1.0" already present on machine
  Normal   Started               44s                  kubelet                          Started container ****************
  Normal   Created               44s                  kubelet                          Created container ****************
  Normal   Started               43s (x2 over 3m45s)  kubelet                          Started container kuma-sidecar
  Normal   Created               43s (x2 over 3m45s)  kubelet                          Created container kuma-sidecar
  Normal   Pulled                43s (x2 over 3m45s)  kubelet                          Container image "***************.dkr.ecr.eu-west-1.amazonaws.com/*****/kuma-dp:2.4.1" already present on machine
  Warning  Unhealthy             43s                  kubelet                          Readiness probe failed: Get "http://10.255.61.118:9000/8080/actuator/health/readiness": dial tcp 10.255.61.118:9000: connect: connection refused
  Normal   CreatedKumaDataplane  43s                  k8s.kuma.io/dataplane-generator  Created Kuma Dataplane: ****************-5fb758f5b-vlwsx
  Warning  Unhealthy             35s (x4 over 39s)    kubelet                          Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  Unhealthy             35s                  kubelet                          Liveness probe failed: HTTP probe failed with statuscode: 503
  Normal   UpdatedKumaDataplane  25s (x3 over 39s)    k8s.kuma.io/dataplane-generator  Updated Kuma Dataplane: ****************-5fb758f5b-vlwsx

Control plane Helm values file configuration:

global:
  image:
    # Use our own image from ECR
    registry: ***************.dkr.ecr.eu-west-1.amazonaws.com/*****

controlPlane:
  extraLabels:
    tags.datadoghq.com/env: dev
    tags.datadoghq.com/service: kuma-cp
    tags.datadoghq.com/version: 2.4.1-helm

  replicas: 2

  podDisruptionBudget:
    enabled: true

  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 1
      memory: 1Gi

  nodeSelector: &nodeSelector
    nodegroup: core
    kubernetes.io/os: linux

  tolerations: &tolerations
    - key: system-no-schedule
      operator: Equal
      value: "true"
      effect: NoSchedule

  podSecurityContext: &podSecurityContext
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault

  containerSecurityContext: &containerSecurityContext
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

cni:
  enabled: true
  chained: true

  # EKS-specific configuration
  netDir: /etc/cni/net.d
  binDir: /opt/cni/bin
  confName: 10-aws.conflist

kubectl:
  image:
    # Use our own image from ECR
    registry: ***************.dkr.ecr.eu-west-1.amazonaws.com/******
    repository: kumahq-kubectl

hooks:
  nodeSelector: *nodeSelector
  tolerations: *tolerations
  podSecurityContext: *podSecurityContext
  containerSecurityContext: *containerSecurityContext

It looks strange it does takes so long and eventually times out. Please could someone shed some light why this happens ?

Kuma Mesh version: v2.4.1 AWS EKS cluster version: 1.25.12 Networking: VPC CNI Add-on

jakubdyszkiewicz commented 12 months ago

Triage: Hey, can you share kuma-dp logs? Does Envoy ever receive the configuration from the CP? If not, it's never ready and it will time out

cbugneac-nex commented 12 months ago

The control plane is up and running:

$ kubectl -n kuma-system get pods
NAME                                  READY   STATUS    RESTARTS   AGE
kuma-control-plane-6899546ff5-7k95r   1/1     Running   0          101m
kuma-control-plane-6899546ff5-wj9mj   1/1     Running   0          101m

$ kubectl -n kuma-system get service
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                AGE
kuma-control-plane   ClusterIP   172.20.171.160   <none>        5680/TCP,5681/TCP,5682/TCP,443/TCP,5676/TCP,5678/TCP   74d

Well, I cannot get the logs as container is not started:

NAME                                     READY   STATUS              RESTARTS   AGE
service-****-698b7985c8-g89f4            0/2     ContainerCreating   0          3m24s

$ kubectl logs service-****-698b7985c8-g89f4
Error from server (BadRequest): container "service-***" in pod "service-****-698b7985c8-g89f4" is waiting to start: ContainerCreating

Waiting a bit and then I get:

NAME                                     READY   STATUS               RESTARTS     AGE
service-****-698b7985c8-df7s4            0/2     PostStartHookError   0 (4s ago)   3m6s

and after a while:

NAME                                     READY   STATUS    RESTARTS      AGE
service-***-698b7985c8-df7s4            2/2     Running   1 (29s ago)   3m31s
cbugneac-nex commented 12 months ago

Here is the logs of kuma-dp after it has been eventually started:

$ kubectl logs service-****-698b7985c8-df7s4 -c kuma-sidecar
2023-09-25T15:12:03.246Z    INFO    Skipping reading config from file
2023-09-25T15:12:03.246Z    INFO    kuma-dp.run effective configuration {"config": "{\"controlPlane\":{\"caCert\":\"-----BEGIN CERTIFICATE-----\\*****\\nNFowEjEQMA4GA1UEAxMHa3VtYS1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC\\nAQoCggEBAMARPqVj2PBkkL2LlH/mxn8SNSoJI9dbpTzTKyq/BOwyr48ehxYOLlnw\\ngXHias4U070k0JncecSlThCKWV4K72cZ5sX2CpDn4BIYPpH3FcntgDeQxeI2rhCl\\nsCD7aKFuTv+y01GaVG14tLYiYkOts/fRqtkFvQ5k7sKhpWgzge/AHxv7qxPyEX1x\\nMAjszjKmay8q0Gn0Pi3xLBbdXp3B1hMUynQuXMK0MyqXmKcGXRGxmUA9XhdJjrx4\\nV7Y4IPx2Ph4IujV9PUKDTpXVvlCmwPB3z2aZ2BumEEZ4zP5oPbVE9qt8JZ5XO5rm\\nmawIG4/RR9pd/4GqQxAsruPCucSp0l0CAwEAAaNhMF8wDgYDVR0PAQH/BAQDAgKk\\nMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/\\nMB0GA1UdDgQWBBQTdy8VejNhKbJkNNtb67IiC1WJWzANBgkqhkiG9w0BAQsFAAOC\\nAQEAeAON1DBTNDD4A3xNJ/ASm+yZOloRodp8KFtFLQ/CNv4zjp7+XTcJgVAK8v0W\\n0riCG50kkgQ4rfCVncxHwsrsJJ/i0V3SvAmCHx/k7azSrXWaaHTxoXgLWJVe7omu\\nbBhj/TUqp0o1+3Uw1hja7kIzNWxTvV9aBKJ5gLIPpGQF16jQh151402YAa2dmk0V\\ndSkdx/GFT7NixQGkKH4lP20kZZqtUSVVKZ2kBsW4SgKvG7IpEDCsTWzt+09B7dNR\\n0P61VXAFmsV+XsB33PdlhmIOnPZlXz9zERDTq1OIBU1WCuT25fH8FUubnre8PFip\\na5xyXte1xkQyif5+ioGuJWNwDA==\\n-----END CERTIFICATE-----\\n\",\"caCertFile\":\"\",\"retry\":{\"backoff\":\"3s\",\"maxDuration\":\"5m0s\"},\"url\":\"https://kuma-control-plane.kuma-system:5678\"},\"dataplane\":{\"drainTime\":\"30s\",\"mesh\":\"default\",\"name\":\"service-****-698b7985c8-df7s4.nsl\",\"proxyType\":\"dataplane\"},\"dataplaneRuntime\":{\"binaryPath\":\"envoy\",\"concurrency\":2,\"dataplaneTokenPath\":\"/var/run/secrets/kubernetes.io/serviceaccount/token\",\"metrics\":{},\"resources\":{}},\"dns\":{\"coreDnsBinaryPath\":\"coredns\",\"coreDnsEmptyPort\":15054,\"coreDnsPort\":15053,\"enabled\":true,\"envoyDnsPort\":15055,\"prometheusPort\":19153}}"}
2023-09-25T15:12:03.246Z    INFO    kuma-dp.run generated configurations will be stored in a temporary directory    {"dir": "/tmp/kuma-dp-2612837746"}
2023-09-25T15:12:03.335Z    INFO    kuma-dp.run fetched Envoy version   {"version": {"Build":"7bba38b743bb3bca22dffb4a21c38ccc155fbef8/1.27.0/Modified/RELEASE/BoringSSL","Version":"1.27.0","KumaDpCompatible":true}}
2023-09-25T15:12:03.335Z    INFO    kuma-dp.run generating bootstrap configuration
2023-09-25T15:12:03.335Z    INFO    dataplane   trying to fetch bootstrap configuration from the Control Plane
2023-09-25T15:12:03.352Z    INFO    kuma-dp.run received bootstrap configuration    {"adminPort": 9901}
2023-09-25T15:12:03.354Z    INFO    kuma-dp.run starting Kuma DP    {"version": "2.4.1"}
2023-09-25T15:12:03.354Z    INFO    kuma-dp.run.access-log-streamer starting resilient component ...
2023-09-25T15:12:03.354Z    INFO    access-log-streamer cleaning existing access log pipe   {"file": "/tmp/kuma-dp-2612837746/kuma-al-service-****-698b7985c8-df7s4.nsl-default.sock"}
2023-09-25T15:12:03.354Z    INFO    access-log-streamer creating access log pipe    {"file": "/tmp/kuma-dp-2612837746/kuma-al-service-****-698b7985c8-df7s4.nsl-default.sock"}
2023-09-25T15:12:03.354Z    INFO    kuma-dp.run.envoy   bootstrap configuration saved to a file {"file": "/tmp/kuma-dp-2612837746/bootstrap.yaml"}
2023-09-25T15:12:03.354Z    INFO    kuma-dp.run.dns-server  configuration saved to a file   {"file": "/tmp/kuma-dp-2612837746/Corefile"}
2023-09-25T15:12:03.354Z    INFO    kuma-dp.run.dns-server  starting DNS Server (coredns)   {"args": ["-conf", "/tmp/kuma-dp-2612837746/Corefile", "-quiet"]}
2023-09-25T15:12:03.354Z    INFO    kuma-dp.run.envoy   starting Envoy  {"path": "/usr/bin/envoy", "arguments": ["--config-path", "/tmp/kuma-dp-2612837746/bootstrap.yaml", "--drain-time-s", "30", "--disable-hot-restart", "--log-level", "info", "--concurrency", "2"]}
2023-09-25T15:12:03.355Z    INFO    metrics-hijacker    starting Metrics Hijacker Server    {"socketPath": "unix:///tmp/kuma-dp-2612837746/kuma-mh-service-****-698b7985c8-df7s4.nsl-default.sock"}
[2023-09-25 15:12:03.410][35][info][main] [source/server/server.cc:413] initializing epoch 0 (base id=0, hot restart version=disabled)
[2023-09-25 15:12:03.410][35][info][main] [source/server/server.cc:415] statically linked extensions:
[2023-09-25 15:12:03.410][35][info][main] [source/server/server.cc:417]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2023-09-25 15:12:03.410][35][info][main] [source/server/server.cc:417]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   network.connection.client: default, envoy_internal
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.common.key_value: envoy.key_value.file_based
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.protocols: dubbo
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.connection_handler: envoy.connection_handler.default
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2023-09-25 15:12:03.411][35][info][main] [source/server/server.cc:417]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.request_id: envoy.request_id.uuid
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.regex_engines: envoy.regex_engines.google_re2
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.resolvers: envoy.ip
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   quic.http_server_connection: quic.http_server_connection.default
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2023-09-25 15:12:03.412][35][info][main] [source/server/server.cc:417]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.header_mutation, envoy.filters.http.upstream_codec
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2023-09-25 15:12:03.413][35][info][main] [source/server/server.cc:417]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2023-09-25 15:12:03.421][35][info][main] [source/server/server.cc:470] HTTP header map info:
[2023-09-25 15:12:03.423][35][info][main] [source/server/server.cc:473]   request header map: 680 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2023-09-25 15:12:03.424][35][info][main] [source/server/server.cc:473]   request trailer map: 128 bytes:
[2023-09-25 15:12:03.424][35][info][main] [source/server/server.cc:473]   response header map: 440 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2023-09-25 15:12:03.424][35][info][main] [source/server/server.cc:473]   response trailer map: 152 bytes: grpc-message,grpc-status
[2023-09-25 15:12:03.433][35][info][main] [source/server/server.cc:840] runtime: layers:
  - name: kuma
    static_layer:
      re2.max_program_size.warn_level: 1000
      envoy.restart_features.use_apple_api_for_dns_lookups: false
      re2.max_program_size.error_level: 4294967295
[2023-09-25 15:12:03.434][35][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901
[2023-09-25 15:12:03.435][35][info][config] [source/server/configuration_impl.cc:144] loading tracing configuration
[2023-09-25 15:12:03.435][35][info][config] [source/server/configuration_impl.cc:103] loading 1 static secret(s)
[2023-09-25 15:12:03.435][35][info][config] [source/server/configuration_impl.cc:109] loading 2 cluster(s)
[2023-09-25 15:12:03.449][35][info][config] [source/server/configuration_impl.cc:113] loading 0 listener(s)
[2023-09-25 15:12:03.449][35][info][config] [source/server/configuration_impl.cc:130] loading stats configuration
[2023-09-25 15:12:03.450][35][info][main] [source/server/server.cc:937] starting main dispatch loop
[2023-09-25 15:12:03.450][35][info][runtime] [source/common/runtime/runtime_impl.cc:577] RTDS has finished initialization
[2023-09-25 15:12:03.450][35][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds
[2023-09-25 15:12:03.451][35][warning][main] [source/server/server.cc:814] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2023-09-25 15:12:05.120][35][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 20 cluster(s), remove 2 cluster(s)
[2023-09-25 15:12:05.348][35][info][upstream] [source/common/upstream/cds_api_helper.cc:69] cds: added/updated 20 cluster(s), skipped 0 unmodified cluster(s)
[2023-09-25 15:12:05.350][35][info][upstream] [source/common/upstream/cluster_manager_impl.cc:200] cm init: initializing secondary clusters
[2023-09-25 15:12:05.358][35][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2023-09-25 15:12:05.358][35][info][main] [source/server/server.cc:918] all clusters initialized. initializing init manager
[2023-09-25 15:12:05.376][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'kuma:envoy:admin'
[2023-09-25 15:12:05.377][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.12:443'
[2023-09-25 15:12:05.378][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.23:80'
[2023-09-25 15:12:05.378][35][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2023-09-25 15:12:05.378][35][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2023-09-25 15:12:05.378][35][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2023-09-25 15:12:05.379][35][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2023-09-25 15:12:05.384][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'probe:listener'
[2023-09-25 15:12:05.386][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.17:80'
[2023-09-25 15:12:05.387][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.23:443'
[2023-09-25 15:12:05.388][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.8:80'
[2023-09-25 15:12:05.389][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.22:80'
[2023-09-25 15:12:05.390][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'plugins:bootstrap:k8s:hooks:apiServerBypass'
[2023-09-25 15:12:05.391][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.13:443'
[2023-09-25 15:12:05.392][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.14:80'
[2023-09-25 15:12:05.393][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.19:80'
[2023-09-25 15:12:05.395][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.1:80'
[2023-09-25 15:12:05.396][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.12:80'
[2023-09-25 15:12:05.398][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.8:443'
[2023-09-25 15:12:05.404][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'inbound:**.***.61.119:8080'
[2023-09-25 15:12:05.406][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'inbound:passthrough:ipv4'
[2023-09-25 15:12:05.408][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.18:443'
[2023-09-25 15:12:05.409][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.6:80'
[2023-09-25 15:12:05.411][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'inbound:passthrough:ipv6'
[2023-09-25 15:12:05.413][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:172.20.203.73:8080'
[2023-09-25 15:12:05.415][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.15:80'
[2023-09-25 15:12:05.417][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.3:80'
[2023-09-25 15:12:05.446][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'kuma:metrics:prometheus'
[2023-09-25 15:12:05.448][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:172.20.240.240:8080'
[2023-09-25 15:12:05.450][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.17:443'
[2023-09-25 15:12:05.452][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:passthrough:ipv6'
[2023-09-25 15:12:05.454][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.11:80'
[2023-09-25 15:12:05.456][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.18:80'
[2023-09-25 15:12:05.457][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.22:443'
[2023-09-25 15:12:05.458][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.7:443'
[2023-09-25 15:12:05.459][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.21:80'
[2023-09-25 15:12:05.460][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.27:80'
[2023-09-25 15:12:05.461][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.7:80'
[2023-09-25 15:12:05.462][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.24:443'
[2023-09-25 15:12:05.463][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.24:80'
[2023-09-25 15:12:05.465][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.25:443'
[2023-09-25 15:12:05.466][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.27:443'
[2023-09-25 15:12:05.467][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.2:80'
[2023-09-25 15:12:05.468][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:172.20.120.174:8080'
[2023-09-25 15:12:05.470][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.0:80'
[2023-09-25 15:12:05.471][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.11:443'
[2023-09-25 15:12:05.472][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.20:443'
[2023-09-25 15:12:05.473][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.16:443'
[2023-09-25 15:12:05.474][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.16:80'
[2023-09-25 15:12:05.475][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.26:443'
[2023-09-25 15:12:05.476][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.6:443'
[2023-09-25 15:12:05.477][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.14:443'
[2023-09-25 15:12:05.478][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.19:443'
[2023-09-25 15:12:05.479][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.25:80'
[2023-09-25 15:12:05.482][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'kuma:dns'
[2023-09-25 15:12:05.483][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.21:443'
[2023-09-25 15:12:05.484][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.26:80'
[2023-09-25 15:12:05.485][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:passthrough:ipv4'
[2023-09-25 15:12:05.487][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:172.20.77.132:8080'
[2023-09-25 15:12:05.488][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.13:80'
[2023-09-25 15:12:05.489][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.15:443'
[2023-09-25 15:12:05.490][35][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:86] lds: add/update listener 'outbound:240.0.0.20:80'
[2023-09-25 15:12:05.490][35][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:858] all dependencies initialized. starting workers
cbugneac-nex commented 12 months ago

Hi @jakubdyszkiewicz here is the log of DP side-car for a test pod:

$ kubectl logs test-pod -c kuma-sidecar
2023-09-26T12:11:47.673Z    INFO    Skipping reading config from file
2023-09-26T12:11:47.673Z    INFO    kuma-dp.run effective configuration
{
  "config": {
    "controlPlane": {
      "caCert": "-----BEGIN CERTIFICATE-----*********\\n-----END CERTIFICATE-----\\n",
      "caCertFile": "",
      "retry": {
        "backoff": "3s",
        "maxDuration": "5m0s"
      },
      "url": "https://kuma-control-plane.kuma-system:5678"
    },
    "dataplane": {
      "drainTime": "30s",
      "mesh": "default",
      "name": "cbugneac-test.nsl",
      "proxyType": "dataplane"
    },
    "dataplaneRuntime": {
      "binaryPath": "envoy",
      "concurrency": 2,
      "dataplaneTokenPath": "/var/run/secrets/kubernetes.io/serviceaccount/token",
      "metrics": {},
      "resources": {}
    },
    "dns": {
      "coreDnsBinaryPath": "coredns",
      "coreDnsEmptyPort": 15054,
      "coreDnsPort": 15053,
      "enabled": true,
      "envoyDnsPort": 15055,
      "prometheusPort": 19153
    }
  }
}
2023-09-26T12:11:47.673Z    INFO    kuma-dp.run generated configurations will be stored in a temporary directory    {"dir": "/tmp/kuma-dp-3898465581"}
2023-09-26T12:11:47.766Z    INFO    kuma-dp.run fetched Envoy version   {"version": {"Build":"7bba38b743bb3bca22dffb4a21c38ccc155fbef8/1.27.0/Modified/RELEASE/BoringSSL","Version":"1.27.0","KumaDpCompatible":true}}
2023-09-26T12:11:47.766Z    INFO    kuma-dp.run generating bootstrap configuration
2023-09-26T12:11:47.767Z    INFO    dataplane   trying to fetch bootstrap configuration from the Control Plane
2023-09-26T12:11:47.773Z    INFO    dataplane   Dataplane entity is not yet found in the Control Plane. If you are running on Kubernetes, CP is most likely still in the process of converting Pod to Dataplane. If it takes too long, check kuma-cp logs. Retrying.    {"backoff": "3s"}
...
...
2023-09-26T12:14:44.883Z    INFO    dataplane   Dataplane entity is not yet found in the Control Plane. If you are running on Kubernetes, CP is most likely still in the process of converting Pod to Dataplane. If it takes too long, check kuma-cp logs. Retrying.    {"backoff": "3s"}
2023-09-26T12:14:47.833Z    INFO    Received signal, stopping instance gracefully   {"signal": "terminated"}
Error: Failed to generate Envoy bootstrap config. context canceled

From the logs seems that DP cannot fetch the config from CP. Is the Kuma mesh itself not allowing network connectivity to https://kuma-control-plane.kuma-system:5678 ?

lobkovilya commented 12 months ago

As we confirmed in Slack the issue here is using localhost in the probe's URL that could be resolved to ipv6 loopback address [::1] instead of 127.0.0.1.

cbugneac-nex commented 12 months ago

Thanks @lobkovilya for help. Can I suggest to add a proper logging message as at the moment from the above logs it's not clear and difficult to guess that the root cause is IPv6 loopback [::1] address ?

cbugneac-nex commented 12 months ago

Hi @lobkovilya some thoughts:

  1. Based on https://kubernetes.io/docs/concepts/services-networking/dual-stack/ since K8s version 1.21 IPv4/IPv6 dual-stack networking is enabled by default. The same for containerd runtime.
  2. Disabling IPv6 in EKS is risky and I have no idea what other services might break due to this change.
  3. Given IPv6 is the future - wouldn't make more sense for the kuma-sidecar (kuma-dp) to bind to both IPv6 and IPv4 stacks ? This will allow to resolve localhost to any loopback IP address.
  4. Optionally - provide a way to overwrite via configuration localhost to 127.0.0.1 or ::1.

Does the above make sense ?

cbugneac-nex commented 11 months ago

@lobkovilya Please can this issue be re-opened as even after update to 2.4.2 and setting in Helm values this configuration it still doesn't work:

  envVars:
    KUMA_BOOTSTRAP_SERVER_PARAMS_ADMIN_ADDRESS: "::1"
    KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_WAIT_FOR_DATAPLANE_READY: true
github-actions[bot] commented 11 months ago

Removing closed state labels due to the issue being reopened.

cbugneac-nex commented 11 months ago

Hi @lobkovilya ,

I've tried to disable IPv6 on pods as AWS VPC CNI Add-on in version 1.15.0 has added support for it via setting DISABLE_POD_V6 = "true". But it makes worser, the pod won't start at all as kuma-sidecar tries to make changes for IPv6 which is disabled:

$ kubectl describe pod ...
...
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               5m46s                  default-scheduler  Successfully assigned default/service-****-dbb8999bd-tv2mq to ip-**-***-***-**.eu-west-1.compute.internal
  Warning  FailedCreatePodSandBox  5m45s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c8fb6332ddd05b7bd76fb445eddae4778165d6a3ec0b42754b9dc941281a2cff": plugin type="kuma-cni" name="kuma-cni" failed (add): pod excluded - could not inject rules into namespace: cannot restore ipv6 iptable rules: failed to add IPv6 inbound address: permission denied
  Warning  FailedCreatePodSandBox  5m44s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a513afa6ec36d0fc201d436739b5f242f7bcdfbac46c5491770ff050f564488d": plugin type="kuma-cni" name="kuma-cni" failed (add): pod excluded - could not inject rules into namespace: cannot restore ipv6 iptable rules: failed to add IPv6 inbound address: permission denied

Inside the kuma-sidecar Pod I see that IPv6 loopback exists:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::6/128 scope global
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 9001 qdisc noqueue state UP
    link/ether 86:86:69:30:0a:fa brd ff:ff:ff:ff:ff:ff
    inet ***.***.***.***/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8486:69ff:fe30:afa/64 scope link
       valid_lft forever preferred_lft forever

Looks like kuma-sidecar doesn't bind to ::1:

$ kubectl exec -it service-****-698b7985c8-8bxq9 -c kuma-sidecar -- /busybox/busybox sh

/home/nonroot $ wget -S -O- [::1]:9901/ready
Connecting to [::1]:9901 ([::1]:9901)
wget: can't connect to remote host: Connection refused

/home/nonroot $ wget -S -O- 127.0.0.1:9901/ready
Connecting to 127.0.0.1:9901 (127.0.0.1:9901)
  HTTP/1.1 200 OK
  content-type: text/plain; charset=UTF-8
  cache-control: no-cache, max-age=0
  x-content-type-options: nosniff
  date: Wed, 27 Sep 2023 10:49:36 GMT
  server: envoy
  connection: close
  transfer-encoding: chunked

writing to stdout
LIVE
-                    100% |***********************************************************************************************************************************************************************************************************|     5  0:00:00 ETA
written to stdout
/home/nonroot $
github-actions[bot] commented 8 months ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

github-actions[bot] commented 5 months ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

lukidzi commented 5 months ago

@cbugneac-nex I believe this was causing it https://github.com/kumahq/kuma/issues/9431 (since you are using CNI). It has been fixed lately and should be released in 2.7. Feel free to reopen if still happens in 2.7