Upstream connection seems unstable in Mesh network

[ventaubain]

Overview of the Issue

I have 2 services in a Mesh network. Services are in a Nomad cluster. The service (1) is connected to the service (2) by an upstream on the localhost:7000. (2) is a socks5 proxy and has 3 replicas (default load balancing policy and this one has a correct comportment) and the request from (1) is socks5-based. At the beginning, both work correctly together but, after few times (not constant and no reason detected yet), the upstream connection seems to fail.

After few times, if I do a Curl (curl --socks5-hostname localhost:7000 <URL>), I obtain curl: (97) Unable to receive initial SOCKS5 response.. If I restart the sidecar of service (1), the connection is retablished and the upstream correctly works. If there is only one container and not 3 under the upstream, the connection seems not stable too. The problem could be correlated to the load balancing.

The service (1) has an another upstream to a Mysql container and has no (detected) problem.

When the active connections (envoy_cluster_upstream_cx_active) is higher than 1k02, the proxy stops and never re-works except if I relaunch it. I use default sidecar configuration so I have:

    Limits = {
      MaxConnections = 512
      MaxPendingRequests = 512
      MaxConcurrentRequests = 512
    }

It can be the reason but why crash with 1k02 and not 512 ?

Reproduction Steps

Steps to reproduce this issue, eg:

• Get a socks5 proxy container (2)
• Create a Nomad job where:
  • 3 instances of the container (2) are initialized
  • 1 container (1) which sends a request through the proxy
  • (1) is connected to (2) by an upstream through the Mesh network

Consul info for both Client and Server

• Nomad job configuration

  group "back"{
      volume "storage" {
          type = "host"
          source = "storage"
          read_only = false
      }
  
      network {
          mode = "bridge"
          port "envoy_metrics" {
              to = 8888
          }
      }
  
      service {
          meta {
              envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}"
          }
          name = "backend"
          port = 8080
  
          tags = [
              "traefik.enable=true",
              "traefik.http.routers.backend.rule=Host(`backend.app`)"
          ]
  
          connect {
              sidecar_service {
                  proxy{
                      upstreams {
                          destination_name = "proxy"
                          local_bind_port  = 7000
                      }
                      upstreams {
                          destination_name = "mysql"
                          local_bind_port = 7001
                      }
                      config {
                          envoy_prometheus_bind_addr = "0.0.0.0:8888"
                      }
                  }
              }
          }
      }
  
      task "backend" {
          driver = "docker"
          volume_mount {
              volume = "storage"
              destination = "/data"
              read_only = false
          }
          resources {
              cpu = 3
              memory = 300
          }
          config {
              image = "172.16.22.10:443/backend:1.2.0"
              force_pull = true
          }
      }
  }
  
  group "proxy"{
      count = 3
      network {
          mode = "bridge"
          port "envoy_metrics" {
              to = 8888
          }
      }
  
      service {
          meta {
              envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}"
          }
          name = "proxy"
          port = 8051
  
          connect {
              sidecar_service {
                  proxy{
                      config {
                          envoy_prometheus_bind_addr = "0.0.0.0:8888"
                      }
                  }
              }
              }
      }
  
      task "proxy" {
          driver = "docker"
          resources {
              cpu = 3
              memory = 300
          }
          config {
              image = "172.16.22.10:443/proxy:1.0.0"
              force_pull = true
          }
      }
  }

Operating system and Environment details

• Linux
• Nomad v1.4.2 (039d70eeef5888164cad05e7ddfd8b6f8220923b)
• Consul v1.13.3 Revision b29e5894

Log Fragments

I don't find explicit error in log.

• Log from the sidecar-proxy (first init + after restart when bug)

  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:390] initializing epoch 0 (base id=0, hot restart version=disabled)
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:392] statically linked extensions:
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   network.connection.client: default, envoy_internal
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.http.cache: envoy.extensions.http.cache.simple
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.regex_engines: envoy.regex_engines.google_re2
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.health_checkers: envoy.health_checkers.redis
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.http.stateful_session: envoy.http.stateful_session.cookie
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.request_id: envoy.request_id.uuid
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.serializers: dubbo.hessian2
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.matching.action: composite-action, skip
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.transports: auto, framed, header, unframed
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.resolvers: envoy.ip
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.common.key_value: envoy.key_value.file_based
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.protocols: dubbo
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
  [2022-11-14 09:31:26.811][1][info][main] [source/server/server.cc:394]   envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san
  [2022-11-14 09:31:26.815][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:31:26.815][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:31:26.815][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:31:26.816][1][info][main] [source/server/server.cc:442] HTTP header map info:
  [2022-11-14 09:31:26.819][1][info][main] [source/server/server.cc:445]   request header map: 656 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-proto,x-ot-span-context,x-request-id
  [2022-11-14 09:31:26.819][1][info][main] [source/server/server.cc:445]   request trailer map: 128 bytes: 
  [2022-11-14 09:31:26.819][1][info][main] [source/server/server.cc:445]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
  [2022-11-14 09:31:26.819][1][info][main] [source/server/server.cc:445]   response trailer map: 152 bytes: grpc-message,grpc-status
  [2022-11-14 09:31:26.829][1][info][main] [source/server/server.cc:811] runtime: layers:
  - name: base
  static_layer:
    re2.max_program_size.error_level: 1048576
  [2022-11-14 09:31:26.830][1][info][admin] [source/server/admin/admin.cc:67] admin address: 127.0.0.2:19001
  [2022-11-14 09:31:26.831][1][info][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:531] c-ares library initialized.
  [2022-11-14 09:31:26.831][1][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
  [2022-11-14 09:31:26.831][1][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
  [2022-11-14 09:31:26.831][1][info][config] [source/server/configuration_impl.cc:97] loading 2 cluster(s)
  [2022-11-14 09:31:26.904][1][info][config] [source/server/configuration_impl.cc:101] loading 1 listener(s)
  [2022-11-14 09:31:26.910][1][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
  [2022-11-14 09:31:26.910][1][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
  [2022-11-14 09:31:26.910][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:221] cm init: initializing cds
  [2022-11-14 09:31:26.910][1][warning][main] [source/server/server.cc:786] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
  [2022-11-14 09:31:26.911][1][info][main] [source/server/server.cc:907] starting main dispatch loop
  [2022-11-14 09:31:26.917][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 3 cluster(s), remove 0 cluster(s)
  [2022-11-14 09:31:26.950][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:31:27.058][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:31:27.159][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 3 cluster(s), skipped 0 unmodified cluster(s)
  [2022-11-14 09:31:27.159][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:195] cm init: initializing secondary clusters
  [2022-11-14 09:31:27.163][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:225] cm init: all clusters initialized
  [2022-11-14 09:31:27.163][1][info][main] [source/server/server.cc:888] all clusters initialized. initializing init manager
  [2022-11-14 09:31:27.167][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'proxy:127.0.0.1:7000'
  [2022-11-14 09:31:27.170][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'public_listener:0.0.0.0:31961'
  [2022-11-14 09:31:27.171][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'mysql:127.0.0.1:7001'
  [2022-11-14 09:31:27.171][1][info][config] [source/server/listener_manager_impl.cc:841] all dependencies initialized. starting workers
  [2022-11-14 09:45:41.191][1][warning][main] [source/server/server.cc:846] caught ENVOY_SIGTERM
  [2022-11-14 09:45:41.191][1][info][main] [source/server/server.cc:977] shutting down server instance
  [2022-11-14 09:45:41.191][1][info][main] [source/server/server.cc:912] main dispatch loop exited
  [2022-11-14 09:45:42.243][1][warning][config] [./source/common/config/grpc_stream.h:160] DeltaAggregatedResources gRPC config stream closed: 13, 
  [2022-11-14 09:45:42.244][1][info][main] [source/server/server.cc:964] exiting
  [2022-11-14 09:45:42.270][1][info][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:540] c-ares library cleaned up.
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:390] initializing epoch 0 (base id=0, hot restart version=disabled)
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:392] statically linked extensions:
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.resolvers: envoy.ip
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.transports: auto, framed, header, unframed
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.regex_engines: envoy.regex_engines.google_re2
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.matching.action: composite-action, skip
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.protocols: dubbo
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.health_checkers: envoy.health_checkers.redis
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.common.key_value: envoy.key_value.file_based
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.request_id: envoy.request_id.uuid
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.serializers: dubbo.hessian2
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.http.stateful_session: envoy.http.stateful_session.cookie
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   network.connection.client: default, envoy_internal
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.http.cache: envoy.extensions.http.cache.simple
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
  [2022-11-14 09:45:44.736][1][info][main] [source/server/server.cc:394]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
  [2022-11-14 09:45:44.741][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:45:44.742][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:45:44.742][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:45:44.742][1][info][main] [source/server/server.cc:442] HTTP header map info:
  [2022-11-14 09:45:44.747][1][info][main] [source/server/server.cc:445]   request header map: 656 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-proto,x-ot-span-context,x-request-id
  [2022-11-14 09:45:44.747][1][info][main] [source/server/server.cc:445]   request trailer map: 128 bytes: 
  [2022-11-14 09:45:44.747][1][info][main] [source/server/server.cc:445]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
  [2022-11-14 09:45:44.747][1][info][main] [source/server/server.cc:445]   response trailer map: 152 bytes: grpc-message,grpc-status
  [2022-11-14 09:45:44.758][1][info][main] [source/server/server.cc:811] runtime: layers:
  - name: base
  static_layer:
    re2.max_program_size.error_level: 1048576
  [2022-11-14 09:45:44.760][1][info][admin] [source/server/admin/admin.cc:67] admin address: 127.0.0.2:19001
  [2022-11-14 09:45:44.760][1][info][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:531] c-ares library initialized.
  [2022-11-14 09:45:44.760][1][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
  [2022-11-14 09:45:44.760][1][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
  [2022-11-14 09:45:44.760][1][info][config] [source/server/configuration_impl.cc:97] loading 2 cluster(s)
  [2022-11-14 09:45:44.841][1][info][config] [source/server/configuration_impl.cc:101] loading 1 listener(s)
  [2022-11-14 09:45:44.847][1][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
  [2022-11-14 09:45:44.847][1][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
  [2022-11-14 09:45:44.847][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:221] cm init: initializing cds
  [2022-11-14 09:45:44.848][1][warning][main] [source/server/server.cc:786] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
  [2022-11-14 09:45:44.848][1][info][main] [source/server/server.cc:907] starting main dispatch loop
  [2022-11-14 09:45:44.858][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 3 cluster(s), remove 0 cluster(s)
  [2022-11-14 09:45:44.894][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:45:45.016][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
  [2022-11-14 09:45:45.122][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 3 cluster(s), skipped 0 unmodified cluster(s)
  [2022-11-14 09:45:45.122][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:195] cm init: initializing secondary clusters
  [2022-11-14 09:45:45.126][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:225] cm init: all clusters initialized
  [2022-11-14 09:45:45.126][1][info][main] [source/server/server.cc:888] all clusters initialized. initializing init manager
  [2022-11-14 09:45:45.130][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'proxy:127.0.0.1:7000'
  [2022-11-14 09:45:45.133][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'public_listener:0.0.0.0:31961'
  [2022-11-14 09:45:45.134][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'mysql:127.0.0.1:7001'
  [2022-11-14 09:45:45.134][1][info][config] [source/server/listener_manager_impl.cc:841] all dependencies initialized. starting workers

• Log from consul-node where the service is hosted (when restart)

  [ERROR] agent.envoy: Error receiving new DeltaDiscoveryRequest; closing request channel: error="rpc error: code = Canceled desc = context canceled"

• Log from nomad master (when restart)

  [ERROR] http: request failed: method=GET path="/v1/client/allocation/00054e32-ca8d-8d0a-9381-10dc488e1c12/exec?task=backend&tty=true&ws_handshake=true&command=%5B%22%2Fbin%2Fsh%22%5D" error="websocket: close 1001 (going away)" code=500

[ventaubain]

There are some "destroy" connections from the backend. It seems to be the problem and not "a bug". How works the sidecar about this "destroy" comportment ?

[ventaubain]

The problem seems to be an intern problem of the service. Not sidecar or Consul. I close the issue. Sorry for the mistake.