Open gschier opened 2 years ago
mattklein123
commented
2 years ago Can you list what extensions you are using? Are you using tracing? There have been other reports of memory leaks using tracing and it would be nice to understand if that might be involved here.
Also, can you get a memory profile? We are still working on getting memory profile dumps working using the new tcmalloc so you may have to recompile using gperftools until we get that working. cc @jmarantz @rojkov @wbpcode
wbpcode
commented
2 years ago As matt said, can you provide some more configs about your scene.
gschier
commented
2 years ago Thanks for the quick response!
In a previous issue around performance, you can find more info on our setup as well as a snippet of our config_dump https://github.com/envoyproxy/envoy/issues/19946. We're not doing anything fancy.
We can start working on getting a memory profile next week.
lambdai
commented
2 years ago re: Heap size doesn't seem to be correlated to num active connections, num draining listeners, or anything else I could find in the stats
How about number of stats ? There are some dynamic allocated stats that may never be destroyed
gschier
commented
2 years ago How about number of stats ? There are some dynamic allocated stats that may never be destroyed
We disable all stats for our custom resources because Envoy can't handle that many. Here's a full dump of /stats
We disable stats for our resources with:
stats_config:
stats_matcher:
exclusion_list:
patterns:
- prefix: cluster.clst_
- prefix: sds.scrt_
- prefix: vhost.vhst_I think to make more progress on this we will need a heap profile. Can you work on getting one? I'm not sure where we are at with built in support for profiles with the new tcmalloc. @rojkov @wbpcode @jmarantz
wbpcode
commented
2 years ago The perf tools should also be ok to get some memory profiles. Then we can try to analyse it. Or could you provide some completely configuration to reproduce it? It hard for us to help to solve this problem if there is no a way to reproduce it. 🤔
JakeCooper
commented
2 years ago Gonna grab a heap profile/flame graph for this
EDIT: Apologies got sidetracked. Have it on my Calendar tomorrow to compile from scratch to get a heap profile/pprof
JakeCooper
commented
2 years ago 1/3 of the envoy replicas is now running the heap profile. Going to let it go through a cycle of climbing heap allocations and will come back with the pprof
Sidenote: Envoy normally take like 7 hours to build from source? I haven't built a massive C++ package from source in eons
wbpcode
commented
2 years ago Sidenote: Envoy normally take like 7 hours to build from source? I haven't built a massive C++ package from source in eons
It depends. You can edit the do_ci.sh to ignore the unit test to shorten this time. And the second pass building would be quickly.
JakeCooper
commented
2 years ago It appears the built version of Envoy isn't responding on 443.
Envoy brew binary [1.23]
Envoy built from bazel [master]
Any way to debug this?
Apr 29 02:37:41 production-serviceproxy-2 systemd[1]: Started Railway Service Proxy.
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: Starting tracking the heap
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:390] initializing epoch 0 (base id=0, hot restart version=11.104)
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:392] statically linked extensions:
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.request_id: envoy.request_id.uuid
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.http.cache: envoy.extensions.http.cache.simple
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.upstreams: envoy.filters.connection_pools.tcp.generic
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.thrift_proxy.transports: auto, framed, header, unframed
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.network.dns_resolver: envoy.network.dns_resolver.cares
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, match-wrapper
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.dubbo_proxy.protocols: dubbo
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.986][4877][info][main] [source/server/server.cc:394] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.common.key_value: envoy.key_value.file_based
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.resolvers: envoy.ip
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.access_logger.extension_filters: envoy.access_loggers.extension_filters.cel
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.matching.action: composite-action, skip
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.dubbo_proxy.serializers: dubbo.hessian2
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.transport_protocol
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] network.connection.client: default, envoy_internal
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.http.stateful_header_formatters: preserve_case
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.retry_priorities: envoy.retry_priorities.previous_priorities
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.http.stateful_session: envoy.http.stateful_session.cookie
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.health_checkers: envoy.health_checkers.redis
Apr 29 02:37:41 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:41.987][4877][info][main] [source/server/server.cc:394] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.089][4877][info][main] [source/server/server.cc:442] HTTP header map info:
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.097][4877][info][main] [source/server/server.cc:445] request header map: 656 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-proto,x-ot-span-context,x-request-id
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.097][4877][info][main] [source/server/server.cc:445] request trailer map: 128 bytes:
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.097][4877][info][main] [source/server/server.cc:445] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.098][4877][info][main] [source/server/server.cc:445] response trailer map: 152 bytes: grpc-message,grpc-status
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.218][4877][info][main] [source/server/server.cc:786] runtime: {}
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.228][4877][info][admin] [source/server/admin/admin.cc:134] admin address: 0.0.0.0:19000
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.236][4877][info][config] [source/server/configuration_impl.cc:127] loading tracing configuration
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.236][4877][info][config] [source/server/configuration_impl.cc:87] loading 0 static secret(s)
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.236][4877][info][config] [source/server/configuration_impl.cc:93] loading 1 cluster(s)
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.313][4877][info][config] [source/server/configuration_impl.cc:97] loading 0 listener(s)
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.313][4877][info][config] [source/server/configuration_impl.cc:109] loading stats configuration
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.315][4877][info][runtime] [source/common/runtime/runtime_impl.cc:462] RTDS has finished initialization
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.315][4877][info][upstream] [source/common/upstream/cluster_manager_impl.cc:204] cm init: initializing cds
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.320][4877][warning][main] [source/server/server.cc:761] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
Apr 29 02:37:42 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:42.324][4877][info][main] [source/server/server.cc:882] starting main dispatch loop
Apr 29 02:37:48 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:37:48.933][4877][info][upstream] [source/common/upstream/cds_api_helper.cc:30] cds: add 21176 cluster(s), remove 0 cluster(s)
Apr 29 02:37:57 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0001.heap (100 MB currently in use)
Apr 29 02:38:18 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0002.heap (200 MB currently in use)
Apr 29 02:38:39 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0003.heap (300 MB currently in use)
Apr 29 02:38:59 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0004.heap (400 MB currently in use)
Apr 29 02:39:21 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0005.heap (500 MB currently in use)
Apr 29 02:39:42 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0006.heap (600 MB currently in use)
Apr 29 02:40:02 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0007.heap (700 MB currently in use)
Apr 29 02:40:24 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0008.heap (800 MB currently in use)
Apr 29 02:40:45 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0009.heap (900 MB currently in use)
Apr 29 02:41:06 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:41:06.183][4877][info][upstream] [source/common/upstream/cds_api_helper.cc:67] cds: added/updated 21176 cluster(s), skipped 0 unmodified cluster(s)
Apr 29 02:41:06 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:41:06.219][4877][info][upstream] [source/common/upstream/cluster_manager_impl.cc:208] cm init: all clusters initialized
Apr 29 02:41:06 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:41:06.219][4877][info][main] [source/server/server.cc:863] all clusters initialized. initializing init manager
Apr 29 02:41:06 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:41:06.487][4877][warning][config] [./source/common/config/grpc_stream.h:160] DeltaAggregatedResources gRPC config stream closed: 13,
Apr 29 02:41:06 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:41:06.488][4877][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener
Apr 29 02:41:06 production-serviceproxy-2 envoy[4877]: [2022-04-29 02:41:06.488][4877][info][config] [source/server/listener_manager_impl.cc:789] all dependencies initialized. starting workers
Apr 29 02:41:09 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0010.heap (1000 MB currently in use)
Apr 29 02:41:17 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0011.heap (1100 MB currently in use)
Apr 29 02:41:25 production-serviceproxy-2 envoy[4877]: Dumping heap profile to /tmp/railwayenvoy.heapprof.0012.heap (1200 MB currently in use)
AI've also let it run for about 30 minutes+ but I'll let it go for longer.
Usually takes about 10-15m to start up, I assume given the number of routes/clusters we've got
wbpcode
commented
2 years ago You can check the stats of the listener 443. If this listener is working or the cert is correct? And the 404 like the route not found error.
JakeCooper
commented
2 years ago This one?
{
"@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
"dynamic_listeners": [
{
"name": "https_listener",
"error_state": {
"failed_configuration": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "https_listener",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 443
}
},
"filter_chains": [
{
"filter_chain_match": {
"server_names": [
"rc-prod.radiologycored.com"
]
},
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "https",
"rds": {
"config_source": {
"ads": {},
"resource_api_version": "V3"
},
--
"h2",
"http/1.1"
]
}
}
}
}
},
"last_update_attempt": "2022-04-29T02:54:21.173Z",
"details": "Didn't find a registered implementation for 'envoy.filters.listener.tls_inspector' with type URL: ''"
}
},
suniltheta
commented
2 years ago Should be same as https://github.com/envoyproxy/envoy/issues/20919. You can set envoy.reloadable_features.no_extension_lookup_by_name to false.
JakeCooper
commented
2 years ago Perfect. Now returning empty responses. Once the routes are loaded, I'll force some traffic through that instance and circle back with the heap profiles
Thank you so much for the lightning quick responses!
JakeCooper
commented
2 years ago Got the pprof heap files! Just went off this but please let me know if there's anything else I can get to make this easier to debug.
mattklein123
commented
2 years ago Can you potentially do some pre-processing of the data following the instructions here? https://gperftools.github.io/gperftools/heapprofile.html. You should be able to generate graphs of the heap as well as differential diffs of the heap which should make it pretty clear where the extra memory is coming from. It will also confirm that symbols are correct.
wbpcode
commented
2 years ago @JakeCooper Hi, thanks very much for you hard works.
As matt said, it would be better to do some pre-processing of these data. Or can you provide your envoy binary that used to generate these data? Then we can try to process these data by ourself. (If you compiled the envoy based on the opensource code base, there should be no privacy problem.)
JakeCooper
commented
2 years ago I'm happy to run anything you'd like! Additionally here's the binary
Let me know what command you'd want run, or any other files. Here to make this as simple as possible and really appreciate all your help!
mattklein123
commented
2 years ago @JakeCooper unfortunately the version that you built doesn't have symbols (fastbuild). You will need to build with something like:
bazel build //source/exe:envoy-static --define tcmalloc=gperftools -c optOh man. Welp, guess I'll try again. Build should be faster now that we've done it once though. Will come back with another binary and set of stuff
mattklein123
commented
2 years ago Unfortunately it won't be faster since you have to rebuild everything to get optimized code with symbols. I would try to use a bigger machine if possible.
JakeCooper
commented
2 years ago It's already operating on a 96 core, 256gb box :S
Also just to clarify: I built it with bazel build --define tcmalloc=gperftools //source/exe:envoy-static. The -c opt is the part that's missing correct? Wanna make sure I get this correct
mattklein123
commented
2 years ago Yes that is correct. Something is really wrong if it's taking 7 hours to build on that machine. It should build in probably 10-20 minutes I would guess.
JakeCooper
commented
2 years ago Indeed. It's almost like Bazel just isn't using more than 1 core :/
Yellow = Instance compiling Envoy
That's fine. It'll get there eventually and I'll throw it up later tonight. Happy weekend!
JakeCooper
commented
2 years ago Got the below. I'm attempting to rerun it with CFLAGS=-Wno-error. Please correct me if this is incorrect, again it's been eons since I used cpp.
New Command: CFLAGS=-Wno-error bazel build //source/exe:envoy-static --define tcmalloc=gperftools -c opt
Old Error:
ERROR: /home/conductor/envoy/source/common/http/BUILD:367:17: Compiling source/common/http/header_map_impl.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 111 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox
In file included from external/com_google_absl/absl/container/inlined_vector.h:53,
from external/com_google_absl/absl/strings/cord.h:79,
from external/com_google_absl/absl/container/internal/hash_function_defaults.h:56,
from external/com_google_absl/absl/container/flat_hash_map.h:41,
from ./source/common/common/fancy_logger.h:7,
from ./source/common/common/logger.h:13,
from ./source/common/common/assert.h:5,
from ./envoy/config/typed_config.h:5,
from ./envoy/http/header_formatter.h:4,
from ./envoy/http/header_map.h:13,
from ./source/common/http/header_map_impl.h:11,
from source/common/http/header_map_impl.cc:1:
external/com_google_absl/absl/container/internal/inlined_vector.h: In constructor 'Envoy::Http::HeaderString::HeaderString()':
external/com_google_absl/absl/container/internal/inlined_vector.h:448:5: error: '<anonymous>.absl::inlined_vector_internal::Storage<char, 128, std::allocator<char> >::data_' is used uninitialized in this function [-Werror=uninitialized]
data_ = other_storage.data_;
^~~~~
cc1plus: all warnings being treated as errors
Target //source/exe:envoy-static failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6754.716s, Critical Path: 4332.65s
INFO: 3868 processes: 1172 internal, 1 local, 2695 processwrapper-sandbox.
FAILED: Build did NOT complete successfullyIf you are using a cloud instance, the problem is probably building against something like ebs. Make sure you use a local ssd and build on that.
The version of gcc you are using is too old. You would probably be better of using the docker container to build as it will replicate CI. See the instructions in the ci directory.
JakeCooper
commented
1 year ago Alrighty. Circling back here as this has grown a LOT and are now prioritizing resolving it
We continue to notice memory climbing, and so we've run a heap profile
Here's a couple of em
Our Edge Envoys (v1.25.1): edge_27g.txt
Our Cluster Envoys (v1.25.1): stacker-at-30g.txt
for the edge proxies, we have a single listener with ~ 12k filter chains, each filter chain entry has a domain match, one downstream TLS context, and a HCM Filter which points to a routing table via RDS.
We reference a single route config (via RDS) for all filter chain entries. This routing table has ~70k vhosts (each matching 1-2 domains), with a route to cluster action. Traffic routes to ~4 ish clusters depending on which middle proxy set we route to.
One of the filter chains is a wildcard match, (eg: *.up.railway.app), and maps to around 60k vhosts, The other 10k filter chains map 1:1 to 10k vhosts for individual domains.
Our current RouteConfig size is ~ 18MB and is updated around once every 2-5 seconds.
Our Listener config size is updated every 3-8 minutes (mostly to add/remove a filter chain). We're using the default 10min --drain-time-s setting.
CC @char8
wbpcode
commented
1 year ago are there lots of long-live http stream or requests?
char8
commented
1 year ago are there lots of long-live http stream or requests?
would long lived http streams show up in the downstream_rq_time metric? our p99 varies between 3-8s there.
We do definitely have websocket connections, not sure about average duration of these, but looking at 1xx ret codes, there are about 3/sec compared to about 600/sec 2xx responses on each edge proxy. [assuming all 1xx responses are 101 upgrades for websockets].
wbpcode
commented
1 year ago As I know, envoy will keep a copy of shared pointer of route table in every active http request (normal http1.1 request, http2 request, http2 stream, websocket request) to ensure the route config will still alive in the whole lifetime of the request.
It means if you route table is refreshed by the RDS, the old route table will not be released immediately. If you route table is refreshed frequently and there are lots of long-live http request, then it's possible to cache lots of old versions tables in your memory and make a illusion of memory leak.
is this a possible cause of this problem? cc @mattklein123
Could you reproduce this problem in your test env? If you can, maybe you can disable all the upgrade and check if this problem could be resolved or not?
mattklein123
commented
1 year ago Our Listener config size is updated every 3-8 minutes (mostly to add/remove a filter chain). We're using the default 10min --drain-time-s setting.
If you are allowing listeners to drain for 10 minutes and you are updating every 3-8 minutes I think by definition you are going to have infinitely climbing memory. Can you drop the drain time to something like 1 minute so it's conservatively less than the update frequency? Granted, the updater should be optimizing to only drain the impacted filter chains, but I could still see this being a problem. (Are you changing every filter chain or only some of them every 3-8 minutes?)
From a quick look at the profile it does look like either the route tables are leaking somehow or there are just too many of them sitting around. Can you actually generate some graph view cyclic charts from the data so we can see more clearly the flow of data in terms of allocation? It will be easier to understand. See https://github.com/envoyproxy/envoy/blob/main/bazel/PPROF.md#analyzing-with-pprof and check out the pprof CLI options to see how to spit out a visual view of the profile.
Regarding the listener updates, take a look at the stats here: https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats#listener-manager. In particular check the listener_in_place_updated stat. If that is not incrementing the majority of the time when you do updates I think that is probably the root of the problem here and we would need to think a bit differently in terms of how you are using envoy / configuring it.
char8
commented
1 year ago Thanks both for the ideas!
Could you reproduce this problem in your test env? If you can, maybe you can disable all the upgrade and check if this problem could be resolved or not?
We're currently working on testing the route config theory by a) trying to make our route configs smaller by about 10x, b) doing a ramp of longlived http connections over an hour to see if we can see similar mem growth. We did test with 5k websockets, but it was a very short ramp to 10k, so the active connections likely only referenced a handful of route configs that changed during the ramp.
Can you actually generate some graph view cyclic charts from the data so we can see more clearly the flow of data in terms of allocation? It will be easier to understand.
do these work?
would each request hold onto a ref for just the matching vhost/route or the entire routeconfig?
re: Listener updates, I forgot but the cluster hosts (the stacker-at-30g profile) don't do any listener updates after startup, but they still show this large mem growth. Looks like the lds inplace update counts roughly follow the lds update attempts count (off by 1) for the edge proxies. See plots below.
We suspected listener updates might have been the issue, especially after watching this talk from EnvoyCon, but we haven't been able to see any changes despite changing LDS update frequency, etc...
We only modify a very small subset of filter chains (mostly to change the SDS secret name after cert renewal), most of the changes are likely to be adding or removing low numbers (< 5) of filter chains per update. We try to keep the order of the filter chains list stable between updates as well.
wbpcode
commented
1 year ago would each request hold onto a ref for just the matching vhost/route or the entire routeconfig?
The entire route config.
mattklein123
commented
1 year ago Thanks those graphs are very helpful. Yes it looks like the issue is many route tables that are in memory.
Our current RouteConfig size is ~ 18MB and is updated around once every 2-5 seconds.
Yeah I think this is probably the issue and I think @wbpcode analysis is correct.
would each request hold onto a ref for just the matching vhost/route or the entire routeconfig?
I had to do some code splunking, but I think the fundamental issue is what you outline, we are snapping the entire route table vs. just the route.
Untangling this is going to be difficult but it should probably involve not snapping the entire config and making the config more granular so only the parts that are actually needed would be saved for the request.
@wbpcode is this something that you would be willing to tackle? It seems right up your alley. :)
In the interim I don't see any solution other than to make your route configs smaller and/or make them more granular where you return different RDS responses for each filter chain.
wbpcode
commented
1 year ago Will you create websocket requests gradually and keep the frequency of refreshing route config? I mean create 5 websocket every second (and ensure these websocket requests have a long time lifetime such as 10min or 15min) and update route config every 3 seconds, for example. By this way, there should be lots old versions route config be cached.
wbpcode
commented
1 year ago @wbpcode is this something that you would be willing to tackle? It seems right up your alley. :)
I will have a try. Whether this is the root cause or not, we still could do some optimizations.
I think we can wrapper the shared part of vh to a object and keep a pointer to this object in every route.
By this way, the request will only ref a single route and the shared part of the vh.
And the shared part of whole route config will be referred by the shared part of vh.
char8
commented
1 year ago Will you create websocket requests gradually and keep the frequency of refreshing route config? I mean create 5 websocket every second (and ensure these websocket requests have a long time lifetime such as 10min or 15min) and update route config every 3 seconds, for example. By this way, there should be lots old versions route config be cached.
Yep! plan on testing this today. I also saw #26045 🙏, so if I can repro the mem growth, will fire off a local build of it and see if I can compare results.
char8
commented
1 year ago Confirmed via a test, rough figures with an 18mb~ route config and ramping up to 8k websocket connections. Base mem use at idle was about ~ 1.5GB. 1) 32GB peak RAM use @ 0.1 RDS updates per second 2) 8GB peak RAM use @ 0.02 RDS updates per second 3) 1.5GB peak RAM use @ 0.02 RDS updates per second with a build of #26045 - can't see any difference in the mem usage graph during the test 🎉
We're using Envoy as our edge proxy at Railway (20k routes, virtual hosts, cluster; 5k secrets) and started running into an issue where heap grows uncontrollably. There was already some leaking, but it increased dramatically after switching from Dockerized Envoy 1.19 to Linux Brew Envoy 1.21. We're currently restarting each of our nodes every few hours to get around this (as noticeable on the posted graphs below).
My question is, does anything jump out immediately as to why this might be happening, and what is the best way to go about debugging this issue if not?
af50070ee60866874b0a9383daf9364e884ded22/1.21.1/Modified/RELEASE/BoringSSLon DebianLet me know if you need any additional info.
📊 Here's also a dump of /stats from one of the nodes:
``` cluster_manager.cds.version_text: "snapshot-2022-04-12T17:59:50Z-536" control_plane.identifier: "" http.http.rds.http_route_config.version_text: "snapshot-2022-04-12T17:59:50Z-378" http.https.rds.https_route_config.version_text: "snapshot-2022-04-12T17:59:50Z-536" listener_manager.lds.version_text: "snapshot-2022-04-12T17:59:50Z-536" cluster.http_cluster.assignment_stale: 0 cluster.http_cluster.assignment_timeout_received: 0 cluster.http_cluster.bind_errors: 0 cluster.http_cluster.circuit_breakers.default.cx_open: 0 cluster.http_cluster.circuit_breakers.default.cx_pool_open: 0 cluster.http_cluster.circuit_breakers.default.rq_open: 0 cluster.http_cluster.circuit_breakers.default.rq_pending_open: 0 cluster.http_cluster.circuit_breakers.default.rq_retry_open: 0 cluster.http_cluster.circuit_breakers.high.cx_open: 0 cluster.http_cluster.circuit_breakers.high.cx_pool_open: 0 cluster.http_cluster.circuit_breakers.high.rq_open: 0 cluster.http_cluster.circuit_breakers.high.rq_pending_open: 0 cluster.http_cluster.circuit_breakers.high.rq_retry_open: 0 cluster.http_cluster.default.total_match_count: 1 cluster.http_cluster.external.upstream_rq_404: 310 cluster.http_cluster.external.upstream_rq_4xx: 310 cluster.http_cluster.external.upstream_rq_completed: 310 cluster.http_cluster.http1.dropped_headers_with_underscores: 0 cluster.http_cluster.http1.metadata_not_supported_error: 0 cluster.http_cluster.http1.requests_rejected_with_underscores_in_headers: 0 cluster.http_cluster.http1.response_flood: 0 cluster.http_cluster.lb_healthy_panic: 0 cluster.http_cluster.lb_local_cluster_not_ok: 0 cluster.http_cluster.lb_recalculate_zone_structures: 0 cluster.http_cluster.lb_subsets_active: 0 cluster.http_cluster.lb_subsets_created: 0 cluster.http_cluster.lb_subsets_fallback: 0 cluster.http_cluster.lb_subsets_fallback_panic: 0 cluster.http_cluster.lb_subsets_removed: 0 cluster.http_cluster.lb_subsets_selected: 0 cluster.http_cluster.lb_zone_cluster_too_small: 0 cluster.http_cluster.lb_zone_no_capacity_left: 0 cluster.http_cluster.lb_zone_number_differs: 0 cluster.http_cluster.lb_zone_routing_all_directly: 0 cluster.http_cluster.lb_zone_routing_cross_zone: 0 cluster.http_cluster.lb_zone_routing_sampled: 0 cluster.http_cluster.max_host_weight: 0 cluster.http_cluster.membership_change: 1 cluster.http_cluster.membership_degraded: 0 cluster.http_cluster.membership_excluded: 0 cluster.http_cluster.membership_healthy: 1 cluster.http_cluster.membership_total: 1 cluster.http_cluster.original_dst_host_invalid: 0 cluster.http_cluster.retry_or_shadow_abandoned: 0 cluster.http_cluster.update_attempt: 0 cluster.http_cluster.update_empty: 0 cluster.http_cluster.update_failure: 0 cluster.http_cluster.update_no_rebuild: 0 cluster.http_cluster.update_success: 0 cluster.http_cluster.upstream_cx_active: 0 cluster.http_cluster.upstream_cx_close_notify: 0 cluster.http_cluster.upstream_cx_connect_attempts_exceeded: 0 cluster.http_cluster.upstream_cx_connect_fail: 0 cluster.http_cluster.upstream_cx_connect_timeout: 0 cluster.http_cluster.upstream_cx_destroy: 306 cluster.http_cluster.upstream_cx_destroy_local: 0 cluster.http_cluster.upstream_cx_destroy_local_with_active_rq: 0 cluster.http_cluster.upstream_cx_destroy_remote: 306 cluster.http_cluster.upstream_cx_destroy_remote_with_active_rq: 0 cluster.http_cluster.upstream_cx_destroy_with_active_rq: 0 cluster.http_cluster.upstream_cx_http1_total: 306 cluster.http_cluster.upstream_cx_http2_total: 0 cluster.http_cluster.upstream_cx_http3_total: 0 cluster.http_cluster.upstream_cx_idle_timeout: 0 cluster.http_cluster.upstream_cx_max_duration_reached: 0 cluster.http_cluster.upstream_cx_max_requests: 0 cluster.http_cluster.upstream_cx_none_healthy: 0 cluster.http_cluster.upstream_cx_overflow: 0 cluster.http_cluster.upstream_cx_pool_overflow: 0 cluster.http_cluster.upstream_cx_protocol_error: 0 cluster.http_cluster.upstream_cx_rx_bytes_buffered: 0 cluster.http_cluster.upstream_cx_rx_bytes_total: 54560 cluster.http_cluster.upstream_cx_total: 306 cluster.http_cluster.upstream_cx_tx_bytes_buffered: 0 cluster.http_cluster.upstream_cx_tx_bytes_total: 148563 cluster.http_cluster.upstream_flow_control_backed_up_total: 0 cluster.http_cluster.upstream_flow_control_drained_total: 0 cluster.http_cluster.upstream_flow_control_paused_reading_total: 0 cluster.http_cluster.upstream_flow_control_resumed_reading_total: 0 cluster.http_cluster.upstream_internal_redirect_failed_total: 0 cluster.http_cluster.upstream_internal_redirect_succeeded_total: 0 cluster.http_cluster.upstream_rq_404: 310 cluster.http_cluster.upstream_rq_4xx: 310 cluster.http_cluster.upstream_rq_active: 0 cluster.http_cluster.upstream_rq_cancelled: 0 cluster.http_cluster.upstream_rq_completed: 310 cluster.http_cluster.upstream_rq_maintenance_mode: 0 cluster.http_cluster.upstream_rq_max_duration_reached: 0 cluster.http_cluster.upstream_rq_pending_active: 0 cluster.http_cluster.upstream_rq_pending_failure_eject: 0 cluster.http_cluster.upstream_rq_pending_overflow: 0 cluster.http_cluster.upstream_rq_pending_total: 306 cluster.http_cluster.upstream_rq_per_try_idle_timeout: 0 cluster.http_cluster.upstream_rq_per_try_timeout: 0 cluster.http_cluster.upstream_rq_retry: 0 cluster.http_cluster.upstream_rq_retry_backoff_exponential: 0 cluster.http_cluster.upstream_rq_retry_backoff_ratelimited: 0 cluster.http_cluster.upstream_rq_retry_limit_exceeded: 0 cluster.http_cluster.upstream_rq_retry_overflow: 0 cluster.http_cluster.upstream_rq_retry_success: 0 cluster.http_cluster.upstream_rq_rx_reset: 0 cluster.http_cluster.upstream_rq_timeout: 0 cluster.http_cluster.upstream_rq_total: 310 cluster.http_cluster.upstream_rq_tx_reset: 0 cluster.http_cluster.version: 0 cluster.railway_xds_cluster.assignment_stale: 0 cluster.railway_xds_cluster.assignment_timeout_received: 0 cluster.railway_xds_cluster.bind_errors: 0 cluster.railway_xds_cluster.circuit_breakers.default.cx_open: 0 cluster.railway_xds_cluster.circuit_breakers.default.cx_pool_open: 0 cluster.railway_xds_cluster.circuit_breakers.default.rq_open: 0 cluster.railway_xds_cluster.circuit_breakers.default.rq_pending_open: 0 cluster.railway_xds_cluster.circuit_breakers.default.rq_retry_open: 0 cluster.railway_xds_cluster.circuit_breakers.high.cx_open: 0 cluster.railway_xds_cluster.circuit_breakers.high.cx_pool_open: 0 cluster.railway_xds_cluster.circuit_breakers.high.rq_open: 0 cluster.railway_xds_cluster.circuit_breakers.high.rq_pending_open: 0 cluster.railway_xds_cluster.circuit_breakers.high.rq_retry_open: 0 cluster.railway_xds_cluster.default.total_match_count: 1 cluster.railway_xds_cluster.http2.dropped_headers_with_underscores: 0 cluster.railway_xds_cluster.http2.header_overflow: 0 cluster.railway_xds_cluster.http2.headers_cb_no_stream: 0 cluster.railway_xds_cluster.http2.inbound_empty_frames_flood: 0 cluster.railway_xds_cluster.http2.inbound_priority_frames_flood: 0 cluster.railway_xds_cluster.http2.inbound_window_update_frames_flood: 0 cluster.railway_xds_cluster.http2.keepalive_timeout: 0 cluster.railway_xds_cluster.http2.metadata_empty_frames: 0 cluster.railway_xds_cluster.http2.outbound_control_flood: 0 cluster.railway_xds_cluster.http2.outbound_flood: 0 cluster.railway_xds_cluster.http2.pending_send_bytes: 0 cluster.railway_xds_cluster.http2.requests_rejected_with_underscores_in_headers: 0 cluster.railway_xds_cluster.http2.rx_messaging_error: 0 cluster.railway_xds_cluster.http2.rx_reset: 0 cluster.railway_xds_cluster.http2.stream_refused_errors: 0 cluster.railway_xds_cluster.http2.streams_active: 1 cluster.railway_xds_cluster.http2.trailers: 0 cluster.railway_xds_cluster.http2.tx_flush_timeout: 0 cluster.railway_xds_cluster.http2.tx_reset: 0 cluster.railway_xds_cluster.internal.upstream_rq_200: 1 cluster.railway_xds_cluster.internal.upstream_rq_2xx: 1 cluster.railway_xds_cluster.internal.upstream_rq_completed: 1 cluster.railway_xds_cluster.lb_healthy_panic: 0 cluster.railway_xds_cluster.lb_local_cluster_not_ok: 0 cluster.railway_xds_cluster.lb_recalculate_zone_structures: 0 cluster.railway_xds_cluster.lb_subsets_active: 0 cluster.railway_xds_cluster.lb_subsets_created: 0 cluster.railway_xds_cluster.lb_subsets_fallback: 0 cluster.railway_xds_cluster.lb_subsets_fallback_panic: 0 cluster.railway_xds_cluster.lb_subsets_removed: 0 cluster.railway_xds_cluster.lb_subsets_selected: 0 cluster.railway_xds_cluster.lb_zone_cluster_too_small: 0 cluster.railway_xds_cluster.lb_zone_no_capacity_left: 0 cluster.railway_xds_cluster.lb_zone_number_differs: 0 cluster.railway_xds_cluster.lb_zone_routing_all_directly: 0 cluster.railway_xds_cluster.lb_zone_routing_cross_zone: 0 cluster.railway_xds_cluster.lb_zone_routing_sampled: 0 cluster.railway_xds_cluster.max_host_weight: 0 cluster.railway_xds_cluster.membership_change: 1 cluster.railway_xds_cluster.membership_degraded: 0 cluster.railway_xds_cluster.membership_excluded: 0 cluster.railway_xds_cluster.membership_healthy: 1 cluster.railway_xds_cluster.membership_total: 1 cluster.railway_xds_cluster.original_dst_host_invalid: 0 cluster.railway_xds_cluster.retry_or_shadow_abandoned: 0 cluster.railway_xds_cluster.update_attempt: 0 cluster.railway_xds_cluster.update_empty: 0 cluster.railway_xds_cluster.update_failure: 0 cluster.railway_xds_cluster.update_no_rebuild: 0 cluster.railway_xds_cluster.update_success: 0 cluster.railway_xds_cluster.upstream_cx_active: 1 cluster.railway_xds_cluster.upstream_cx_close_notify: 0 cluster.railway_xds_cluster.upstream_cx_connect_attempts_exceeded: 0 cluster.railway_xds_cluster.upstream_cx_connect_fail: 0 cluster.railway_xds_cluster.upstream_cx_connect_timeout: 0 cluster.railway_xds_cluster.upstream_cx_destroy: 0 cluster.railway_xds_cluster.upstream_cx_destroy_local: 0 cluster.railway_xds_cluster.upstream_cx_destroy_local_with_active_rq: 0 cluster.railway_xds_cluster.upstream_cx_destroy_remote: 0 cluster.railway_xds_cluster.upstream_cx_destroy_remote_with_active_rq: 0 cluster.railway_xds_cluster.upstream_cx_destroy_with_active_rq: 0 cluster.railway_xds_cluster.upstream_cx_http1_total: 0 cluster.railway_xds_cluster.upstream_cx_http2_total: 1 cluster.railway_xds_cluster.upstream_cx_http3_total: 0 cluster.railway_xds_cluster.upstream_cx_idle_timeout: 0 cluster.railway_xds_cluster.upstream_cx_max_duration_reached: 0 cluster.railway_xds_cluster.upstream_cx_max_requests: 0 cluster.railway_xds_cluster.upstream_cx_none_healthy: 0 cluster.railway_xds_cluster.upstream_cx_overflow: 0 cluster.railway_xds_cluster.upstream_cx_pool_overflow: 0 cluster.railway_xds_cluster.upstream_cx_protocol_error: 0 cluster.railway_xds_cluster.upstream_cx_rx_bytes_buffered: 30 cluster.railway_xds_cluster.upstream_cx_rx_bytes_total: 2296019362 cluster.railway_xds_cluster.upstream_cx_total: 1 cluster.railway_xds_cluster.upstream_cx_tx_bytes_buffered: 0 cluster.railway_xds_cluster.upstream_cx_tx_bytes_total: 51817804 cluster.railway_xds_cluster.upstream_flow_control_backed_up_total: 0 cluster.railway_xds_cluster.upstream_flow_control_drained_total: 0 cluster.railway_xds_cluster.upstream_flow_control_paused_reading_total: 0 cluster.railway_xds_cluster.upstream_flow_control_resumed_reading_total: 0 cluster.railway_xds_cluster.upstream_internal_redirect_failed_total: 0 cluster.railway_xds_cluster.upstream_internal_redirect_succeeded_total: 0 cluster.railway_xds_cluster.upstream_rq_200: 1 cluster.railway_xds_cluster.upstream_rq_2xx: 1 cluster.railway_xds_cluster.upstream_rq_active: 1 cluster.railway_xds_cluster.upstream_rq_cancelled: 0 cluster.railway_xds_cluster.upstream_rq_completed: 1 cluster.railway_xds_cluster.upstream_rq_maintenance_mode: 0 cluster.railway_xds_cluster.upstream_rq_max_duration_reached: 0 cluster.railway_xds_cluster.upstream_rq_pending_active: 0 cluster.railway_xds_cluster.upstream_rq_pending_failure_eject: 0 cluster.railway_xds_cluster.upstream_rq_pending_overflow: 0 cluster.railway_xds_cluster.upstream_rq_pending_total: 1 cluster.railway_xds_cluster.upstream_rq_per_try_idle_timeout: 0 cluster.railway_xds_cluster.upstream_rq_per_try_timeout: 0 cluster.railway_xds_cluster.upstream_rq_retry: 0 cluster.railway_xds_cluster.upstream_rq_retry_backoff_exponential: 0 cluster.railway_xds_cluster.upstream_rq_retry_backoff_ratelimited: 0 cluster.railway_xds_cluster.upstream_rq_retry_limit_exceeded: 0 cluster.railway_xds_cluster.upstream_rq_retry_overflow: 0 cluster.railway_xds_cluster.upstream_rq_retry_success: 0 cluster.railway_xds_cluster.upstream_rq_rx_reset: 0 cluster.railway_xds_cluster.upstream_rq_timeout: 0 cluster.railway_xds_cluster.upstream_rq_total: 1 cluster.railway_xds_cluster.upstream_rq_tx_reset: 0 cluster.railway_xds_cluster.version: 0 cluster_manager.active_clusters: 19947 cluster_manager.cds.init_fetch_timeout: 0 cluster_manager.cds.update_attempt: 120 cluster_manager.cds.update_failure: 0 cluster_manager.cds.update_rejected: 0 cluster_manager.cds.update_success: 119 cluster_manager.cds.update_time: 1649802536288 cluster_manager.cds.version: 2486829869471955647 cluster_manager.cluster_added: 20028 cluster_manager.cluster_modified: 0 cluster_manager.cluster_removed: 81 cluster_manager.cluster_updated: 0 cluster_manager.cluster_updated_via_merge: 0 cluster_manager.update_merge_cancelled: 0 cluster_manager.update_out_of_merge_window: 0 cluster_manager.warming_clusters: 0 control_plane.connected_state: 1 control_plane.pending_requests: 0 control_plane.rate_limit_enforced: 0 envoy.overload_actions.reset_high_memory_stream.count: 0 filesystem.flushed_by_timer: 0 filesystem.reopen_failed: 0 filesystem.write_buffered: 0 filesystem.write_completed: 0 filesystem.write_failed: 0 filesystem.write_total_buffered: 0 http.admin.downstream_cx_active: 1 http.admin.downstream_cx_delayed_close_timeout: 0 http.admin.downstream_cx_destroy: 713 http.admin.downstream_cx_destroy_active_rq: 0 http.admin.downstream_cx_destroy_local: 0 http.admin.downstream_cx_destroy_local_active_rq: 0 http.admin.downstream_cx_destroy_remote: 713 http.admin.downstream_cx_destroy_remote_active_rq: 0 http.admin.downstream_cx_drain_close: 0 http.admin.downstream_cx_http1_active: 1 http.admin.downstream_cx_http1_total: 714 http.admin.downstream_cx_http2_active: 0 http.admin.downstream_cx_http2_total: 0 http.admin.downstream_cx_http3_active: 0 http.admin.downstream_cx_http3_total: 0 http.admin.downstream_cx_idle_timeout: 0 http.admin.downstream_cx_max_duration_reached: 0 http.admin.downstream_cx_max_requests_reached: 0 http.admin.downstream_cx_overload_disable_keepalive: 0 http.admin.downstream_cx_protocol_error: 0 http.admin.downstream_cx_rx_bytes_buffered: 84 http.admin.downstream_cx_rx_bytes_total: 111602 http.admin.downstream_cx_ssl_active: 0 http.admin.downstream_cx_ssl_total: 0 http.admin.downstream_cx_total: 714 http.admin.downstream_cx_tx_bytes_buffered: 0 http.admin.downstream_cx_tx_bytes_total: 22759646 http.admin.downstream_cx_upgrades_active: 0 http.admin.downstream_cx_upgrades_total: 0 http.admin.downstream_flow_control_paused_reading_total: 0 http.admin.downstream_flow_control_resumed_reading_total: 0 http.admin.downstream_rq_1xx: 0 http.admin.downstream_rq_2xx: 713 http.admin.downstream_rq_3xx: 0 http.admin.downstream_rq_4xx: 0 http.admin.downstream_rq_5xx: 0 http.admin.downstream_rq_active: 1 http.admin.downstream_rq_completed: 713 http.admin.downstream_rq_failed_path_normalization: 0 http.admin.downstream_rq_header_timeout: 0 http.admin.downstream_rq_http1_total: 714 http.admin.downstream_rq_http2_total: 0 http.admin.downstream_rq_http3_total: 0 http.admin.downstream_rq_idle_timeout: 0 http.admin.downstream_rq_max_duration_reached: 0 http.admin.downstream_rq_non_relative_path: 0 http.admin.downstream_rq_overload_close: 0 http.admin.downstream_rq_redirected_with_normalized_path: 0 http.admin.downstream_rq_rejected_via_ip_detection: 0 http.admin.downstream_rq_response_before_rq_complete: 0 http.admin.downstream_rq_rx_reset: 0 http.admin.downstream_rq_timeout: 0 http.admin.downstream_rq_too_large: 0 http.admin.downstream_rq_total: 714 http.admin.downstream_rq_tx_reset: 0 http.admin.downstream_rq_ws_on_non_ws_route: 0 http.admin.rs_too_large: 0 http.async-client.no_cluster: 0 http.async-client.no_route: 0 http.async-client.passthrough_internal_redirect_bad_location: 0 http.async-client.passthrough_internal_redirect_no_route: 0 http.async-client.passthrough_internal_redirect_predicate: 0 http.async-client.passthrough_internal_redirect_too_many_redirects: 0 http.async-client.passthrough_internal_redirect_unsafe_scheme: 0 http.async-client.rq_direct_response: 0 http.async-client.rq_redirect: 0 http.async-client.rq_reset_after_downstream_response_started: 0 http.async-client.rq_total: 1 http.http.downstream_cx_active: 1224 http.http.downstream_cx_delayed_close_timeout: 339 http.http.downstream_cx_destroy: 1399 http.http.downstream_cx_destroy_active_rq: 0 http.http.downstream_cx_destroy_local: 339 http.http.downstream_cx_destroy_local_active_rq: 0 http.http.downstream_cx_destroy_remote: 1060 http.http.downstream_cx_destroy_remote_active_rq: 0 http.http.downstream_cx_drain_close: 0 http.http.downstream_cx_http1_active: 1224 http.http.downstream_cx_http1_total: 2623 http.http.downstream_cx_http2_active: 0 http.http.downstream_cx_http2_total: 0 http.http.downstream_cx_http3_active: 0 http.http.downstream_cx_http3_total: 0 http.http.downstream_cx_idle_timeout: 0 http.http.downstream_cx_max_duration_reached: 0 http.http.downstream_cx_max_requests_reached: 0 http.http.downstream_cx_overload_disable_keepalive: 0 http.http.downstream_cx_protocol_error: 2 http.http.downstream_cx_rx_bytes_buffered: 694759 http.http.downstream_cx_rx_bytes_total: 1675285917 http.http.downstream_cx_ssl_active: 0 http.http.downstream_cx_ssl_total: 0 http.http.downstream_cx_total: 2623 http.http.downstream_cx_tx_bytes_buffered: 0 http.http.downstream_cx_tx_bytes_total: 1172707534 http.http.downstream_cx_upgrades_active: 0 http.http.downstream_cx_upgrades_total: 0 http.http.downstream_flow_control_paused_reading_total: 0 http.http.downstream_flow_control_resumed_reading_total: 0 http.http.downstream_rq_1xx: 0 http.http.downstream_rq_2xx: 0 http.http.downstream_rq_3xx: 2956423 http.http.downstream_rq_4xx: 315 http.http.downstream_rq_5xx: 0 http.http.downstream_rq_active: 0 http.http.downstream_rq_completed: 2956738 http.http.downstream_rq_failed_path_normalization: 0 http.http.downstream_rq_header_timeout: 0 http.http.downstream_rq_http1_total: 2956738 http.http.downstream_rq_http2_total: 0 http.http.downstream_rq_http3_total: 0 http.http.downstream_rq_idle_timeout: 0 http.http.downstream_rq_max_duration_reached: 0 http.http.downstream_rq_non_relative_path: 0 http.http.downstream_rq_overload_close: 0 http.http.downstream_rq_redirected_with_normalized_path: 0 http.http.downstream_rq_rejected_via_ip_detection: 0 http.http.downstream_rq_response_before_rq_complete: 106 http.http.downstream_rq_rx_reset: 106 http.http.downstream_rq_timeout: 0 http.http.downstream_rq_too_large: 0 http.http.downstream_rq_total: 2956738 http.http.downstream_rq_tx_reset: 0 http.http.downstream_rq_ws_on_non_ws_route: 0 http.http.no_cluster: 0 http.http.no_route: 2 http.http.passthrough_internal_redirect_bad_location: 0 http.http.passthrough_internal_redirect_no_route: 0 http.http.passthrough_internal_redirect_predicate: 0 http.http.passthrough_internal_redirect_too_many_redirects: 0 http.http.passthrough_internal_redirect_unsafe_scheme: 0 http.http.rds.http_route_config.config_reload: 1 http.http.rds.http_route_config.config_reload_time_ms: 1649797937577 http.http.rds.http_route_config.init_fetch_timeout: 1 http.http.rds.http_route_config.update_attempt: 3 http.http.rds.http_route_config.update_empty: 0 http.http.rds.http_route_config.update_failure: 0 http.http.rds.http_route_config.update_rejected: 0 http.http.rds.http_route_config.update_success: 1 http.http.rds.http_route_config.update_time: 1649797937577 http.http.rds.http_route_config.version: 10194100587458576616 http.http.rq_direct_response: 2956423 http.http.rq_redirect: 0 http.http.rq_reset_after_downstream_response_started: 0 http.http.rq_total: 2956735 http.http.rs_too_large: 0 http.http.tracing.client_enabled: 0 http.http.tracing.health_check: 0 http.http.tracing.not_traceable: 0 http.http.tracing.random_sampling: 0 http.http.tracing.service_forced: 0 http.https.downstream_cx_active: 5643 http.https.downstream_cx_delayed_close_timeout: 3542 http.https.downstream_cx_destroy: 175304 http.https.downstream_cx_destroy_active_rq: 5329 http.https.downstream_cx_destroy_local: 9729 http.https.downstream_cx_destroy_local_active_rq: 0 http.https.downstream_cx_destroy_remote: 165575 http.https.downstream_cx_destroy_remote_active_rq: 5329 http.https.downstream_cx_drain_close: 40644 http.https.downstream_cx_http1_active: 990 http.https.downstream_cx_http1_total: 123681 http.https.downstream_cx_http2_active: 4624 http.https.downstream_cx_http2_total: 53927 http.https.downstream_cx_http3_active: 0 http.https.downstream_cx_http3_total: 0 http.https.downstream_cx_idle_timeout: 18 http.https.downstream_cx_max_duration_reached: 0 http.https.downstream_cx_max_requests_reached: 0 http.https.downstream_cx_overload_disable_keepalive: 0 http.https.downstream_cx_protocol_error: 12 http.https.downstream_cx_rx_bytes_buffered: 1189909 http.https.downstream_cx_rx_bytes_total: 3785524095 http.https.downstream_cx_ssl_active: 5643 http.https.downstream_cx_ssl_total: 180947 http.https.downstream_cx_total: 180947 http.https.downstream_cx_tx_bytes_buffered: 4930064 http.https.downstream_cx_tx_bytes_total: 11391226786 http.https.downstream_cx_upgrades_active: 933 http.https.downstream_cx_upgrades_total: 10883 http.https.downstream_flow_control_paused_reading_total: 28 http.https.downstream_flow_control_resumed_reading_total: 28 http.https.downstream_rq_1xx: 6351 http.https.downstream_rq_2xx: 906891 http.https.downstream_rq_3xx: 13561 http.https.downstream_rq_4xx: 103747 http.https.downstream_rq_5xx: 11552 http.https.downstream_rq_active: 943 http.https.downstream_rq_completed: 1042102 http.https.downstream_rq_failed_path_normalization: 0 http.https.downstream_rq_header_timeout: 0 http.https.downstream_rq_http1_total: 325770 http.https.downstream_rq_http2_total: 717528 http.https.downstream_rq_http3_total: 0 http.https.downstream_rq_idle_timeout: 42 http.https.downstream_rq_max_duration_reached: 0 http.https.downstream_rq_non_relative_path: 0 http.https.downstream_rq_overload_close: 0 http.https.downstream_rq_redirected_with_normalized_path: 0 http.https.downstream_rq_rejected_via_ip_detection: 0 http.https.downstream_rq_response_before_rq_complete: 15012 http.https.downstream_rq_rx_reset: 15473 http.https.downstream_rq_timeout: 0 http.https.downstream_rq_too_large: 0 http.https.downstream_rq_total: 1043298 http.https.downstream_rq_tx_reset: 1390 http.https.downstream_rq_ws_on_non_ws_route: 0 http.https.no_cluster: 10 http.https.no_route: 100 http.https.passthrough_internal_redirect_bad_location: 0 http.https.passthrough_internal_redirect_no_route: 0 http.https.passthrough_internal_redirect_predicate: 0 http.https.passthrough_internal_redirect_too_many_redirects: 0 http.https.passthrough_internal_redirect_unsafe_scheme: 0 http.https.rds.https_route_config.config_reload: 142 http.https.rds.https_route_config.config_reload_time_ms: 1649802538356 http.https.rds.https_route_config.init_fetch_timeout: 1 http.https.rds.https_route_config.update_attempt: 144 http.https.rds.https_route_config.update_empty: 0 http.https.rds.https_route_config.update_failure: 0 http.https.rds.https_route_config.update_rejected: 0 http.https.rds.https_route_config.update_success: 142 http.https.rds.https_route_config.update_time: 1649802538357 http.https.rds.https_route_config.version: 2486829869471955647 http.https.rq_direct_response: 28262 http.https.rq_redirect: 0 http.https.rq_reset_after_downstream_response_started: 0 http.https.rq_total: 1042353 http.https.rs_too_large: 0 http.https.tracing.client_enabled: 0 http.https.tracing.health_check: 0 http.https.tracing.not_traceable: 0 http.https.tracing.random_sampling: 0 http.https.tracing.service_forced: 0 http.https.user_agent.android.downstream_cx_destroy_remote_active_rq: 1 http.https.user_agent.android.downstream_cx_total: 41976 http.https.user_agent.android.downstream_rq_total: 41976 http.https.user_agent.ios.downstream_cx_destroy_remote_active_rq: 0 http.https.user_agent.ios.downstream_cx_total: 245 http.https.user_agent.ios.downstream_rq_total: 245 http1.dropped_headers_with_underscores: 0 http1.metadata_not_supported_error: 0 http1.requests_rejected_with_underscores_in_headers: 0 http1.response_flood: 0 http2.dropped_headers_with_underscores: 0 http2.header_overflow: 0 http2.headers_cb_no_stream: 0 http2.inbound_empty_frames_flood: 0 http2.inbound_priority_frames_flood: 0 http2.inbound_window_update_frames_flood: 0 http2.keepalive_timeout: 0 http2.metadata_empty_frames: 0 http2.outbound_control_flood: 0 http2.outbound_flood: 0 http2.pending_send_bytes: 0 http2.requests_rejected_with_underscores_in_headers: 0 http2.rx_messaging_error: 0 http2.rx_reset: 87 http2.stream_refused_errors: 0 http2.streams_active: 7 http2.trailers: 0 http2.tx_flush_timeout: 0 http2.tx_reset: 2667 http_inspector.http10_found: 1 http_inspector.http11_found: 2620 http_inspector.http2_found: 0 http_inspector.http_not_found: 2 http_inspector.read_error: 3 listener.0.0.0.0_443.downstream_cx_active: 5643 listener.0.0.0.0_443.downstream_cx_destroy: 175304 listener.0.0.0.0_443.downstream_cx_overflow: 0 listener.0.0.0.0_443.downstream_cx_overload_reject: 0 listener.0.0.0.0_443.downstream_cx_total: 180947 listener.0.0.0.0_443.downstream_cx_transport_socket_connect_timeout: 0 listener.0.0.0.0_443.downstream_global_cx_overflow: 0 listener.0.0.0.0_443.downstream_pre_cx_active: 3 listener.0.0.0.0_443.downstream_pre_cx_timeout: 65 listener.0.0.0.0_443.http.https.downstream_rq_1xx: 6351 listener.0.0.0.0_443.http.https.downstream_rq_2xx: 906891 listener.0.0.0.0_443.http.https.downstream_rq_3xx: 13561 listener.0.0.0.0_443.http.https.downstream_rq_4xx: 103747 listener.0.0.0.0_443.http.https.downstream_rq_5xx: 11552 listener.0.0.0.0_443.http.https.downstream_rq_completed: 1042102 listener.0.0.0.0_443.no_filter_chain_match: 0 listener.0.0.0.0_443.server_ssl_socket_factory.downstream_context_secrets_not_ready: 0 listener.0.0.0.0_443.server_ssl_socket_factory.ssl_context_update_by_sds: 1986 listener.0.0.0.0_443.server_ssl_socket_factory.upstream_context_secrets_not_ready: 0 listener.0.0.0.0_443.ssl.ciphers.AES128-SHA: 2 listener.0.0.0.0_443.ssl.ciphers.ECDHE-RSA-AES128-GCM-SHA256: 817 listener.0.0.0.0_443.ssl.ciphers.ECDHE-RSA-AES128-SHA: 972 listener.0.0.0.0_443.ssl.ciphers.ECDHE-RSA-CHACHA20-POLY1305: 47131 listener.0.0.0.0_443.ssl.ciphers.TLS_AES_128_GCM_SHA256: 14968 listener.0.0.0.0_443.ssl.ciphers.TLS_AES_256_GCM_SHA384: 114174 listener.0.0.0.0_443.ssl.ciphers.TLS_CHACHA20_POLY1305_SHA256: 487 listener.0.0.0.0_443.ssl.connection_error: 765 listener.0.0.0.0_443.ssl.curves.P-256: 1607 listener.0.0.0.0_443.ssl.curves.X25519: 176942 listener.0.0.0.0_443.ssl.fail_verify_cert_hash: 0 listener.0.0.0.0_443.ssl.fail_verify_error: 0 listener.0.0.0.0_443.ssl.fail_verify_no_cert: 0 listener.0.0.0.0_443.ssl.fail_verify_san: 0 listener.0.0.0.0_443.ssl.handshake: 178551 listener.0.0.0.0_443.ssl.no_certificate: 178551 listener.0.0.0.0_443.ssl.ocsp_staple_failed: 0 listener.0.0.0.0_443.ssl.ocsp_staple_omitted: 12656 listener.0.0.0.0_443.ssl.ocsp_staple_requests: 12656 listener.0.0.0.0_443.ssl.ocsp_staple_responses: 0 listener.0.0.0.0_443.ssl.session_reused: 29364 listener.0.0.0.0_443.ssl.versions.TLSv1: 31 listener.0.0.0.0_443.ssl.versions.TLSv1.1: 1 listener.0.0.0.0_443.ssl.versions.TLSv1.2: 48890 listener.0.0.0.0_443.ssl.versions.TLSv1.3: 129629 listener.0.0.0.0_443.worker_0.downstream_cx_active: 83 listener.0.0.0.0_443.worker_0.downstream_cx_total: 2859 listener.0.0.0.0_443.worker_1.downstream_cx_active: 90 listener.0.0.0.0_443.worker_1.downstream_cx_total: 2950 listener.0.0.0.0_443.worker_10.downstream_cx_active: 86 listener.0.0.0.0_443.worker_10.downstream_cx_total: 2859 listener.0.0.0.0_443.worker_11.downstream_cx_active: 94 listener.0.0.0.0_443.worker_11.downstream_cx_total: 2841 listener.0.0.0.0_443.worker_12.downstream_cx_active: 88 listener.0.0.0.0_443.worker_12.downstream_cx_total: 2875 listener.0.0.0.0_443.worker_13.downstream_cx_active: 89 listener.0.0.0.0_443.worker_13.downstream_cx_total: 2827 listener.0.0.0.0_443.worker_14.downstream_cx_active: 70 listener.0.0.0.0_443.worker_14.downstream_cx_total: 2893 listener.0.0.0.0_443.worker_15.downstream_cx_active: 87 listener.0.0.0.0_443.worker_15.downstream_cx_total: 2854 listener.0.0.0.0_443.worker_16.downstream_cx_active: 75 listener.0.0.0.0_443.worker_16.downstream_cx_total: 2815 listener.0.0.0.0_443.worker_17.downstream_cx_active: 77 listener.0.0.0.0_443.worker_17.downstream_cx_total: 2873 listener.0.0.0.0_443.worker_18.downstream_cx_active: 77 listener.0.0.0.0_443.worker_18.downstream_cx_total: 2731 listener.0.0.0.0_443.worker_19.downstream_cx_active: 89 listener.0.0.0.0_443.worker_19.downstream_cx_total: 2883 listener.0.0.0.0_443.worker_2.downstream_cx_active: 109 listener.0.0.0.0_443.worker_2.downstream_cx_total: 2756 listener.0.0.0.0_443.worker_20.downstream_cx_active: 80 listener.0.0.0.0_443.worker_20.downstream_cx_total: 2813 listener.0.0.0.0_443.worker_21.downstream_cx_active: 94 listener.0.0.0.0_443.worker_21.downstream_cx_total: 2878 listener.0.0.0.0_443.worker_22.downstream_cx_active: 90 listener.0.0.0.0_443.worker_22.downstream_cx_total: 2796 listener.0.0.0.0_443.worker_23.downstream_cx_active: 76 listener.0.0.0.0_443.worker_23.downstream_cx_total: 2830 listener.0.0.0.0_443.worker_24.downstream_cx_active: 81 listener.0.0.0.0_443.worker_24.downstream_cx_total: 2841 listener.0.0.0.0_443.worker_25.downstream_cx_active: 104 listener.0.0.0.0_443.worker_25.downstream_cx_total: 2885 listener.0.0.0.0_443.worker_26.downstream_cx_active: 85 listener.0.0.0.0_443.worker_26.downstream_cx_total: 2766 listener.0.0.0.0_443.worker_27.downstream_cx_active: 99 listener.0.0.0.0_443.worker_27.downstream_cx_total: 2724 listener.0.0.0.0_443.worker_28.downstream_cx_active: 92 listener.0.0.0.0_443.worker_28.downstream_cx_total: 2980 listener.0.0.0.0_443.worker_29.downstream_cx_active: 100 listener.0.0.0.0_443.worker_29.downstream_cx_total: 2892 listener.0.0.0.0_443.worker_3.downstream_cx_active: 90 listener.0.0.0.0_443.worker_3.downstream_cx_total: 2678 listener.0.0.0.0_443.worker_30.downstream_cx_active: 83 listener.0.0.0.0_443.worker_30.downstream_cx_total: 2785 listener.0.0.0.0_443.worker_31.downstream_cx_active: 81 listener.0.0.0.0_443.worker_31.downstream_cx_total: 2768 listener.0.0.0.0_443.worker_32.downstream_cx_active: 88 listener.0.0.0.0_443.worker_32.downstream_cx_total: 2720 listener.0.0.0.0_443.worker_33.downstream_cx_active: 81 listener.0.0.0.0_443.worker_33.downstream_cx_total: 2876 listener.0.0.0.0_443.worker_34.downstream_cx_active: 74 listener.0.0.0.0_443.worker_34.downstream_cx_total: 2806 listener.0.0.0.0_443.worker_35.downstream_cx_active: 81 listener.0.0.0.0_443.worker_35.downstream_cx_total: 2835 listener.0.0.0.0_443.worker_36.downstream_cx_active: 77 listener.0.0.0.0_443.worker_36.downstream_cx_total: 2822 listener.0.0.0.0_443.worker_37.downstream_cx_active: 95 listener.0.0.0.0_443.worker_37.downstream_cx_total: 2843 listener.0.0.0.0_443.worker_38.downstream_cx_active: 95 listener.0.0.0.0_443.worker_38.downstream_cx_total: 2952 listener.0.0.0.0_443.worker_39.downstream_cx_active: 88 listener.0.0.0.0_443.worker_39.downstream_cx_total: 2679 listener.0.0.0.0_443.worker_4.downstream_cx_active: 100 listener.0.0.0.0_443.worker_4.downstream_cx_total: 2772 listener.0.0.0.0_443.worker_40.downstream_cx_active: 96 listener.0.0.0.0_443.worker_40.downstream_cx_total: 2844 listener.0.0.0.0_443.worker_41.downstream_cx_active: 71 listener.0.0.0.0_443.worker_41.downstream_cx_total: 2707 listener.0.0.0.0_443.worker_42.downstream_cx_active: 93 listener.0.0.0.0_443.worker_42.downstream_cx_total: 2769 listener.0.0.0.0_443.worker_43.downstream_cx_active: 88 listener.0.0.0.0_443.worker_43.downstream_cx_total: 2755 listener.0.0.0.0_443.worker_44.downstream_cx_active: 86 listener.0.0.0.0_443.worker_44.downstream_cx_total: 2808 listener.0.0.0.0_443.worker_45.downstream_cx_active: 90 listener.0.0.0.0_443.worker_45.downstream_cx_total: 2727 listener.0.0.0.0_443.worker_46.downstream_cx_active: 89 listener.0.0.0.0_443.worker_46.downstream_cx_total: 2803 listener.0.0.0.0_443.worker_47.downstream_cx_active: 95 listener.0.0.0.0_443.worker_47.downstream_cx_total: 2880 listener.0.0.0.0_443.worker_48.downstream_cx_active: 97 listener.0.0.0.0_443.worker_48.downstream_cx_total: 2770 listener.0.0.0.0_443.worker_49.downstream_cx_active: 88 listener.0.0.0.0_443.worker_49.downstream_cx_total: 2769 listener.0.0.0.0_443.worker_5.downstream_cx_active: 82 listener.0.0.0.0_443.worker_5.downstream_cx_total: 2887 listener.0.0.0.0_443.worker_50.downstream_cx_active: 90 listener.0.0.0.0_443.worker_50.downstream_cx_total: 2879 listener.0.0.0.0_443.worker_51.downstream_cx_active: 84 listener.0.0.0.0_443.worker_51.downstream_cx_total: 2852 listener.0.0.0.0_443.worker_52.downstream_cx_active: 76 listener.0.0.0.0_443.worker_52.downstream_cx_total: 2912 listener.0.0.0.0_443.worker_53.downstream_cx_active: 85 listener.0.0.0.0_443.worker_53.downstream_cx_total: 2939 listener.0.0.0.0_443.worker_54.downstream_cx_active: 102 listener.0.0.0.0_443.worker_54.downstream_cx_total: 2843 listener.0.0.0.0_443.worker_55.downstream_cx_active: 106 listener.0.0.0.0_443.worker_55.downstream_cx_total: 2727 listener.0.0.0.0_443.worker_56.downstream_cx_active: 95 listener.0.0.0.0_443.worker_56.downstream_cx_total: 2787 listener.0.0.0.0_443.worker_57.downstream_cx_active: 86 listener.0.0.0.0_443.worker_57.downstream_cx_total: 2865 listener.0.0.0.0_443.worker_58.downstream_cx_active: 76 listener.0.0.0.0_443.worker_58.downstream_cx_total: 2922 listener.0.0.0.0_443.worker_59.downstream_cx_active: 100 listener.0.0.0.0_443.worker_59.downstream_cx_total: 2951 listener.0.0.0.0_443.worker_6.downstream_cx_active: 102 listener.0.0.0.0_443.worker_6.downstream_cx_total: 2979 listener.0.0.0.0_443.worker_60.downstream_cx_active: 93 listener.0.0.0.0_443.worker_60.downstream_cx_total: 2860 listener.0.0.0.0_443.worker_61.downstream_cx_active: 70 listener.0.0.0.0_443.worker_61.downstream_cx_total: 2786 listener.0.0.0.0_443.worker_62.downstream_cx_active: 93 listener.0.0.0.0_443.worker_62.downstream_cx_total: 2948 listener.0.0.0.0_443.worker_63.downstream_cx_active: 88 listener.0.0.0.0_443.worker_63.downstream_cx_total: 2785 listener.0.0.0.0_443.worker_7.downstream_cx_active: 99 listener.0.0.0.0_443.worker_7.downstream_cx_total: 2795 listener.0.0.0.0_443.worker_8.downstream_cx_active: 74 listener.0.0.0.0_443.worker_8.downstream_cx_total: 2613 listener.0.0.0.0_443.worker_9.downstream_cx_active: 101 listener.0.0.0.0_443.worker_9.downstream_cx_total: 2828 listener.0.0.0.0_80.downstream_cx_active: 1224 listener.0.0.0.0_80.downstream_cx_destroy: 1399 listener.0.0.0.0_80.downstream_cx_overflow: 0 listener.0.0.0.0_80.downstream_cx_overload_reject: 0 listener.0.0.0.0_80.downstream_cx_total: 2623 listener.0.0.0.0_80.downstream_cx_transport_socket_connect_timeout: 0 listener.0.0.0.0_80.downstream_global_cx_overflow: 0 listener.0.0.0.0_80.downstream_pre_cx_active: 0 listener.0.0.0.0_80.downstream_pre_cx_timeout: 6 listener.0.0.0.0_80.http.http.downstream_rq_1xx: 0 listener.0.0.0.0_80.http.http.downstream_rq_2xx: 0 listener.0.0.0.0_80.http.http.downstream_rq_3xx: 2956422 listener.0.0.0.0_80.http.http.downstream_rq_4xx: 315 listener.0.0.0.0_80.http.http.downstream_rq_5xx: 0 listener.0.0.0.0_80.http.http.downstream_rq_completed: 2956738 listener.0.0.0.0_80.no_filter_chain_match: 0 listener.0.0.0.0_80.worker_0.downstream_cx_active: 12 listener.0.0.0.0_80.worker_0.downstream_cx_total: 42 listener.0.0.0.0_80.worker_1.downstream_cx_active: 17 listener.0.0.0.0_80.worker_1.downstream_cx_total: 40 listener.0.0.0.0_80.worker_10.downstream_cx_active: 25 listener.0.0.0.0_80.worker_10.downstream_cx_total: 44 listener.0.0.0.0_80.worker_11.downstream_cx_active: 12 listener.0.0.0.0_80.worker_11.downstream_cx_total: 33 listener.0.0.0.0_80.worker_12.downstream_cx_active: 24 listener.0.0.0.0_80.worker_12.downstream_cx_total: 52 listener.0.0.0.0_80.worker_13.downstream_cx_active: 21 listener.0.0.0.0_80.worker_13.downstream_cx_total: 44 listener.0.0.0.0_80.worker_14.downstream_cx_active: 25 listener.0.0.0.0_80.worker_14.downstream_cx_total: 47 listener.0.0.0.0_80.worker_15.downstream_cx_active: 20 listener.0.0.0.0_80.worker_15.downstream_cx_total: 43 listener.0.0.0.0_80.worker_16.downstream_cx_active: 24 listener.0.0.0.0_80.worker_16.downstream_cx_total: 52 listener.0.0.0.0_80.worker_17.downstream_cx_active: 21 listener.0.0.0.0_80.worker_17.downstream_cx_total: 45 listener.0.0.0.0_80.worker_18.downstream_cx_active: 15 listener.0.0.0.0_80.worker_18.downstream_cx_total: 40 listener.0.0.0.0_80.worker_19.downstream_cx_active: 23 listener.0.0.0.0_80.worker_19.downstream_cx_total: 48 listener.0.0.0.0_80.worker_2.downstream_cx_active: 18 listener.0.0.0.0_80.worker_2.downstream_cx_total: 39 listener.0.0.0.0_80.worker_20.downstream_cx_active: 16 listener.0.0.0.0_80.worker_20.downstream_cx_total: 36 listener.0.0.0.0_80.worker_21.downstream_cx_active: 13 listener.0.0.0.0_80.worker_21.downstream_cx_total: 31 listener.0.0.0.0_80.worker_22.downstream_cx_active: 13 listener.0.0.0.0_80.worker_22.downstream_cx_total: 36 listener.0.0.0.0_80.worker_23.downstream_cx_active: 24 listener.0.0.0.0_80.worker_23.downstream_cx_total: 47 listener.0.0.0.0_80.worker_24.downstream_cx_active: 15 listener.0.0.0.0_80.worker_24.downstream_cx_total: 34 listener.0.0.0.0_80.worker_25.downstream_cx_active: 19 listener.0.0.0.0_80.worker_25.downstream_cx_total: 51 listener.0.0.0.0_80.worker_26.downstream_cx_active: 21 listener.0.0.0.0_80.worker_26.downstream_cx_total: 42 listener.0.0.0.0_80.worker_27.downstream_cx_active: 23 listener.0.0.0.0_80.worker_27.downstream_cx_total: 47 listener.0.0.0.0_80.worker_28.downstream_cx_active: 25 listener.0.0.0.0_80.worker_28.downstream_cx_total: 45 listener.0.0.0.0_80.worker_29.downstream_cx_active: 16 listener.0.0.0.0_80.worker_29.downstream_cx_total: 34 listener.0.0.0.0_80.worker_3.downstream_cx_active: 16 listener.0.0.0.0_80.worker_3.downstream_cx_total: 47 listener.0.0.0.0_80.worker_30.downstream_cx_active: 19 listener.0.0.0.0_80.worker_30.downstream_cx_total: 47 listener.0.0.0.0_80.worker_31.downstream_cx_active: 14 listener.0.0.0.0_80.worker_31.downstream_cx_total: 31 listener.0.0.0.0_80.worker_32.downstream_cx_active: 19 listener.0.0.0.0_80.worker_32.downstream_cx_total: 42 listener.0.0.0.0_80.worker_33.downstream_cx_active: 21 listener.0.0.0.0_80.worker_33.downstream_cx_total: 48 listener.0.0.0.0_80.worker_34.downstream_cx_active: 28 listener.0.0.0.0_80.worker_34.downstream_cx_total: 53 listener.0.0.0.0_80.worker_35.downstream_cx_active: 15 listener.0.0.0.0_80.worker_35.downstream_cx_total: 43 listener.0.0.0.0_80.worker_36.downstream_cx_active: 17 listener.0.0.0.0_80.worker_36.downstream_cx_total: 43 listener.0.0.0.0_80.worker_37.downstream_cx_active: 19 listener.0.0.0.0_80.worker_37.downstream_cx_total: 39 listener.0.0.0.0_80.worker_38.downstream_cx_active: 16 listener.0.0.0.0_80.worker_38.downstream_cx_total: 37 listener.0.0.0.0_80.worker_39.downstream_cx_active: 19 listener.0.0.0.0_80.worker_39.downstream_cx_total: 43 listener.0.0.0.0_80.worker_4.downstream_cx_active: 17 listener.0.0.0.0_80.worker_4.downstream_cx_total: 28 listener.0.0.0.0_80.worker_40.downstream_cx_active: 27 listener.0.0.0.0_80.worker_40.downstream_cx_total: 49 listener.0.0.0.0_80.worker_41.downstream_cx_active: 12 listener.0.0.0.0_80.worker_41.downstream_cx_total: 31 listener.0.0.0.0_80.worker_42.downstream_cx_active: 21 listener.0.0.0.0_80.worker_42.downstream_cx_total: 47 listener.0.0.0.0_80.worker_43.downstream_cx_active: 16 listener.0.0.0.0_80.worker_43.downstream_cx_total: 32 listener.0.0.0.0_80.worker_44.downstream_cx_active: 20 listener.0.0.0.0_80.worker_44.downstream_cx_total: 33 listener.0.0.0.0_80.worker_45.downstream_cx_active: 24 listener.0.0.0.0_80.worker_45.downstream_cx_total: 41 listener.0.0.0.0_80.worker_46.downstream_cx_active: 20 listener.0.0.0.0_80.worker_46.downstream_cx_total: 42 listener.0.0.0.0_80.worker_47.downstream_cx_active: 19 listener.0.0.0.0_80.worker_47.downstream_cx_total: 40 listener.0.0.0.0_80.worker_48.downstream_cx_active: 17 listener.0.0.0.0_80.worker_48.downstream_cx_total: 32 listener.0.0.0.0_80.worker_49.downstream_cx_active: 19 listener.0.0.0.0_80.worker_49.downstream_cx_total: 34 listener.0.0.0.0_80.worker_5.downstream_cx_active: 18 listener.0.0.0.0_80.worker_5.downstream_cx_total: 42 listener.0.0.0.0_80.worker_50.downstream_cx_active: 26 listener.0.0.0.0_80.worker_50.downstream_cx_total: 52 listener.0.0.0.0_80.worker_51.downstream_cx_active: 20 listener.0.0.0.0_80.worker_51.downstream_cx_total: 45 listener.0.0.0.0_80.worker_52.downstream_cx_active: 19 listener.0.0.0.0_80.worker_52.downstream_cx_total: 40 listener.0.0.0.0_80.worker_53.downstream_cx_active: 19 listener.0.0.0.0_80.worker_53.downstream_cx_total: 46 listener.0.0.0.0_80.worker_54.downstream_cx_active: 22 listener.0.0.0.0_80.worker_54.downstream_cx_total: 45 listener.0.0.0.0_80.worker_55.downstream_cx_active: 24 listener.0.0.0.0_80.worker_55.downstream_cx_total: 54 listener.0.0.0.0_80.worker_56.downstream_cx_active: 12 listener.0.0.0.0_80.worker_56.downstream_cx_total: 34 listener.0.0.0.0_80.worker_57.downstream_cx_active: 13 listener.0.0.0.0_80.worker_57.downstream_cx_total: 32 listener.0.0.0.0_80.worker_58.downstream_cx_active: 19 listener.0.0.0.0_80.worker_58.downstream_cx_total: 39 listener.0.0.0.0_80.worker_59.downstream_cx_active: 23 listener.0.0.0.0_80.worker_59.downstream_cx_total: 42 listener.0.0.0.0_80.worker_6.downstream_cx_active: 24 listener.0.0.0.0_80.worker_6.downstream_cx_total: 39 listener.0.0.0.0_80.worker_60.downstream_cx_active: 22 listener.0.0.0.0_80.worker_60.downstream_cx_total: 40 listener.0.0.0.0_80.worker_61.downstream_cx_active: 15 listener.0.0.0.0_80.worker_61.downstream_cx_total: 34 listener.0.0.0.0_80.worker_62.downstream_cx_active: 19 listener.0.0.0.0_80.worker_62.downstream_cx_total: 41 listener.0.0.0.0_80.worker_63.downstream_cx_active: 19 listener.0.0.0.0_80.worker_63.downstream_cx_total: 35 listener.0.0.0.0_80.worker_7.downstream_cx_active: 22 listener.0.0.0.0_80.worker_7.downstream_cx_total: 41 listener.0.0.0.0_80.worker_8.downstream_cx_active: 20 listener.0.0.0.0_80.worker_8.downstream_cx_total: 42 listener.0.0.0.0_80.worker_9.downstream_cx_active: 11 listener.0.0.0.0_80.worker_9.downstream_cx_total: 26 listener.admin.downstream_cx_active: 1 listener.admin.downstream_cx_destroy: 713 listener.admin.downstream_cx_overflow: 0 listener.admin.downstream_cx_overload_reject: 0 listener.admin.downstream_cx_total: 714 listener.admin.downstream_cx_transport_socket_connect_timeout: 0 listener.admin.downstream_global_cx_overflow: 0 listener.admin.downstream_pre_cx_active: 0 listener.admin.downstream_pre_cx_timeout: 0 listener.admin.http.admin.downstream_rq_1xx: 0 listener.admin.http.admin.downstream_rq_2xx: 713 listener.admin.http.admin.downstream_rq_3xx: 0 listener.admin.http.admin.downstream_rq_4xx: 0 listener.admin.http.admin.downstream_rq_5xx: 0 listener.admin.http.admin.downstream_rq_completed: 713 listener.admin.main_thread.downstream_cx_active: 1 listener.admin.main_thread.downstream_cx_total: 714 listener.admin.no_filter_chain_match: 0 listener_manager.lds.init_fetch_timeout: 0 listener_manager.lds.update_attempt: 144 listener_manager.lds.update_failure: 0 listener_manager.lds.update_rejected: 0 listener_manager.lds.update_success: 143 listener_manager.lds.update_time: 1649802540829 listener_manager.lds.version: 2486829869471955647 listener_manager.listener_added: 2 listener_manager.listener_create_failure: 0 listener_manager.listener_create_success: 9216 listener_manager.listener_in_place_updated: 142 listener_manager.listener_modified: 142 listener_manager.listener_removed: 0 listener_manager.listener_stopped: 0 listener_manager.total_filter_chains_draining: 22 listener_manager.total_listeners_active: 2 listener_manager.total_listeners_draining: 0 listener_manager.total_listeners_warming: 0 listener_manager.workers_started: 1 main_thread.watchdog_mega_miss: 287 main_thread.watchdog_miss: 289 overload.envoy.overload_actions.shrink_heap.active: 0 overload.envoy.overload_actions.shrink_heap.scale_percent: 0 overload.envoy.overload_actions.shrink_heap.shrink_count: 0 overload.envoy.overload_actions.stop_accepting_requests.active: 0 overload.envoy.overload_actions.stop_accepting_requests.scale_percent: 0 overload.envoy.resource_monitors.fixed_heap.failed_updates: 0 overload.envoy.resource_monitors.fixed_heap.pressure: 23 overload.envoy.resource_monitors.fixed_heap.skipped_updates: 0 runtime.admin_overrides_active: 0 runtime.deprecated_feature_seen_since_process_start: 0 runtime.deprecated_feature_use: 0 runtime.load_error: 0 runtime.load_success: 1 runtime.num_keys: 0 runtime.num_layers: 0 runtime.override_dir_exists: 0 runtime.override_dir_not_exists: 1 server.compilation_settings.fips_mode: 0 server.concurrency: 64 server.days_until_first_cert_expiring: 0 server.debug_assertion_failures: 0 server.dropped_stat_flushes: 0 server.dynamic_unknown_fields: 0 server.envoy_bug_failures: 0 server.hot_restart_epoch: 0 server.hot_restart_generation: 1 server.live: 1 server.main_thread.watchdog_mega_miss: 287 server.main_thread.watchdog_miss: 289 server.memory_allocated: 22625564096 server.memory_heap_size: 24519901184 server.memory_physical_size: 23603857926 server.parent_connections: 0 server.seconds_until_first_ocsp_response_expiring: 0 server.state: 0 server.static_unknown_fields: 0 server.stats_recent_lookups: 6514310 server.total_connections: 6876 server.uptime: 5344 server.version: 11489287 server.wip_protos: 0 server.worker_0.watchdog_mega_miss: 0 server.worker_0.watchdog_miss: 0 server.worker_1.watchdog_mega_miss: 0 server.worker_1.watchdog_miss: 0 server.worker_10.watchdog_mega_miss: 0 server.worker_10.watchdog_miss: 1 server.worker_11.watchdog_mega_miss: 0 server.worker_11.watchdog_miss: 0 server.worker_12.watchdog_mega_miss: 0 server.worker_12.watchdog_miss: 1 server.worker_13.watchdog_mega_miss: 0 server.worker_13.watchdog_miss: 0 server.worker_14.watchdog_mega_miss: 0 server.worker_14.watchdog_miss: 0 server.worker_15.watchdog_mega_miss: 0 server.worker_15.watchdog_miss: 0 server.worker_16.watchdog_mega_miss: 0 server.worker_16.watchdog_miss: 0 server.worker_17.watchdog_mega_miss: 0 server.worker_17.watchdog_miss: 0 server.worker_18.watchdog_mega_miss: 0 server.worker_18.watchdog_miss: 0 server.worker_19.watchdog_mega_miss: 0 server.worker_19.watchdog_miss: 0 server.worker_2.watchdog_mega_miss: 0 server.worker_2.watchdog_miss: 0 server.worker_20.watchdog_mega_miss: 0 server.worker_20.watchdog_miss: 0 server.worker_21.watchdog_mega_miss: 0 server.worker_21.watchdog_miss: 0 server.worker_22.watchdog_mega_miss: 0 server.worker_22.watchdog_miss: 0 server.worker_23.watchdog_mega_miss: 0 server.worker_23.watchdog_miss: 0 server.worker_24.watchdog_mega_miss: 0 server.worker_24.watchdog_miss: 2 server.worker_25.watchdog_mega_miss: 0 server.worker_25.watchdog_miss: 0 server.worker_26.watchdog_mega_miss: 0 server.worker_26.watchdog_miss: 0 server.worker_27.watchdog_mega_miss: 0 server.worker_27.watchdog_miss: 0 server.worker_28.watchdog_mega_miss: 0 server.worker_28.watchdog_miss: 2 server.worker_29.watchdog_mega_miss: 0 server.worker_29.watchdog_miss: 0 server.worker_3.watchdog_mega_miss: 0 server.worker_3.watchdog_miss: 0 server.worker_30.watchdog_mega_miss: 0 server.worker_30.watchdog_miss: 0 server.worker_31.watchdog_mega_miss: 0 server.worker_31.watchdog_miss: 0 server.worker_32.watchdog_mega_miss: 0 server.worker_32.watchdog_miss: 1 server.worker_33.watchdog_mega_miss: 0 server.worker_33.watchdog_miss: 0 server.worker_34.watchdog_mega_miss: 0 server.worker_34.watchdog_miss: 1 server.worker_35.watchdog_mega_miss: 0 server.worker_35.watchdog_miss: 0 server.worker_36.watchdog_mega_miss: 0 server.worker_36.watchdog_miss: 0 server.worker_37.watchdog_mega_miss: 0 server.worker_37.watchdog_miss: 1 server.worker_38.watchdog_mega_miss: 0 server.worker_38.watchdog_miss: 1 server.worker_39.watchdog_mega_miss: 0 server.worker_39.watchdog_miss: 1 server.worker_4.watchdog_mega_miss: 0 server.worker_4.watchdog_miss: 0 server.worker_40.watchdog_mega_miss: 0 server.worker_40.watchdog_miss: 1 server.worker_41.watchdog_mega_miss: 0 server.worker_41.watchdog_miss: 1 server.worker_42.watchdog_mega_miss: 0 server.worker_42.watchdog_miss: 0 server.worker_43.watchdog_mega_miss: 0 server.worker_43.watchdog_miss: 0 server.worker_44.watchdog_mega_miss: 0 server.worker_44.watchdog_miss: 0 server.worker_45.watchdog_mega_miss: 0 server.worker_45.watchdog_miss: 1 server.worker_46.watchdog_mega_miss: 0 server.worker_46.watchdog_miss: 0 server.worker_47.watchdog_mega_miss: 0 server.worker_47.watchdog_miss: 0 server.worker_48.watchdog_mega_miss: 0 server.worker_48.watchdog_miss: 0 server.worker_49.watchdog_mega_miss: 0 server.worker_49.watchdog_miss: 0 server.worker_5.watchdog_mega_miss: 0 server.worker_5.watchdog_miss: 0 server.worker_50.watchdog_mega_miss: 0 server.worker_50.watchdog_miss: 0 server.worker_51.watchdog_mega_miss: 0 server.worker_51.watchdog_miss: 0 server.worker_52.watchdog_mega_miss: 0 server.worker_52.watchdog_miss: 1 server.worker_53.watchdog_mega_miss: 0 server.worker_53.watchdog_miss: 0 server.worker_54.watchdog_mega_miss: 0 server.worker_54.watchdog_miss: 0 server.worker_55.watchdog_mega_miss: 0 server.worker_55.watchdog_miss: 1 server.worker_56.watchdog_mega_miss: 0 server.worker_56.watchdog_miss: 0 server.worker_57.watchdog_mega_miss: 0 server.worker_57.watchdog_miss: 0 server.worker_58.watchdog_mega_miss: 0 server.worker_58.watchdog_miss: 1 server.worker_59.watchdog_mega_miss: 0 server.worker_59.watchdog_miss: 2 server.worker_6.watchdog_mega_miss: 0 server.worker_6.watchdog_miss: 0 server.worker_60.watchdog_mega_miss: 0 server.worker_60.watchdog_miss: 0 server.worker_61.watchdog_mega_miss: 0 server.worker_61.watchdog_miss: 2 server.worker_62.watchdog_mega_miss: 0 server.worker_62.watchdog_miss: 0 server.worker_63.watchdog_mega_miss: 0 server.worker_63.watchdog_miss: 0 server.worker_7.watchdog_mega_miss: 0 server.worker_7.watchdog_miss: 0 server.worker_8.watchdog_mega_miss: 0 server.worker_8.watchdog_miss: 0 server.worker_9.watchdog_mega_miss: 0 server.worker_9.watchdog_miss: 0 tls_inspector.alpn_found: 57616 tls_inspector.alpn_not_found: 123041 tls_inspector.client_hello_too_large: 0 tls_inspector.connection_closed: 1498 tls_inspector.read_error: 7 tls_inspector.sni_found: 135963 tls_inspector.sni_not_found: 44694 tls_inspector.tls_found: 180657 tls_inspector.tls_not_found: 290 vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry: 0 vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry_limit_exceeded: 0 vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry_overflow: 0 vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry_success: 0 vhost.fallback_virtualhost.vcluster.other.upstream_rq_timeout: 0 vhost.fallback_virtualhost.vcluster.other.upstream_rq_total: 0 vhost.health_virtualhost.vcluster.other.upstream_rq_retry: 0 vhost.health_virtualhost.vcluster.other.upstream_rq_retry_limit_exceeded: 0 vhost.health_virtualhost.vcluster.other.upstream_rq_retry_overflow: 0 vhost.health_virtualhost.vcluster.other.upstream_rq_retry_success: 0 vhost.health_virtualhost.vcluster.other.upstream_rq_timeout: 0 vhost.health_virtualhost.vcluster.other.upstream_rq_total: 0 vhost.http_virtualhost.vcluster.other.upstream_rq_retry: 0 vhost.http_virtualhost.vcluster.other.upstream_rq_retry_limit_exceeded: 0 vhost.http_virtualhost.vcluster.other.upstream_rq_retry_overflow: 0 vhost.http_virtualhost.vcluster.other.upstream_rq_retry_success: 0 vhost.http_virtualhost.vcluster.other.upstream_rq_timeout: 0 vhost.http_virtualhost.vcluster.other.upstream_rq_total: 0 workers.watchdog_mega_miss: 0 workers.watchdog_miss: 21 ```Also tagging @JakeCooper and @soggycactus from the team on this one.