Closed shoenig closed 3 years ago
After the reconfig the upstream listener is listed as draining_state
meaning envoy keeps it around for a period of time after change/removal. I mostly ever encountered this in the context of modifying the listener in which case the listening socket is transferred to the updated listener and the old one is orphaned to drain existing connections. I have yet to be in a scenario where there was no need for transfer like this.
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining
If you wait long enough do you see this envoy log on the downstream?
[2021-06-17 09:16:43.970][2409938][info][upstream] [source/server/lds_api.cc:60] lds: remove listener 'destination:127.0.0.1:8999'
I locally reproduced the scenario and by the time I was executing the curl it didn't work anymore.
It may have also been one of those fun connection pooling issues if you weren't explicitly switching the service to an L7 protocol. Connections would have been established when the now-draining listener was actively listening, but after the listening socket was destroyed if you were using a pooled long-lived connection from curl it wouldn't have re-dialed, instead preferring to use the established socket.
If you set this config entry out of band before redoing your test scenario does the errant behavior still manifest?
kind = "service-defaults"
name = "destination"
protocol = "http"
Annnnnd it did start finally failing, after about 12 minutes
2021/06/17 14:31:59 INFO [client] -> GET response code: (200)
2021/06/17 14:43:41 ERROR [client] -> GET error: Get "http://127.0.0.1:8999": dial tcp 127.0.0.1:8999: connect: connection refused
Feel free to close this out if that's working as intended, but this long of a lingering listener seems like a lot?
The envoy default value is 10m: https://www.envoyproxy.io/docs/envoy/latest/operations/cli.html?highlight=drain#cmdoption-drain-time-s
Awesome
While working on https://github.com/hashicorp/nomad/pull/10776 which fixes Nomad to update a Connect proxy's upstreams without restarting the proxy or parent task, I noticed this behavior where my tasks would still be able to make use of proxy upstreams even after updating the sidecar proxy to no longer have upstreams. Seems like the envoy should stop the listener when the upstream is removed. Not using ACLs or intentions.
Checking our service registrations has upstream as expected
Connection working as expected
Modify client service to remove upstreams
Run modified job
Consul synced service
Checking upstream removed from service definition
Notice connections still working (!)
Exec into container, lets look around
config dump response
``` { "configs": [ { "@type": "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump", "bootstrap": { "node": { "id": "_nomad-task-8db34479-3402-9437-de6e-15016f1e4fe8-group-client-client--sidecar-proxy", "cluster": "client", "metadata": { "namespace": "default", "envoy_version": "1.16.2" }, "hidden_envoy_deprecated_build_version": "e98e41a8e168af7acae8079fc0cd68155f699aa3/1.16.2/Clean/RELEASE/BoringSSL", "user_agent_name": "envoy", "user_agent_build_version": { "version": { "major_number": 1, "minor_number": 16, "patch": 2 }, "metadata": { "revision.status": "Clean", "revision.sha": "e98e41a8e168af7acae8079fc0cd68155f699aa3", "build.type": "RELEASE", "ssl.version": "BoringSSL" } }, "extensions": [ { "name": "envoy.transport_sockets.alts", "category": "envoy.transport_sockets.downstream" }, { "name": "envoy.transport_sockets.quic", "category": "envoy.transport_sockets.downstream" }, { "name": "envoy.transport_sockets.raw_buffer", "category": "envoy.transport_sockets.downstream" }, { "name": "envoy.transport_sockets.tap", "category": "envoy.transport_sockets.downstream" }, { "name": "envoy.transport_sockets.tls", "category": "envoy.transport_sockets.downstream" }, { "name": "raw_buffer", "category": "envoy.transport_sockets.downstream" }, { "name": "tls", "category": "envoy.transport_sockets.downstream" }, { "name": "quiche", "category": "envoy.quic_server_codec" }, { "name": "envoy.extensions.http.cache.simple", "category": "envoy.http.cache" }, { "name": "envoy.ip", "category": "envoy.resolvers" }, { "name": "dubbo.hessian2", "category": "envoy.dubbo_proxy.serializers" }, { "name": "envoy.filters.connection_pools.http.generic", "category": "envoy.upstreams" }, { "name": "envoy.filters.connection_pools.http.http", "category": "envoy.upstreams" }, { "name": "envoy.filters.connection_pools.http.tcp", "category": "envoy.upstreams" }, { "name": "default", "category": "envoy.dubbo_proxy.route_matchers" }, { "name": "envoy.compression.gzip.compressor", "category": "envoy.compression.compressor" }, { "name": "udp_default_writer", "category": "envoy.udp_packet_writers" }, { "name": "udp_gso_batch_writer", "category": "envoy.udp_packet_writers" }, { "name": "envoy.access_loggers.file", "category": "envoy.access_loggers" }, { "name": "envoy.access_loggers.http_grpc", "category": "envoy.access_loggers" }, { "name": "envoy.access_loggers.tcp_grpc", "category": "envoy.access_loggers" }, { "name": "envoy.file_access_log", "category": "envoy.access_loggers" }, { "name": "envoy.http_grpc_access_log", "category": "envoy.access_loggers" }, { "name": "envoy.tcp_grpc_access_log", "category": "envoy.access_loggers" }, { "name": "quiche_quic_listener", "category": "envoy.udp_listeners" }, { "name": "raw_udp_listener", "category": "envoy.udp_listeners" }, { "name": "envoy.internal_redirect_predicates.allow_listed_routes", "category": "envoy.internal_redirect_predicates" }, { "name": "envoy.internal_redirect_predicates.previous_routes", "category": "envoy.internal_redirect_predicates" }, { "name": "envoy.internal_redirect_predicates.safe_cross_scheme", "category": "envoy.internal_redirect_predicates" }, { "name": "dubbo", "category": "envoy.dubbo_proxy.protocols" }, { "name": "auto", "category": "envoy.thrift_proxy.transports" }, { "name": "framed", "category": "envoy.thrift_proxy.transports" }, { "name": "header", "category": "envoy.thrift_proxy.transports" }, { "name": "unframed", "category": "envoy.thrift_proxy.transports" }, { "name": "envoy.retry_priorities.previous_priorities", "category": "envoy.retry_priorities" }, { "name": "envoy.dog_statsd", "category": "envoy.stats_sinks" }, { "name": "envoy.metrics_service", "category": "envoy.stats_sinks" }, { "name": "envoy.stat_sinks.dog_statsd", "category": "envoy.stats_sinks" }, { "name": "envoy.stat_sinks.hystrix", "category": "envoy.stats_sinks" }, { "name": "envoy.stat_sinks.metrics_service", "category": "envoy.stats_sinks" }, { "name": "envoy.stat_sinks.statsd", "category": "envoy.stats_sinks" }, { "name": "envoy.statsd", "category": "envoy.stats_sinks" }, { "name": "envoy.dynamic.ot", "category": "envoy.tracers" }, { "name": "envoy.lightstep", "category": "envoy.tracers" }, { "name": "envoy.tracers.datadog", "category": "envoy.tracers" }, { "name": "envoy.tracers.dynamic_ot", "category": "envoy.tracers" }, { "name": "envoy.tracers.lightstep", "category": "envoy.tracers" }, { "name": "envoy.tracers.opencensus", "category": "envoy.tracers" }, { "name": "envoy.tracers.xray", "category": "envoy.tracers" }, { "name": "envoy.tracers.zipkin", "category": "envoy.tracers" }, { "name": "envoy.zipkin", "category": "envoy.tracers" }, { "name": "envoy.transport_sockets.alts", "category": "envoy.transport_sockets.upstream" }, { "name": "envoy.transport_sockets.quic", "category": "envoy.transport_sockets.upstream" }, { "name": "envoy.transport_sockets.raw_buffer", "category": "envoy.transport_sockets.upstream" }, { "name": "envoy.transport_sockets.tap", "category": "envoy.transport_sockets.upstream" }, { "name": "envoy.transport_sockets.tls", "category": "envoy.transport_sockets.upstream" }, { "name": "envoy.transport_sockets.upstream_proxy_protocol", "category": "envoy.transport_sockets.upstream" }, { "name": "raw_buffer", "category": "envoy.transport_sockets.upstream" }, { "name": "tls", "category": "envoy.transport_sockets.upstream" }, { "name": "envoy.resource_monitors.fixed_heap", "category": "envoy.resource_monitors" }, { "name": "envoy.resource_monitors.injected_resource", "category": "envoy.resource_monitors" }, { "name": "envoy.watchdog.abort_action", "category": "envoy.guarddog_actions" }, { "name": "envoy.watchdog.profile_action", "category": "envoy.guarddog_actions" }, { "name": "envoy.filters.listener.http_inspector", "category": "envoy.filters.listener" }, { "name": "envoy.filters.listener.original_dst", "category": "envoy.filters.listener" }, { "name": "envoy.filters.listener.original_src", "category": "envoy.filters.listener" }, { "name": "envoy.filters.listener.proxy_protocol", "category": "envoy.filters.listener" }, { "name": "envoy.filters.listener.tls_inspector", "category": "envoy.filters.listener" }, { "name": "envoy.listener.http_inspector", "category": "envoy.filters.listener" }, { "name": "envoy.listener.original_dst", "category": "envoy.filters.listener" }, { "name": "envoy.listener.original_src", "category": "envoy.filters.listener" }, { "name": "envoy.listener.proxy_protocol", "category": "envoy.filters.listener" }, { "name": "envoy.listener.tls_inspector", "category": "envoy.filters.listener" }, { "name": "envoy.retry_host_predicates.omit_canary_hosts", "category": "envoy.retry_host_predicates" }, { "name": "envoy.retry_host_predicates.omit_host_metadata", "category": "envoy.retry_host_predicates" }, { "name": "envoy.retry_host_predicates.previous_hosts", "category": "envoy.retry_host_predicates" }, { "name": "envoy.extensions.network.socket_interface.default_socket_interface", "category": "envoy.bootstrap" }, { "name": "envoy.cluster.eds", "category": "envoy.clusters" }, { "name": "envoy.cluster.logical_dns", "category": "envoy.clusters" }, { "name": "envoy.cluster.original_dst", "category": "envoy.clusters" }, { "name": "envoy.cluster.static", "category": "envoy.clusters" }, { "name": "envoy.cluster.strict_dns", "category": "envoy.clusters" }, { "name": "envoy.clusters.aggregate", "category": "envoy.clusters" }, { "name": "envoy.clusters.dynamic_forward_proxy", "category": "envoy.clusters" }, { "name": "envoy.clusters.redis", "category": "envoy.clusters" }, { "name": "envoy.grpc_credentials.aws_iam", "category": "envoy.grpc_credentials" }, { "name": "envoy.grpc_credentials.default", "category": "envoy.grpc_credentials" }, { "name": "envoy.grpc_credentials.file_based_metadata", "category": "envoy.grpc_credentials" }, { "name": "envoy.filters.thrift.rate_limit", "category": "envoy.thrift_proxy.filters" }, { "name": "envoy.filters.thrift.router", "category": "envoy.thrift_proxy.filters" }, { "name": "envoy.client_ssl_auth", "category": "envoy.filters.network" }, { "name": "envoy.echo", "category": "envoy.filters.network" }, { "name": "envoy.ext_authz", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.client_ssl_auth", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.direct_response", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.dubbo_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.echo", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.ext_authz", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.http_connection_manager", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.kafka_broker", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.local_ratelimit", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.mongo_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.mysql_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.postgres_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.ratelimit", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.rbac", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.redis_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.rocketmq_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.sni_cluster", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.sni_dynamic_forward_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.tcp_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.thrift_proxy", "category": "envoy.filters.network" }, { "name": "envoy.filters.network.zookeeper_proxy", "category": "envoy.filters.network" }, { "name": "envoy.http_connection_manager", "category": "envoy.filters.network" }, { "name": "envoy.mongo_proxy", "category": "envoy.filters.network" }, { "name": "envoy.ratelimit", "category": "envoy.filters.network" }, { "name": "envoy.redis_proxy", "category": "envoy.filters.network" }, { "name": "envoy.tcp_proxy", "category": "envoy.filters.network" }, { "name": "envoy.buffer", "category": "envoy.filters.http" }, { "name": "envoy.cors", "category": "envoy.filters.http" }, { "name": "envoy.csrf", "category": "envoy.filters.http" }, { "name": "envoy.ext_authz", "category": "envoy.filters.http" }, { "name": "envoy.fault", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.adaptive_concurrency", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.admission_control", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.aws_lambda", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.aws_request_signing", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.buffer", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.cache", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.cdn_loop", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.compressor", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.cors", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.csrf", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.decompressor", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.dynamic_forward_proxy", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.dynamo", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.ext_authz", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.fault", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.grpc_http1_bridge", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.grpc_http1_reverse_bridge", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.grpc_json_transcoder", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.grpc_stats", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.grpc_web", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.gzip", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.header_to_metadata", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.health_check", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.ip_tagging", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.jwt_authn", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.local_ratelimit", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.lua", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.oauth", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.on_demand", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.original_src", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.ratelimit", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.rbac", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.router", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.squash", "category": "envoy.filters.http" }, { "name": "envoy.filters.http.tap", "category": "envoy.filters.http" }, { "name": "envoy.grpc_http1_bridge", "category": "envoy.filters.http" }, { "name": "envoy.grpc_json_transcoder", "category": "envoy.filters.http" }, { "name": "envoy.grpc_web", "category": "envoy.filters.http" }, { "name": "envoy.gzip", "category": "envoy.filters.http" }, { "name": "envoy.health_check", "category": "envoy.filters.http" }, { "name": "envoy.http_dynamo_filter", "category": "envoy.filters.http" }, { "name": "envoy.ip_tagging", "category": "envoy.filters.http" }, { "name": "envoy.local_rate_limit", "category": "envoy.filters.http" }, { "name": "envoy.lua", "category": "envoy.filters.http" }, { "name": "envoy.rate_limit", "category": "envoy.filters.http" }, { "name": "envoy.router", "category": "envoy.filters.http" }, { "name": "envoy.squash", "category": "envoy.filters.http" }, { "name": "envoy.filters.udp.dns_filter", "category": "envoy.filters.udp_listener" }, { "name": "envoy.filters.udp_listener.udp_proxy", "category": "envoy.filters.udp_listener" }, { "name": "quiche", "category": "envoy.quic_client_codec" }, { "name": "envoy.filters.dubbo.router", "category": "envoy.dubbo_proxy.filters" }, { "name": "envoy.health_checkers.redis", "category": "envoy.health_checkers" }, { "name": "envoy.compression.gzip.decompressor", "category": "envoy.compression.decompressor" }, { "name": "auto", "category": "envoy.thrift_proxy.protocols" }, { "name": "binary", "category": "envoy.thrift_proxy.protocols" }, { "name": "binary/non-strict", "category": "envoy.thrift_proxy.protocols" }, { "name": "compact", "category": "envoy.thrift_proxy.protocols" }, { "name": "twitter", "category": "envoy.thrift_proxy.protocols" } ] }, "static_resources": { "clusters": [ { "name": "local_agent", "type": "STATIC", "connect_timeout": "1s", "hidden_envoy_deprecated_hosts": [ { "pipe": { "path": "alloc/tmp/consul_grpc.sock" } } ], "http2_protocol_options": {} } ] }, "dynamic_resources": { "lds_config": { "ads": {} }, "cds_config": { "ads": {} }, "ads_config": { "api_type": "GRPC", "grpc_services": [ { "envoy_grpc": { "cluster_name": "local_agent" }, "initial_metadata": [ { "key": "x-consul-token" } ] } ] } }, "admin": { "access_log_path": "/dev/null", "address": { "socket_address": { "address": "127.0.0.1", "port_value": 19001 } } }, "stats_config": { "stats_tags": [ { "tag_name": "consul.destination.custom_hash", "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.service_subset", "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.service", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.namespace", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.datacenter", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.routing_type", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.trust_domain", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)" }, { "tag_name": "consul.destination.target", "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.destination.full_target", "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)" }, { "tag_name": "consul.upstream.service", "regex": "^(?:tcp|http)\\.upstream\\.(([^.]+)(?:\\.[^.]+)?\\.[^.]+\\.)" }, { "tag_name": "consul.upstream.datacenter", "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)" }, { "tag_name": "consul.upstream.namespace", "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.([^.]+))?\\.[^.]+\\.)" }, { "tag_name": "consul.custom_hash", "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.service_subset", "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.service", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.namespace", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.datacenter", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.routing_type", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.trust_domain", "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)" }, { "tag_name": "consul.target", "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)" }, { "tag_name": "consul.full_target", "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)" }, { "tag_name": "local_cluster", "fixed_value": "client" }, { "tag_name": "consul.source.service", "fixed_value": "client" }, { "tag_name": "consul.source.namespace", "fixed_value": "default" }, { "tag_name": "consul.source.datacenter", "fixed_value": "dc1" } ], "use_all_default_tags": true }, "layered_runtime": { "layers": [ { "name": "static_layer", "static_layer": { "envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1": true, "envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name": true, "envoy.deprecated_features:envoy.api.v2.Cluster.tls_context": true } } ] } }, "last_updated": "2021-06-17T13:44:44.786Z" }, { "@type": "type.googleapis.com/envoy.admin.v3.ClustersConfigDump", "version_info": "00000003", "static_clusters": [ { "cluster": { "@type": "type.googleapis.com/envoy.api.v2.Cluster", "name": "local_agent", "type": "STATIC", "connect_timeout": "1s", "hosts": [ { "pipe": { "path": "alloc/tmp/consul_grpc.sock" } } ], "http2_protocol_options": {} }, "last_updated": "2021-06-17T13:44:44.814Z" } ], "dynamic_active_clusters": [ { "version_info": "00000001", "cluster": { "@type": "type.googleapis.com/envoy.api.v2.Cluster", "name": "local_app", "type": "STATIC", "connect_timeout": "5s", "load_assignment": { "cluster_name": "local_app", "endpoints": [ { "lb_endpoints": [ { "endpoint": { "address": { "socket_address": { "address": "127.0.0.1", "port_value": 0 } } } } ] } ] } }, "last_updated": "2021-06-17T13:44:44.843Z" } ] }, { "@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump", "version_info": "00000003", "dynamic_listeners": [ { "name": "public_listener:0.0.0.0:23131", "active_state": { "version_info": "00000001", "listener": { "@type": "type.googleapis.com/envoy.api.v2.Listener", "name": "public_listener:0.0.0.0:23131", "address": { "socket_address": { "address": "0.0.0.0", "port_value": 23131 } }, "filter_chains": [ { "tls_context": { "common_tls_context": { "tls_params": {}, "tls_certificates": [ { "certificate_chain": { "inline_string": "-----BEGIN CERTIFICATE-----\nMIICRDCCAeqgAwIBAgIBCDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktZXRk\nczJzZXUuY29uc3VsLmNhLjZlZGQ4YjIyLmNvbnN1bDAeFw0yMTA2MTcxMzQzNDRa\nFw0yMTA2MjAxMzQzNDRaMC0xKzApBgNVBAMTImNsaWVudC5zdmMuZGVmYXVsdC42\nZWRkOGIyMi5jb25zdWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARY6HS3/2EG\nRC6HdHoX7bNwXNTt732sHCh5zvJhaDLarm90JIJUzhOE8CVI3Y0La89BwJAFWo2v\nsrwbA55lQ9iTo4H2MIHzMA4GA1UdDwEB/wQEAwIDuDAdBgNVHSUEFjAUBggrBgEF\nBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHQ4EIgQgyWnTAJsqz//+\nNpneBCHzJEsyh0pcqBcvvDXqYEX41xAwKwYDVR0jBCQwIoAgYgiKwcOlnGeU7I2w\n/419CQn2tkjOOplcaP327xIraiQwXAYDVR0RBFUwU4ZRc3BpZmZlOi8vNmVkZDhi\nMjItYzJlZS0wNjhmLTJlYzgtMDYwOWM3NTVlOGRhLmNvbnN1bC9ucy9kZWZhdWx0\nL2RjL2RjMS9zdmMvY2xpZW50MAoGCCqGSM49BAMCA0gAMEUCIQDoFT/TYNzsjgn4\ng1KK+Qd/2KCjd7Q2JcZEtOqkQuSYPQIgBCqVg+XoXGIp1WfjiOzxKAVI/3FdMBxU\naMTpAPgJbEY=\n-----END CERTIFICATE-----\n" }, "private_key": { "inline_string": "[redacted]" } } ], "validation_context": { "trusted_ca": { "inline_string": "-----BEGIN CERTIFICATE-----\nMIICEDCCAbWgAwIBAgIBBzAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktZXRk\nczJzZXUuY29uc3VsLmNhLjZlZGQ4YjIyLmNvbnN1bDAeFw0yMTA2MTcxMzM1MjRa\nFw0zMTA2MTcxMzM1MjRaMDExLzAtBgNVBAMTJnByaS1ldGRzMnNldS5jb25zdWwu\nY2EuNmVkZDhiMjIuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEJdM/\n6VmktlNI81/vMRFNyvR9XKOTzmoCrFBU4DGbxmE+pIhvX0WkpOfKEpcNHdm52fCd\nfoN5bMlzkt0JLoEj5qOBvTCBujAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw\nAwEB/zApBgNVHQ4EIgQgYgiKwcOlnGeU7I2w/419CQn2tkjOOplcaP327xIraiQw\nKwYDVR0jBCQwIoAgYgiKwcOlnGeU7I2w/419CQn2tkjOOplcaP327xIraiQwPwYD\nVR0RBDgwNoY0c3BpZmZlOi8vNmVkZDhiMjItYzJlZS0wNjhmLTJlYzgtMDYwOWM3\nNTVlOGRhLmNvbnN1bDAKBggqhkjOPQQDAgNJADBGAiEAmxo8lpl5M0/hMXslbAX9\nTDeRUEXuPF+noYA9d1RcdWYCIQDyBFnVcofowba6bKOw3oGZqaC3tU0qXjRQK12J\ncle2OQ==\n-----END CERTIFICATE-----\n" } } }, "require_client_certificate": true }, "filters": [ { "name": "envoy.filters.network.rbac", "config": { "stat_prefix": "connect_authz", "rules": { "action": "DENY" } } }, { "name": "envoy.tcp_proxy", "config": { "stat_prefix": "public_listener", "cluster": "local_app" } } ] } ] }, "last_updated": "2021-06-17T13:44:44.883Z" } }, { "name": "destination:127.0.0.1:8999", "draining_state": { "version_info": "00000001", "listener": { "@type": "type.googleapis.com/envoy.api.v2.Listener", "name": "destination:127.0.0.1:8999", "address": { "socket_address": { "address": "127.0.0.1", "port_value": 8999 } }, "filter_chains": [ { "filters": [ { "name": "envoy.tcp_proxy", "config": { "stat_prefix": "upstream.destination.default.dc1", "cluster": "destination.default.dc1.internal.6edd8b22-c2ee-068f-2ec8-0609c755e8da.consul" } } ] } ] }, "last_updated": "2021-06-17T13:44:44.883Z" } } ] }, { "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump" } ] } ```At this point I don't know what I'm looking at, but it seems like the envoy config is still holding on to my
destination
destination? LMK if I'm just holding it wrong.