search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.12k stars 4.82k forks source link

Envoy: connection refused (via Consul) #23026

Closed locinus closed 2 years ago

locinus commented 2 years ago

Description: An Envoy Docker sidecar is deployed with each of our workload services (Nomad cluster, Docker deployments, private network of our own between our servers), and suppose to connect with a Consul service mesh.

Our service is correctly deployed and accessible on the corresponding (Nomad) network. But Consul has trouble validating the Envoy sidecar healthcheck:

Connect Sidecar Listening
ServiceName my-service-sidecar-proxy
CheckID service:_nomad-task-xxxxxx-xxxx-xxxxxx-xxxxx-xxxxxxxxxx-group-my-service-my-service-interface-sidecar-proxy:2
Type tcp
Output dial tcp 10.0.0.72:30241: connect: connection refused

From our servers, the workload service is accessible via our private network, or Nomad network, but the Envoy is not accessible.

# Workload service
server-node-0:~$ wget 10.0.0.63:28947/health
--2022-09-07 15:38:28--  http://10.0.0.63:28947/health
Connecting to 10.0.0.63:28947... connected.
HTTP request sent, awaiting response... 200 OK
# Envoy
server-node-0:~$ wget 10.0.0.63:30241
--2022-09-07 15:38:57--  http://10.0.0.63:30241/
Connecting to 10.0.0.63:30241... failed: Connection refused.

From the server where the services are deployed, or within their Docker container, a netstat indicates only the workload service port is open, not the Envoy one.

client-node-1:~$ netstat -a | grep 28947
tcp        0      0 client-node-1:49236 client-node-1:28947 TIME_WAIT
client-node-1:~$ netstat -a | grep 30241
client-node-1:~$

Yet, Envoy process seems to run properly in its sidecar container. Its admin port (19001) is open, and responses well:

root@3d7c60d8b94a:/# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
envoy        1  0.1  1.3 2256608 53300 ?       Ssl  15:53   0:05 envoy -c /secrets/envoy_bootstrap.json -l debug --concurrency 1 --disable-hot-restart
root@3d7c60d8b94a:/# curl -X POST 127.0.0.2:19001/ready
LIVE

What could cause Envoy to not properly listen to its port, so it can properly act as a proxy?

With debug logs on, we noticed the following messages, not knowing if it could be related:

[2022-09-07 17:03:13.173][1][debug][config] [./source/common/config/grpc_stream.h:61] Establishing new gRPC bidi stream for rpc DeltaAggregatedResources(stream .envoy.service.discovery.v3.DeltaDiscoveryRequest) returns (stream .envoy.service.discovery.v3.DeltaDiscoveryResponse);

[2022-09-07 17:03:13.173][1][debug][router] [source/common/router/router.cc:471] [C0][S3688902394289443788] cluster 'local_agent' match for URL '/envoy.service.discovery.v3.AggregatedDiscoveryService/DeltaAggregatedResources'
[2022-09-07 17:03:13.173][1][debug][router] [source/common/router/router.cc:675] [C0][S3688902394289443788] router decoding headers:
':method', 'POST'
':path', '/envoy.service.discovery.v3.AggregatedDiscoveryService/DeltaAggregatedResources'
':authority', 'local_agent'
':scheme', 'http'
'te', 'trailers'
'content-type', 'application/grpc'
'x-consul-token', 'bf88ad2f-7d22-716c-6e5f-8c404f866d6e'
'x-envoy-internal', 'true'
'x-forwarded-for', '172.26.64.4'

[2022-09-07 17:03:13.173][1][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2022-09-07 17:03:13.173][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:290] trying to create new connection
[2022-09-07 17:03:13.173][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2022-09-07 17:03:13.173][1][debug][http2] [source/common/http/http2/codec_impl.cc:1788] [C294] updating connection-level initial window size to 268435456
[2022-09-07 17:03:13.173][1][debug][connection] [./source/common/network/connection_impl.h:89] [C294] current connecting state: true
[2022-09-07 17:03:13.173][1][debug][client] [source/common/http/codec_client.cc:57] [C294] connecting
[2022-09-07 17:03:13.173][1][debug][connection] [source/common/network/connection_impl.cc:924] [C294] connecting to alloc/tmp/consul_grpc.sock
[2022-09-07 17:03:13.174][1][debug][connection] [source/common/network/connection_impl.cc:683] [C294] connected
[2022-09-07 17:03:13.174][1][debug][client] [source/common/http/codec_client.cc:89] [C294] connected
[2022-09-07 17:03:13.174][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:327] [C294] attaching to next stream
[2022-09-07 17:03:13.174][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:181] [C294] creating stream
[2022-09-07 17:03:13.174][1][debug][router] [source/common/router/upstream_request.cc:424] [C0][S3688902394289443788] pool ready
[2022-09-07 17:03:13.179][1][debug][connection] [source/common/network/connection_impl.cc:651] [C294] remote close
[2022-09-07 17:03:13.179][1][debug][connection] [source/common/network/connection_impl.cc:250] [C294] closing socket: 0
[2022-09-07 17:03:13.179][1][debug][client] [source/common/http/codec_client.cc:108] [C294] disconnect. resetting 1 pending requests
[2022-09-07 17:03:13.179][1][debug][client] [source/common/http/codec_client.cc:140] [C294] request reset
[2022-09-07 17:03:13.179][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:214] [C294] destroying stream: 0 remaining
[2022-09-07 17:03:13.179][1][debug][router] [source/common/router/router.cc:1198] [C0][S3688902394289443788] upstream reset: reset reason: connection termination, transport failure reason:
[2022-09-07 17:03:13.179][1][debug][http] [source/common/http/async_client_impl.cc:101] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'

[2022-09-07 17:03:13.179][1][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 4161s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2022-09-07 17:03:13.179][1][debug][config] [source/common/config/grpc_subscription_impl.cc:113] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
[2022-09-07 17:03:13.179][1][debug][config] [source/common/config/grpc_subscription_impl.cc:113] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
[2022-09-07 17:03:13.179][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:483] [C294] client disconnected, failure reason:
[2022-09-07 17:03:13.179][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:453] invoking idle callbacks - is_draining_for_deletion_=false

Admin and Stats Output:

Include the admin output for the following endpoints:

/stats cluster_manager.cds.version_text: "" control_plane.identifier: "" listener_manager.lds.version_text: "" cluster.local_agent.assignment_stale: 0 cluster.local_agent.assignment_timeout_received: 0 cluster.local_agent.bind_errors: 0 cluster.local_agent.circuit_breakers.default.cx_open: 0 cluster.local_agent.circuit_breakers.default.cx_pool_open: 0 cluster.local_agent.circuit_breakers.default.rq_open: 0 cluster.local_agent.circuit_breakers.default.rq_pending_open: 0 cluster.local_agent.circuit_breakers.default.rq_retry_open: 0 cluster.local_agent.circuit_breakers.high.cx_open: 0 cluster.local_agent.circuit_breakers.high.cx_pool_open: 0 cluster.local_agent.circuit_breakers.high.rq_open: 0 cluster.local_agent.circuit_breakers.high.rq_pending_open: 0 cluster.local_agent.circuit_breakers.high.rq_retry_open: 0 cluster.local_agent.default.total_match_count: 1 cluster.local_agent.http2.deferred_stream_close: 0 cluster.local_agent.http2.dropped_headers_with_underscores: 0 cluster.local_agent.http2.header_overflow: 0 cluster.local_agent.http2.headers_cb_no_stream: 0 cluster.local_agent.http2.inbound_empty_frames_flood: 0 cluster.local_agent.http2.inbound_priority_frames_flood: 0 cluster.local_agent.http2.inbound_window_update_frames_flood: 0 cluster.local_agent.http2.keepalive_timeout: 0 cluster.local_agent.http2.metadata_empty_frames: 0 cluster.local_agent.http2.outbound_control_flood: 0 cluster.local_agent.http2.outbound_flood: 0 cluster.local_agent.http2.pending_send_bytes: 0 cluster.local_agent.http2.requests_rejected_with_underscores_in_headers: 0 cluster.local_agent.http2.rx_messaging_error: 0 cluster.local_agent.http2.rx_reset: 0 cluster.local_agent.http2.stream_refused_errors: 0 cluster.local_agent.http2.streams_active: 0 cluster.local_agent.http2.trailers: 0 cluster.local_agent.http2.tx_flush_timeout: 0 cluster.local_agent.http2.tx_reset: 0 cluster.local_agent.internal.upstream_rq_503: 512 cluster.local_agent.internal.upstream_rq_5xx: 512 cluster.local_agent.internal.upstream_rq_completed: 512 cluster.local_agent.lb_healthy_panic: 0 cluster.local_agent.lb_local_cluster_not_ok: 0 cluster.local_agent.lb_recalculate_zone_structures: 0 cluster.local_agent.lb_subsets_active: 0 cluster.local_agent.lb_subsets_created: 0 cluster.local_agent.lb_subsets_fallback: 0 cluster.local_agent.lb_subsets_fallback_panic: 0 cluster.local_agent.lb_subsets_removed: 0 cluster.local_agent.lb_subsets_selected: 0 cluster.local_agent.lb_zone_cluster_too_small: 0 cluster.local_agent.lb_zone_no_capacity_left: 0 cluster.local_agent.lb_zone_number_differs: 0 cluster.local_agent.lb_zone_routing_all_directly: 0 cluster.local_agent.lb_zone_routing_cross_zone: 0 cluster.local_agent.lb_zone_routing_sampled: 0 cluster.local_agent.max_host_weight: 0 cluster.local_agent.membership_change: 1 cluster.local_agent.membership_degraded: 0 cluster.local_agent.membership_excluded: 0 cluster.local_agent.membership_healthy: 1 cluster.local_agent.membership_total: 1 cluster.local_agent.original_dst_host_invalid: 0 cluster.local_agent.retry_or_shadow_abandoned: 0 cluster.local_agent.update_attempt: 0 cluster.local_agent.update_empty: 0 cluster.local_agent.update_failure: 0 cluster.local_agent.update_no_rebuild: 0 cluster.local_agent.update_success: 0 cluster.local_agent.upstream_cx_active: 0 cluster.local_agent.upstream_cx_close_notify: 0 cluster.local_agent.upstream_cx_connect_attempts_exceeded: 0 cluster.local_agent.upstream_cx_connect_fail: 0 cluster.local_agent.upstream_cx_connect_timeout: 0 cluster.local_agent.upstream_cx_connect_with_0_rtt: 0 cluster.local_agent.upstream_cx_destroy: 512 cluster.local_agent.upstream_cx_destroy_local: 0 cluster.local_agent.upstream_cx_destroy_local_with_active_rq: 0 cluster.local_agent.upstream_cx_destroy_remote: 512 cluster.local_agent.upstream_cx_destroy_remote_with_active_rq: 512 cluster.local_agent.upstream_cx_destroy_with_active_rq: 512 cluster.local_agent.upstream_cx_http1_total: 0 cluster.local_agent.upstream_cx_http2_total: 512 cluster.local_agent.upstream_cx_http3_total: 0 cluster.local_agent.upstream_cx_idle_timeout: 0 cluster.local_agent.upstream_cx_max_duration_reached: 0 cluster.local_agent.upstream_cx_max_requests: 0 cluster.local_agent.upstream_cx_none_healthy: 0 cluster.local_agent.upstream_cx_overflow: 0 cluster.local_agent.upstream_cx_pool_overflow: 0 cluster.local_agent.upstream_cx_protocol_error: 0 cluster.local_agent.upstream_cx_rx_bytes_buffered: 0 cluster.local_agent.upstream_cx_rx_bytes_total: 0 cluster.local_agent.upstream_cx_total: 512 cluster.local_agent.upstream_cx_tx_bytes_buffered: 0 cluster.local_agent.upstream_cx_tx_bytes_total: 28183495 cluster.local_agent.upstream_flow_control_backed_up_total: 0 cluster.local_agent.upstream_flow_control_drained_total: 0 cluster.local_agent.upstream_flow_control_paused_reading_total: 0 cluster.local_agent.upstream_flow_control_resumed_reading_total: 0 cluster.local_agent.upstream_http3_broken: 0 cluster.local_agent.upstream_internal_redirect_failed_total: 0 cluster.local_agent.upstream_internal_redirect_succeeded_total: 0 cluster.local_agent.upstream_rq_0rtt: 0 cluster.local_agent.upstream_rq_503: 512 cluster.local_agent.upstream_rq_5xx: 512 cluster.local_agent.upstream_rq_active: 0 cluster.local_agent.upstream_rq_cancelled: 0 cluster.local_agent.upstream_rq_completed: 512 cluster.local_agent.upstream_rq_maintenance_mode: 0 cluster.local_agent.upstream_rq_max_duration_reached: 0 cluster.local_agent.upstream_rq_pending_active: 0 cluster.local_agent.upstream_rq_pending_failure_eject: 512 cluster.local_agent.upstream_rq_pending_overflow: 0 cluster.local_agent.upstream_rq_pending_total: 512 cluster.local_agent.upstream_rq_per_try_idle_timeout: 0 cluster.local_agent.upstream_rq_per_try_timeout: 0 cluster.local_agent.upstream_rq_retry: 0 cluster.local_agent.upstream_rq_retry_backoff_exponential: 0 cluster.local_agent.upstream_rq_retry_backoff_ratelimited: 0 cluster.local_agent.upstream_rq_retry_limit_exceeded: 0 cluster.local_agent.upstream_rq_retry_overflow: 0 cluster.local_agent.upstream_rq_retry_success: 0 cluster.local_agent.upstream_rq_rx_reset: 0 cluster.local_agent.upstream_rq_timeout: 0 cluster.local_agent.upstream_rq_total: 512 cluster.local_agent.upstream_rq_tx_reset: 0 cluster.local_agent.version: 0 cluster_manager.active_clusters: 1 cluster_manager.cds.init_fetch_timeout: 1 cluster_manager.cds.update_attempt: 514 cluster_manager.cds.update_failure: 512 cluster_manager.cds.update_rejected: 0 cluster_manager.cds.update_success: 0 cluster_manager.cds.update_time: 0 cluster_manager.cds.version: 0 cluster_manager.cluster_added: 1 cluster_manager.cluster_modified: 0 cluster_manager.cluster_removed: 0 cluster_manager.cluster_updated: 0 cluster_manager.cluster_updated_via_merge: 0 cluster_manager.update_merge_cancelled: 0 cluster_manager.update_out_of_merge_window: 0 cluster_manager.warming_clusters: 0 control_plane.connected_state: 0 control_plane.pending_requests: 0 control_plane.rate_limit_enforced: 0 envoy.overload_actions.reset_high_memory_stream.count: 0 filesystem.flushed_by_timer: 761 filesystem.reopen_failed: 0 filesystem.write_buffered: 22 filesystem.write_completed: 20 filesystem.write_failed: 0 filesystem.write_total_buffered: 0 http.admin.downstream_cx_active: 1 http.admin.downstream_cx_delayed_close_timeout: 0 http.admin.downstream_cx_destroy: 22 http.admin.downstream_cx_destroy_active_rq: 0 http.admin.downstream_cx_destroy_local: 0 http.admin.downstream_cx_destroy_local_active_rq: 0 http.admin.downstream_cx_destroy_remote: 22 http.admin.downstream_cx_destroy_remote_active_rq: 0 http.admin.downstream_cx_drain_close: 0 http.admin.downstream_cx_http1_active: 1 http.admin.downstream_cx_http1_total: 23 http.admin.downstream_cx_http2_active: 0 http.admin.downstream_cx_http2_total: 0 http.admin.downstream_cx_http3_active: 0 http.admin.downstream_cx_http3_total: 0 http.admin.downstream_cx_idle_timeout: 0 http.admin.downstream_cx_max_duration_reached: 0 http.admin.downstream_cx_max_requests_reached: 0 http.admin.downstream_cx_overload_disable_keepalive: 0 http.admin.downstream_cx_protocol_error: 0 http.admin.downstream_cx_rx_bytes_buffered: 85 http.admin.downstream_cx_rx_bytes_total: 2021 http.admin.downstream_cx_ssl_active: 0 http.admin.downstream_cx_ssl_total: 0 http.admin.downstream_cx_total: 23 http.admin.downstream_cx_tx_bytes_buffered: 0 http.admin.downstream_cx_tx_bytes_total: 214786 http.admin.downstream_cx_upgrades_active: 0 http.admin.downstream_cx_upgrades_total: 0 http.admin.downstream_flow_control_paused_reading_total: 0 http.admin.downstream_flow_control_resumed_reading_total: 0 http.admin.downstream_rq_1xx: 0 http.admin.downstream_rq_2xx: 21 http.admin.downstream_rq_3xx: 0 http.admin.downstream_rq_4xx: 2 http.admin.downstream_rq_5xx: 0 http.admin.downstream_rq_active: 1 http.admin.downstream_rq_completed: 23 http.admin.downstream_rq_failed_path_normalization: 0 http.admin.downstream_rq_header_timeout: 0 http.admin.downstream_rq_http1_total: 23 http.admin.downstream_rq_http2_total: 0 http.admin.downstream_rq_http3_total: 0 http.admin.downstream_rq_idle_timeout: 0 http.admin.downstream_rq_max_duration_reached: 0 http.admin.downstream_rq_non_relative_path: 0 http.admin.downstream_rq_overload_close: 0 http.admin.downstream_rq_redirected_with_normalized_path: 0 http.admin.downstream_rq_rejected_via_ip_detection: 0 http.admin.downstream_rq_response_before_rq_complete: 0 http.admin.downstream_rq_rx_reset: 0 http.admin.downstream_rq_timeout: 0 http.admin.downstream_rq_too_large: 0 http.admin.downstream_rq_total: 23 http.admin.downstream_rq_tx_reset: 0 http.admin.downstream_rq_ws_on_non_ws_route: 0 http.admin.rs_too_large: 0 http.async-client.no_cluster: 0 http.async-client.no_route: 0 http.async-client.passthrough_internal_redirect_bad_location: 0 http.async-client.passthrough_internal_redirect_no_route: 0 http.async-client.passthrough_internal_redirect_predicate: 0 http.async-client.passthrough_internal_redirect_too_many_redirects: 0 http.async-client.passthrough_internal_redirect_unsafe_scheme: 0 http.async-client.rq_direct_response: 0 http.async-client.rq_redirect: 0 http.async-client.rq_reset_after_downstream_response_started: 0 http.async-client.rq_total: 512 http1.dropped_headers_with_underscores: 0 http1.metadata_not_supported_error: 0 http1.requests_rejected_with_underscores_in_headers: 0 http1.response_flood: 0 listener.admin.downstream_cx_active: 1 listener.admin.downstream_cx_destroy: 22 listener.admin.downstream_cx_overflow: 0 listener.admin.downstream_cx_overload_reject: 0 listener.admin.downstream_cx_total: 23 listener.admin.downstream_cx_transport_socket_connect_timeout: 0 listener.admin.downstream_global_cx_overflow: 0 listener.admin.downstream_listener_filter_error: 0 listener.admin.downstream_listener_filter_remote_close: 0 listener.admin.downstream_pre_cx_active: 0 listener.admin.downstream_pre_cx_timeout: 0 listener.admin.http.admin.downstream_rq_1xx: 0 listener.admin.http.admin.downstream_rq_2xx: 21 listener.admin.http.admin.downstream_rq_3xx: 0 listener.admin.http.admin.downstream_rq_4xx: 2 listener.admin.http.admin.downstream_rq_5xx: 0 listener.admin.http.admin.downstream_rq_completed: 23 listener.admin.main_thread.downstream_cx_active: 1 listener.admin.main_thread.downstream_cx_total: 23 listener.admin.no_filter_chain_match: 0 listener_manager.lds.init_fetch_timeout: 1 listener_manager.lds.update_attempt: 507 listener_manager.lds.update_failure: 505 listener_manager.lds.update_rejected: 0 listener_manager.lds.update_success: 0 listener_manager.lds.update_time: 0 listener_manager.lds.version: 0 listener_manager.listener_added: 0 listener_manager.listener_create_failure: 0 listener_manager.listener_create_success: 0 listener_manager.listener_in_place_updated: 0 listener_manager.listener_modified: 0 listener_manager.listener_removed: 0 listener_manager.listener_stopped: 0 listener_manager.total_filter_chains_draining: 0 listener_manager.total_listeners_active: 0 listener_manager.total_listeners_draining: 0 listener_manager.total_listeners_warming: 0 listener_manager.workers_started: 1 main_thread.watchdog_mega_miss: 0 main_thread.watchdog_miss: 0 runtime.admin_overrides_active: 0 runtime.deprecated_feature_seen_since_process_start: 0 runtime.deprecated_feature_use: 0 runtime.load_error: 0 runtime.load_success: 1 runtime.num_keys: 1 runtime.num_layers: 1 runtime.override_dir_exists: 0 runtime.override_dir_not_exists: 1 server.compilation_settings.fips_mode: 0 server.concurrency: 1 server.days_until_first_cert_expiring: 4294967295 server.debug_assertion_failures: 0 server.dropped_stat_flushes: 0 server.dynamic_unknown_fields: 0 server.envoy_bug_failures: 0 server.hot_restart_epoch: 0 server.live: 1 server.main_thread.watchdog_mega_miss: 0 server.main_thread.watchdog_miss: 0 server.memory_allocated: 6873608 server.memory_heap_size: 12582912 server.memory_physical_size: 17951658 server.parent_connections: 0 server.seconds_until_first_ocsp_response_expiring: 0 server.state: 0 server.static_unknown_fields: 0 server.stats_recent_lookups: 4815 server.total_connections: 0 server.uptime: 7612 server.version: 13519303 server.wip_protos: 0 server.worker_0.watchdog_mega_miss: 0 server.worker_0.watchdog_miss: 0 workers.watchdog_mega_miss: 0 workers.watchdog_miss: 0 cluster.local_agent.upstream_cx_connect_ms: P0(nan,0) P25(nan,0) P50(nan,0) P75(nan,0) P90(nan,0) P95(nan,1.012142857142857) P99(nan,1.0852857142857142) P99.5(nan,1.0944285714285713) P99.9(nan,16.488) P100(nan,17) cluster.local_agent.upstream_cx_length_ms: P0(nan,1) P25(nan,4.003157894736842) P50(nan,5.023684210526316) P75(nan,6.013636363636364) P90(nan,7.003478260869565) P95(nan,8.024285714285714) P99(nan,10.969999999999999) P99.5(nan,14.439999999999998) P99.9(nan,24.488) P100(nan,25) cluster_manager.cds.update_duration: No recorded values http.admin.downstream_cx_length_ms: P0(nan,3) P25(nan,3.055) P50(nan,4.02) P75(nan,6.025) P90(nan,11.8) P95(nan,29.9) P99(nan,37.78) P99.5(nan,37.89) P99.9(nan,37.978) P100(nan,38) http.admin.downstream_rq_time: P0(nan,0) P25(nan,1.09) P50(nan,2.0416666666666665) P75(nan,2.0875) P90(nan,4.08) P95(nan,5.09) P99(nan,7.078) P99.5(nan,7.089) P99.9(nan,7.0978) P100(nan,7.1) listener.admin.downstream_cx_length_ms: P0(nan,3) P25(nan,3.055) P50(nan,4.02) P75(nan,6.025) P90(nan,11.8) P95(nan,29.9) P99(nan,37.78) P99.5(nan,37.89) P99.9(nan,37.978) P100(nan,38) listener_manager.lds.update_duration: No recorded values server.initialization_time_ms: P0(nan,30000) P25(nan,30250) P50(nan,30500) P75(nan,30750) P90(nan,30900) P95(nan,30950) P99(nan,30990) P99.5(nan,30995) P99.9(nan,30999) P100(nan,31000)
/clusters local_agent::observability_name::local_agent local_agent::default_priority::max_connections::1024 local_agent::default_priority::max_pending_requests::1024 local_agent::default_priority::max_requests::1024 local_agent::default_priority::max_retries::3 local_agent::high_priority::max_connections::1024 local_agent::high_priority::max_pending_requests::1024 local_agent::high_priority::max_requests::1024 local_agent::high_priority::max_retries::3 local_agent::added_via_api::false local_agent::alloc/tmp/consul_grpc.sock::cx_active::0 local_agent::alloc/tmp/consul_grpc.sock::cx_connect_fail::0 local_agent::alloc/tmp/consul_grpc.sock::cx_total::530 local_agent::alloc/tmp/consul_grpc.sock::rq_active::0 local_agent::alloc/tmp/consul_grpc.sock::rq_error::530 local_agent::alloc/tmp/consul_grpc.sock::rq_success::0 local_agent::alloc/tmp/consul_grpc.sock::rq_timeout::0 local_agent::alloc/tmp/consul_grpc.sock::rq_total::530 local_agent::alloc/tmp/consul_grpc.sock::hostname:: local_agent::alloc/tmp/consul_grpc.sock::health_flags::healthy local_agent::alloc/tmp/consul_grpc.sock::weight::1 local_agent::alloc/tmp/consul_grpc.sock::region:: local_agent::alloc/tmp/consul_grpc.sock::zone:: local_agent::alloc/tmp/consul_grpc.sock::sub_zone:: local_agent::alloc/tmp/consul_grpc.sock::canary::false local_agent::alloc/tmp/consul_grpc.sock::priority::0 local_agent::alloc/tmp/consul_grpc.sock::success_rate::-1 local_agent::alloc/tmp/consul_grpc.sock::local_origin_success_rate::-1
/routes invalid path. admin commands are: (...)
/server_info (truncated: too big for GitHub post) { "version": "ce49c7f65668a22b80d1e83c35d170741bb8d46a/1.23.0/Clean/RELEASE/BoringSSL", "state": "LIVE", "hot_restart_version": "disabled", "command_line_options": { "base_id": "0", "use_dynamic_base_id": false, "base_id_path": "", "concurrency": 1, "config_path": "/secrets/envoy_bootstrap.json", "config_yaml": "", "allow_unknown_static_fields": false, "reject_unknown_dynamic_fields": false, "ignore_unknown_dynamic_fields": false, "admin_address_path": "", "local_address_ip_version": "v4", "log_level": "debug", "component_log_level": "", "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] [%g:%#] %v", "log_format_escaped": false, "log_path": "", "service_cluster": "", "service_node": "", "service_zone": "", "drain_strategy": "Gradual", "mode": "Serve", "disable_hot_restart": true, "enable_mutex_tracing": false, "restart_epoch": 0, "cpuset_threads": false, "disabled_extensions": [], "enable_fine_grain_logging": false, "socket_path": "@envoy_domain_socket", "socket_mode": 0, "enable_core_dump": false, "stats_tag": [], "file_flush_interval": "10s", "drain_time": "600s", "parent_shutdown_time": "900s" }, "node": { "id": "_nomad-task-12e0179e-4c41-1cc3-70af-a6e730429d9c-group-xml-sandre-xml-sandre-interface-sidecar-proxy", "cluster": "xml-sandre", "metadata": { "partition": "default", "namespace": "default" }, "dynamic_parameters": {}, "user_agent_name": "envoy", "user_agent_build_version": { "version": { "major_number": 1, "minor_number": 23, "patch": 0 }, "metadata": { "build.type": "RELEASE", "revision.status": "Clean", "revision.sha": "ce49c7f65668a22b80d1e83c35d170741bb8d46a", "ssl.version": "BoringSSL" } }, "extensions": [ { "name": "envoy.filters.connection_pools.tcp.generic", "category": "envoy.upstreams", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.upstreams.tcp.generic.v3.GenericConnectionPoolProto" ] }, { "name": "envoy.filters.listener.http_inspector", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.listener.http_inspector.v3.HttpInspector" ] }, { "name": "envoy.filters.listener.original_dst", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.listener.original_dst.v3.OriginalDst" ] }, { "name": "envoy.filters.listener.original_src", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.listener.original_src.v3.OriginalSrc" ] }, { "name": "envoy.filters.listener.proxy_protocol", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol" ] }, { "name": "envoy.filters.listener.tls_inspector", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector" ] }, { "name": "envoy.listener.http_inspector", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.listener.original_dst", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.listener.original_src", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.listener.proxy_protocol", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.listener.tls_inspector", "category": "envoy.filters.listener", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.key_value.file_based", "category": "envoy.common.key_value", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.key_value.file_based.v3.FileBasedKeyValueStoreConfig" ] }, { "name": "envoy.cluster.eds", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.cluster.logical_dns", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.cluster.original_dst", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.cluster.static", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.cluster.strict_dns", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.clusters.aggregate", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.clusters.dynamic_forward_proxy", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.clusters.redis", "category": "envoy.clusters", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.matching.matchers.consistent_hashing", "category": "envoy.matching.input_matchers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.input_matchers.consistent_hashing.v3.ConsistentHashing" ] }, { "name": "envoy.matching.matchers.ip", "category": "envoy.matching.input_matchers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.input_matchers.ip.v3.Ip" ] }, { "name": "auto", "category": "envoy.thrift_proxy.transports", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "framed", "category": "envoy.thrift_proxy.transports", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "header", "category": "envoy.thrift_proxy.transports", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "unframed", "category": "envoy.thrift_proxy.transports", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.http.stateful_session.cookie", "category": "envoy.http.stateful_session", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.http.stateful_session.cookie.v3.CookieBasedSessionState" ] }, { "name": "envoy.filters.thrift.header_to_metadata", "category": "envoy.thrift_proxy.filters", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.thrift_proxy.filters.header_to_metadata.v3.HeaderToMetadata" ] }, { "name": "envoy.filters.thrift.rate_limit", "category": "envoy.thrift_proxy.filters", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.thrift_proxy.filters.ratelimit.v3.RateLimit" ] }, { "name": "envoy.filters.thrift.router", "category": "envoy.thrift_proxy.filters", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.thrift_proxy.router.v3.Router" ] }, { "name": "envoy.formatter.metadata", "category": "envoy.formatter", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.formatter.metadata.v3.Metadata" ] }, { "name": "envoy.formatter.req_without_query", "category": "envoy.formatter", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.formatter.req_without_query.v3.ReqWithoutQuery" ] }, { "name": "envoy.extensions.http.cache.simple", "category": "envoy.http.cache", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.cache.simple_http_cache.v3.SimpleHttpCacheConfig" ] }, { "name": "envoy.rbac.matchers.upstream_ip_port", "category": "envoy.rbac.matchers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.rbac.matchers.upstream_ip_port.v3.UpstreamIpPortMatcher" ] }, { "name": "envoy.http.header_validators.envoy_default", "category": "envoy.http.header_validators", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.http.header_validators.envoy_default.v3.HeaderValidatorConfig" ] }, { "name": "envoy.ip", "category": "envoy.resolvers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.filters.dubbo.router", "category": "envoy.dubbo_proxy.filters", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.dubbo_proxy.router.v3.Router" ] }, { "name": "envoy.quic.crypto_stream.server.quiche", "category": "envoy.quic.server.crypto_stream", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.quic.crypto_stream.v3.CryptoServerStreamConfig" ] }, { "name": "envoy.udp_packet_writer.default", "category": "envoy.udp_packet_writer", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.udp_packet_writer.v3.UdpDefaultWriterFactory" ] }, { "name": "envoy.udp_packet_writer.gso", "category": "envoy.udp_packet_writer", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.udp_packet_writer.v3.UdpGsoBatchWriterFactory" ] }, { "name": "composite-action", "category": "envoy.matching.action", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.http.composite.v3.ExecuteFilterAction" ] }, { "name": "skip", "category": "envoy.matching.action", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.common.matcher.action.v3.SkipFilter" ] }, { "name": "envoy.matching.inputs.application_protocol", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.ApplicationProtocolInput" ] }, { "name": "envoy.matching.inputs.destination_ip", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.DestinationIPInput" ] }, { "name": "envoy.matching.inputs.destination_port", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.DestinationPortInput" ] }, { "name": "envoy.matching.inputs.direct_source_ip", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.DirectSourceIPInput" ] }, { "name": "envoy.matching.inputs.dns_san", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.ssl.v3.DnsSanInput" ] }, { "name": "envoy.matching.inputs.server_name", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.ServerNameInput" ] }, { "name": "envoy.matching.inputs.source_ip", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.SourceIPInput" ] }, { "name": "envoy.matching.inputs.source_port", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.SourcePortInput" ] }, { "name": "envoy.matching.inputs.source_type", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.SourceTypeInput" ] }, { "name": "envoy.matching.inputs.subject", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.ssl.v3.SubjectInput" ] }, { "name": "envoy.matching.inputs.transport_protocol", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.TransportProtocolInput" ] }, { "name": "envoy.matching.inputs.uri_san", "category": "envoy.matching.network.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.ssl.v3.UriSanInput" ] }, { "name": "envoy.access_loggers.file", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.file.v3.FileAccessLog" ] }, { "name": "envoy.access_loggers.http_grpc", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig" ] }, { "name": "envoy.access_loggers.open_telemetry", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.open_telemetry.v3.OpenTelemetryAccessLogConfig" ] }, { "name": "envoy.access_loggers.stderr", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.stream.v3.StderrAccessLog" ] }, { "name": "envoy.access_loggers.stdout", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.stream.v3.StdoutAccessLog" ] }, { "name": "envoy.access_loggers.tcp_grpc", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.grpc.v3.TcpGrpcAccessLogConfig" ] }, { "name": "envoy.access_loggers.wasm", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.access_loggers.wasm.v3.WasmAccessLog" ] }, { "name": "envoy.file_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.http_grpc_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.open_telemetry_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.stderr_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.stdout_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.tcp_grpc_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.wasm_access_log", "category": "envoy.access_loggers", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.watchdog.abort_action", "category": "envoy.guarddog_actions", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.watchdog.v3.AbortActionConfig" ] }, "TRUNCATED": "TOO BIG FOR GITHUB POST", { "name": "envoy.watchdog.profile_action", "category": "envoy.guarddog_actions", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.watchdog.profile_action.v3.ProfileActionConfig" ] }, { "name": "envoy.matching.common_inputs.environment_variable", "category": "envoy.matching.common_inputs", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.environment_variable.v3.Config" ] }, { "name": "envoy.rate_limit_descriptors.expr", "category": "envoy.rate_limit_descriptors", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.rate_limit_descriptors.expr.v3.Descriptor" ] }, { "name": "auto", "category": "envoy.thrift_proxy.protocols", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "binary", "category": "envoy.thrift_proxy.protocols", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "binary/non-strict", "category": "envoy.thrift_proxy.protocols", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "compact", "category": "envoy.thrift_proxy.protocols", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "twitter", "category": "envoy.thrift_proxy.protocols", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "default", "category": "network.connection.client", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy_internal", "category": "network.connection.client", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.matching.inputs.destination_ip", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.DestinationIPInput" ] }, { "name": "envoy.matching.inputs.destination_port", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.DestinationPortInput" ] }, { "name": "envoy.matching.inputs.direct_source_ip", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.DirectSourceIPInput" ] }, { "name": "envoy.matching.inputs.dns_san", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.ssl.v3.DnsSanInput" ] }, { "name": "envoy.matching.inputs.request_headers", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.type.matcher.v3.HttpRequestHeaderMatchInput" ] }, { "name": "envoy.matching.inputs.request_trailers", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.type.matcher.v3.HttpRequestTrailerMatchInput" ] }, { "name": "envoy.matching.inputs.response_headers", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.type.matcher.v3.HttpResponseHeaderMatchInput" ] }, { "name": "envoy.matching.inputs.response_trailers", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.type.matcher.v3.HttpResponseTrailerMatchInput" ] }, { "name": "envoy.matching.inputs.server_name", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.ServerNameInput" ] }, { "name": "envoy.matching.inputs.source_ip", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.SourceIPInput" ] }, { "name": "envoy.matching.inputs.source_port", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.SourcePortInput" ] }, { "name": "envoy.matching.inputs.source_type", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.network.v3.SourceTypeInput" ] }, { "name": "envoy.matching.inputs.subject", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.ssl.v3.SubjectInput" ] }, { "name": "envoy.matching.inputs.uri_san", "category": "envoy.matching.http.input", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.matching.common_inputs.ssl.v3.UriSanInput" ] }, { "name": "envoy.echo", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.ext_authz", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.filters.network.connection_limit", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.connection_limit.v3.ConnectionLimit" ] }, { "name": "envoy.filters.network.direct_response", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.direct_response.v3.Config" ] }, { "name": "envoy.filters.network.dubbo_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.dubbo_proxy.v3.DubboProxy" ] }, { "name": "envoy.filters.network.echo", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.echo.v3.Echo" ] }, { "name": "envoy.filters.network.ext_authz", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.ext_authz.v3.ExtAuthz" ] }, { "name": "envoy.filters.network.http_connection_manager", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager" ] }, { "name": "envoy.filters.network.local_ratelimit", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.local_ratelimit.v3.LocalRateLimit" ] }, { "name": "envoy.filters.network.mongo_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.mongo_proxy.v3.MongoProxy" ] }, { "name": "envoy.filters.network.ratelimit", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.ratelimit.v3.RateLimit" ] }, { "name": "envoy.filters.network.rbac", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.rbac.v3.RBAC" ] }, { "name": "envoy.filters.network.redis_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.redis_proxy.v3.RedisProxy" ] }, { "name": "envoy.filters.network.sni_cluster", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.sni_cluster.v3.SniCluster" ] }, { "name": "envoy.filters.network.sni_dynamic_forward_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.sni_dynamic_forward_proxy.v3.FilterConfig" ] }, { "name": "envoy.filters.network.tcp_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy" ] }, { "name": "envoy.filters.network.thrift_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.thrift_proxy.v3.ThriftProxy" ] }, { "name": "envoy.filters.network.wasm", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.wasm.v3.Wasm" ] }, { "name": "envoy.filters.network.zookeeper_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [ "envoy.extensions.filters.network.zookeeper_proxy.v3.ZooKeeperProxy" ] }, { "name": "envoy.http_connection_manager", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.mongo_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.ratelimit", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.redis_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] }, { "name": "envoy.tcp_proxy", "category": "envoy.filters.network", "type_descriptor": "", "disabled": false, "type_urls": [] } ], "client_features": [], "listening_addresses": [] }, "uptime_current_epoch": "709s", "uptime_all_epochs": "709s" }

Config:

envoy-bootstrap.json { "admin": { "access_log_path": "/dev/null", "address": { "socket_address": { "address": "127.0.0.2", "port_value": 19001 } } }, "node": { "cluster": "xml-sandre", "id": "_nomad-task-12e0179e-4c41-1cc3-70af-a6e730429d9c-group-xml-sandre-xml-sandre-interface-sidecar-proxy", "metadata": { "namespace": "default", "partition": "default" } }, "layered_runtime": { "layers": [ { "name": "base", "static_layer": { "re2.max_program_size.error_level": 1048576 } } ] }, "static_resources": { "clusters": [ { "name": "local_agent", "ignore_health_on_host_removal": false, "connect_timeout": "1s", "type": "STATIC", "http2_protocol_options": {}, "loadAssignment": { "clusterName": "local_agent", "endpoints": [ { "lbEndpoints": [ { "endpoint": { "address": { "pipe": { "path": "alloc/tmp/consul_grpc.sock" } } } } ] } ] } } ] }, "stats_config": { "stats_tags": [ { "tag_name": "nomad.alloc_id", "fixed_value": "12e0179e-4c41-1cc3-70af-a6e730429d9c" }, { "regex": "^cluster\\.(?:passthrough~)?((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.custom_hash" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.service_subset" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.service" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.namespace" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:([^.]+)\\.)?[^.]+\\.internal[^.]*\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.partition" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.datacenter" }, { "regex": "^cluster\\.([^.]+\\.(?:[^.]+\\.)?([^.]+)\\.external\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.peer" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.routing_type" }, { "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)", "tag_name": "consul.destination.trust_domain" }, { "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.destination.target" }, { "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)", "tag_name": "consul.destination.full_target" }, { "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.(([^.]+)(?:\\.[^.]+)?(?:\\.[^.]+)?\\.[^.]+\\.)", "tag_name": "consul.upstream.service" }, { "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.[^.]+)?\\.([^.]+)\\.)", "tag_name": "consul.upstream.datacenter" }, { "regex": "^(?:tcp|http)\\.upstream_peered\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)", "tag_name": "consul.upstream.peer" }, { "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.([^.]+(?:\\.([^.]+))?(?:\\.[^.]+)?\\.[^.]+\\.)", "tag_name": "consul.upstream.namespace" }, { "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.([^.]+))?\\.[^.]+\\.)", "tag_name": "consul.upstream.partition" }, { "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.custom_hash" }, { "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.service_subset" }, { "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.service" }, { "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.namespace" }, { "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)", "tag_name": "consul.datacenter" }, { "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)", "tag_name": "consul.routing_type" }, { "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)", "tag_name": "consul.trust_domain" }, { "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)", "tag_name": "consul.target" }, { "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)", "tag_name": "consul.full_target" }, { "tag_name": "local_cluster", "fixed_value": "xml-sandre" }, { "tag_name": "consul.source.service", "fixed_value": "xml-sandre" }, { "tag_name": "consul.source.namespace", "fixed_value": "default" }, { "tag_name": "consul.source.partition", "fixed_value": "default" }, { "tag_name": "consul.source.datacenter", "fixed_value": "nomeau" } ], "use_all_default_tags": true }, "dynamic_resources": { "lds_config": { "ads": {}, "resource_api_version": "V3" }, "cds_config": { "ads": {}, "resource_api_version": "V3" }, "ads_config": { "api_type": "DELTA_GRPC", "transport_api_version": "V3", "grpc_services": { "initial_metadata": [ { "key": "x-consul-token", "value": "bf88ad2f-7d22-716c-6e5f-8c404f866d6e" } ], "envoy_grpc": { "cluster_name": "local_agent" } } } } }

Logs: See above.

Versions:

Envoy version: 1.23.0

Consul version: 1.13.1

locinus commented 2 years ago

After discovering this message within Consul logs:

consul grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50356": tls: first record does not look like a TLS handshake

the root cause has been identified, as Consul 1.13.1 ships in with an incompatibility against Envoy, if using certain parameters (namely auto-encrypt or auto-config).

The solution is then:

We rollbacked to Consul 1.12 (and now Envoy fails at communicating with its attached service, but this is surely another problem).