Service Mash is not balancing the workload evenly across multiple instances in Nomad cluster.

Hi!

I have 10 gateway instances (Ingress Gateway) and 10 "Proxy A" (Envoy) instances to which traffic from the gateway goes. Than I have 2 "Proxy B" instances (Envoy also) that accept traffic from "Proxy A".

Load balancing from "Proxy A" to "Proxy B" working incorrect.

From the network load on the host, I can see that traffic is only going to one instance "Proxy B".

1st instance of Proxy B

2nd instance of Proxy B

If I redeployed nomad job "Proxy A" a workload balancing is going correctly.

1st instance of Proxy B

2nd instance of Proxy B

But when I redeployed the nomad job "Proxy B" the balancing "breaks down" again.

I tried writing the congfiguration to Consul for the "Proxy A"

Kind           = "service-resolver"
Name           = "Proxy A"
LoadBalancer = {
  Policy = "round_robin"
}

and for "Proxy B"

Kind           = "service-resolver"
Name           = "Proxy B"
LoadBalancer = {
  Policy = "round_robin"
}

Envoy configuration file of "Proxy A"

``` node: cluster: test id: proxy_a admin: access_log: - name: admin_access typed_config: "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog path: {{ env "NOMAD_ALLOC_DIR" }}/logs/admin_access.log address: socket_address: address: 0.0.0.0 port_value: 19901 dynamic_resources: ads_config: api_type: DELTA_GRPC transport_api_version: V3 grpc_services: - envoy_grpc: cluster_name: xds_grpc static_resources: listeners: - name: proxy_a address: socket_address: address: 0.0.0.0 port_value: 8162 filter_chains: - filters: - name: envoy.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: proxy_a http2_protocol_options: allow_connect: true upgrade_configs: - upgrade_type: websocket rds: route_config_name: proxy_a config_source: resource_api_version: V3 ads: {} http_filters: - name: envoy.filters.http.grpc_web typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext require_client_certificate: true common_tls_context: validation_context: trusted_ca: filename: {{ env "NOMAD_SECRETS_DIR" }}/gateway/tls/gateway-ca.pem tls_certificates: - certificate_chain: filename: {{ env "NOMAD_SECRETS_DIR" }}/gateway/tls/downstream.pem private_key: filename: {{ env "NOMAD_SECRETS_DIR" }}/gateway/tls/downstream-key.pem alpn_protocols: [ "h2,http/1.1" ] clusters: - name: proxy_b connect_timeout: 0.25s type: STATIC lb_policy: ROUND_ROBIN load_assignment: cluster_name: proxy_b endpoints: - lb_endpoints: - endpoint: address: socket_address: address: {{ env "NOMAD_UPSTREAM_IP_proxy_b" }} port_value: {{ env "NOMAD_UPSTREAM_PORT_proxy_b" }} circuit_breakers: thresholds: - priority: "DEFAULT" max_connections: 1000000000 max_pending_requests: 1000000000 max_requests: 1000000000 max_retries: 1000000000 retry_budget: budget_percent: value: 25.0 track_remaining: true - priority: "HIGH" max_connections: 1000000000 max_pending_requests: 1000000000 max_requests: 1000000000 max_retries: 1000000000 retry_budget: budget_percent: value: 25.0 track_remaining: true typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions explicit_http_config: http2_protocol_options: {} - name: xds_grpc load_assignment: cluster_name: xds_grpc endpoints: - lb_endpoints: - endpoint: address: socket_address: address: {{ env "NOMAD_UPSTREAM_IP_[[ .my.xds_upstream ]]" }} port_value: {{ env "NOMAD_UPSTREAM_PORT_[[ .my.xds_upstream ]]" }} circuit_breakers: thresholds: - priority: "DEFAULT" max_connections: 1000000000 max_pending_requests: 1000000000 max_requests: 1000000000 max_retries: 1000000000 retry_budget: budget_percent: value: 25.0 track_remaining: true - priority: "HIGH" max_connections: 1000000000 max_pending_requests: 1000000000 max_requests: 1000000000 max_retries: 1000000000 retry_budget: budget_percent: value: 25.0 track_remaining: true typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions upstream_http_protocol_options: auto_sni: true common_http_protocol_options: idle_timeout: 1s explicit_http_config: http2_protocol_options: max_concurrent_streams: 100 ```

Server nomad version

Nomad v1.8.3
BuildDate 2024-08-13T07:37:30Z
Revision 63b636e5cbaca312cf6ea63e040f445f05f00478

Server consul version

Consul v1.19.1
Revision 9f62fb41
Build Date 2024-07-11T14:47:27Z

Client nomad version

Nomad v1.5.6
BuildDate 2023-05-19T18:26:13Z
Revision 8af70885c02ab921dedbdf6bc406a1e886866f80

Client consul version

Consul v1.14.7
Revision d97acc0a
Build Date 2023-05-16T01:36:41Z

hashicorp / consul