Service-Defaults not applied to prepared query upstreams

hamishforbes commented 2 years ago

Overview of the Issue

I'm trying to use service-defaults config entries to increase the Envoy sidecar Max Connection limits as documented here: https://www.consul.io/docs/connect/config-entries/service-defaults

My upstreams are defined using prepared queries (because I need to lookup specific instances of services via tag).

UpstreamConfig defaults appear to be ignored for any upstreams defined using prepared queries rather than the plain service lookup.

Reproduction Steps

Kubernetes manifests and terraform example I used to create config entries and prepared query https://gist.github.com/hamishforbes/39aeecc466a3b314a7d628f07f005948

tl;dr: 2 upstreams on the client pod, one for server and one for prepared_query:test_server
The prepared query just looks for server services tagged test.
The service-defaults entry for the client service sets UpstreamConfig.Defaults.Limits.MaxConnections = 2000

The upstream cluster for the prepared query does not pickup the configuration but the plain cluster does

> k exec -it (k get pod -lname=client -o jsonpath="{.items[0].metadata.name}") -c client -- curl localhost:19000/clusters | grep 'consul::default_priority::max_connections'
c0657655~server.default.ap-southeast-2.internal.c0aedc2c-bb39-3212-2849-82e71c5640db.consul::default_priority::max_connections::2000
test.server.default.ap-southeast-2.query.c0aedc2c-bb39-3212-2849-82e71c5640db.consul::default_priority::max_connections::1024

The prepared query returns the same service as the plain lookup (snipped some output for brevity)

> curl -s -H"X-Consul-Token: $CONSUL_HTTP_TOKEN" "$CONSUL_HTTP_ADDR/v1/query/test_server/execute?dc=ap-southeast-2" | jq '.Nodes[0].Service'
{
  "ID": "server-647f5d445c-g6mgc-server",
  "Service": "server",
  "Tags": [
    "test"
  ],
 ...
  "Meta": {
    "k8s-namespace": "test",
    "k8s-service-name": "server",
    "managed-by": "consul-k8s-endpoints-controller",
    "pod-name": "server-647f5d445c-g6mgc"
  },
  ...
}

> curl -s -H"X-Consul-Token: $CONSUL_HTTP_TOKEN" "$CONSUL_HTTP_ADDR/v1/catalog/service/server?dc=ap-southeast-2" | jq
[
  {
    ...
    "ServiceID": "server-647f5d445c-g6mgc-server",
    "ServiceName": "server",
    "ServiceTags": [
      "test"
    ],
    ...
    "ServiceMeta": {
      "k8s-namespace": "test",
      "k8s-service-name": "server",
      "managed-by": "consul-k8s-endpoints-controller",
      "pod-name": "server-647f5d445c-g6mgc"
    },
    ...
  }
]

Consul info for both Client and Server

Consul 1.10.4 client and server

consul-k8s integration installed via Helm - v0.37.0

Amier3 commented 2 years ago

Hey @hamishforbes

Apologies for the extremely late reply. I wanted to follow up and clarify that this wasn't neccesarily a bug -- we currently don't expose escape hatch overrides in service-defaults. We're tracking this feature through https://github.com/hashicorp/consul/issues/12040 and in the interm we're looking at increasing the base max connections to alleviate some of the burden on operators to upkeep large amounts of replicas as a workaround.

I'll keep this open till this is resolved.

hamishforbes commented 2 years ago

Hi, this isn't about escape hatch overrides though.

This is about the standard Upstream Config service-defaults not being applied when services are configured via prepared queries

hashicorp / consul