Connect: Allow for a prepared query to be used in more than one envoy upstream

[cbarbara-okta]

Feature Description

We are looking to use the same prepared query in multiple envoy upstream configurations, the difference in those upstreams will be the port and the envoy_cluster_json of those upstreams for both timeouts & circuit breaking reasons.

Snippet of our current hcl:

...
        upstreams = [
            {
                destination_type = "prepared_query"
                destination_name = "our-service-query"
                local_bind_port = 9988
                config {
                  envoy_cluster_json = <<EOL
                    {
                      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                      "name": "prepared_query:our-service-query",
                      "type": "EDS",
...

Right now I believe there is a not well documented requirement that the envoy cluster name needs to match the destination_type + ":" + destination_name of the upstream. Envoy requires unique cluster names in its config, so we cannot create two different upstreams in the consul hcl that use the same prepared query.

We are unable to create another upstream using the same prepared query. If we attempt to create a second upstream with a envoy cluster name that does not equal prepared_query:our-service-query it will not be populated with any servers. And if you attempt to create a second upstream with an envoy cluster name of prepared_query:our-service-query again envoy will not startup correctly with the warning:

[2019-11-13 06:00:51.787][000052][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: duplicate cluster prepared_query:our-service-query found

Use Case(s)

Example config we would like support:

service {
  name = "our-proxy"
  port = 9999
  connect {
    sidecar_service {
      port = 22000
      proxy {
        config {
        }
        upstreams = [
            {
                destination_type = "prepared_query"
                destination_name = "our-service-query"
                local_bind_port = 9988
                config {
                  envoy_cluster_json = <<EOL
                    {
                      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                      "name": "upstream_1",
                      "type": "EDS",
                      "eds_cluster_config": {
                        "eds_config": {
                          "ads": {}
                        }
                      },
                      "connect_timeout": "1s",
                      "circuit_breakers": {
                        "thresholds": [
                          {
                            "priority": "DEFAULT",
                            "max_connections": 200,
                            "max_pending_requests": 2,
                            "max_requests": 200,
                            "max_retries": 5
                          }
                        ]
                      }
                    }
                  EOL
                }
            },
            {
                destination_type = "prepared_query"
                destination_name = "our-service-query"
                local_bind_port = 9989
                config {
                  envoy_cluster_json = <<EOL
                    {
                      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                      "name": "upstream_2",
                      "type": "EDS",
                      "eds_cluster_config": {
                        "eds_config": {
                          "ads": {}
                        }
                      },
                      "connect_timeout": "3s",
                      "circuit_breakers": {
                        "thresholds": [
                          {
                            "priority": "DEFAULT",
                            "max_connections": 20,
                            "max_pending_requests": 2,
                            "max_requests": 20,
                            "max_retries": 5
                          }
                        ]
                      }
                    }
                  EOL
                }
            }
        ]
      }
    }
  }
}

If the envoy cluster names did not need to match the prepared queries then we could reuse the queries for multiple upstreams. But now it appears that we have to create and maintain multiple copies of the queries that only differ by name but have the same content.

[stale[bot]]

Hey there, We wanted to check in on this request since it has been inactive for at least 60 days. If you think this is still an important issue in the latest version of Consul or its documentation please reply with a comment here which will cause it to stay open for investigation. If there is still no activity on this issue for 30 more days, we will go ahead and close it.

Feel free to check out the community forum as well! Thank you!

[cbarbara-okta]

not stale, still looking for a reply from the consul connect team here.