search

hashicorp / consul-k8s

First-class support for Consul Service Mesh on Kubernetes
https://www.consul.io/docs/k8s
Mozilla Public License 2.0
670 stars 324 forks source link

connect service upstream with prepared query not registering #224

Closed cchatfield closed 4 years ago

cchatfield commented 4 years ago

Envoy: envoyproxy/envoy:v1.10.0 Consul: consul:1.6.2 Consul k8s: "hashicorp/consul-k8s:0.9.5"

I am using the following annotation to define upstreams:

"consul.hashicorp.com/connect-service-upstreams": "prepared_query:active_vault:8200, vault:8201, prepared_query:6687bd19-5654-76be-d764-5bcc67fe8fb5:8202"

The prepared query is registered with consul and returns this when executed:

{
    "Service": "vault",
    "Nodes": [
        {
            "Node": {
                "ID": "c6e224c5-669e-762a-b778-e764f34510d3",
                "Node": "ip-10-163-43-96.us-west-2.compute.internal",
                "Address": "10.163.43.22",
                "Datacenter": "dc1",
                "TaggedAddresses": {
                    "lan": "10.163.43.22",
                    "wan": "10.163.43.22"
                },
                "Meta": {
                    "consul-network-segment": "",
                    "pod-name": "hashicorp-consul-jwst9"
                },
                "CreateIndex": 3585749,
                "ModifyIndex": 3585759
            },
            "Service": {
                "ID": "vault:10.163.43.101:8200",
                "Service": "vault",
                "Tags": [
                    "active"
                ],
                "Address": "10.163.43.101",
                "Meta": null,
                "Port": 8200,
                "Weights": {
                    "Passing": 1,
                    "Warning": 1
                },
                "EnableTagOverride": false,
                "Proxy": {
                    "MeshGateway": {},
                    "Expose": {}
                },
                "Connect": {},
                "CreateIndex": 3731021,
                "ModifyIndex": 3731451
            },
            "Checks": [
                {
                    "Node": "ip-10-163-43-96.us-west-2.compute.internal",
                    "CheckID": "serfHealth",
                    "Name": "Serf Health Status",
                    "Status": "passing",
                    "Notes": "",
                    "Output": "Agent alive and reachable",
                    "ServiceID": "",
                    "ServiceName": "",
                    "ServiceTags": null,
                    "Type": "",
                    "Definition": {},
                    "CreateIndex": 3585749,
                    "ModifyIndex": 3585749
                },
                {
                    "Node": "ip-10-163-43-96.us-west-2.compute.internal",
                    "CheckID": "vault:10.163.43.101:8200:vault-sealed-check",
                    "Name": "Vault Sealed Status",
                    "Status": "passing",
                    "Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
                    "Output": "Vault Unsealed",
                    "ServiceID": "vault:10.163.43.101:8200",
                    "ServiceName": "vault",
                    "ServiceTags": [
                        "active"
                    ],
                    "Type": "ttl",
                    "Definition": {},
                    "CreateIndex": 3731022,
                    "ModifyIndex": 3731452
                }
            ]
        }
    ],
    "DNS": {
        "TTL": ""
    },
    "Datacenter": "dc1",
    "Failovers": 0,
    "Index": 0,
    "LastContact": 0,
    "KnownLeader": true,
    "ConsistencyLevel": "leader"
}

The "vault" upstream is available on 8201, but neither prepared query is registered.

Envoy Sidecar:

[2020-03-05 01:03:43.697][000001][info][main] [source/server/server.cc:206] initializing epoch 0 (hot restart version=10.200.16384.323.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=5931112)
[2020-03-05 01:03:43.697][000001][info][main] [source/server/server.cc:208] statically linked extensions:
[2020-03-05 01:03:43.697][000001][info][main] [source/server/server.cc:210]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:213]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:216]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:219]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:221]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:223]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:226]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:229]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2020-03-05 01:03:43.703][000001][info][main] [source/server/server.cc:271] admin address: 127.0.0.1:19000
[2020-03-05 01:03:43.704][000001][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2020-03-05 01:03:43.704][000001][info][config] [source/server/configuration_impl.cc:56] loading 2 cluster(s)
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:67] loading 0 listener(s)
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:92] loading tracing configuration
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:101]   loading tracing driver: envoy.zipkin
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:112] loading stats sink configuration
[2020-03-05 01:03:43.711][000001][info][main] [source/server/server.cc:463] starting main dispatch loop
[2020-03-05 01:03:43.712][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:132] cm init: initializing cds
[2020-03-05 01:03:43.716][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster local_app during init
[2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster vault.default.dc1.internal.d16dc2ba-6d47-f242-bc30-77d364f5596e.consul during init
[2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:112] cm init: initializing secondary clusters
[2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:136] cm init: all clusters initialized
[2020-03-05 01:03:43.721][000001][info][main] [source/server/server.cc:435] all clusters initialized. initializing init manager
[2020-03-05 01:03:43.724][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'public_listener:10.163.42.200:20000'
[2020-03-05 01:03:43.725][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'vault:127.0.0.1:8201'
[2020-03-05 01:03:43.725][000001][info][config] [source/server/listener_manager_impl.cc:961] all dependencies initialized. starting workers
[2020-03-05 01:18:43.723][000001][info][main] [source/server/drain_manager_impl.cc:63] shutting down parent after drain
adilyse commented 4 years ago

Hi @cchatfield,

Thanks for filing this! I've looked into it, and there's definitely a bug here. We're not properly stripping spaces off of the prepared_query tag while processing, so your second prepared query is not getting recognized and properly parsed.

For your case, this manifests as only two upstreams being registered:

I've put together a fix in #233 , but you could fix it for now by taking out the spaces between the upstreams in your annotation:

"consul.hashicorp.com/connect-service-upstreams": "prepared_query:active_vault:8200,vault:8201,prepared_query:6687bd19-5654-76be-d764-5bcc67fe8fb5:8202"