stale[bot]
commented
4 years ago Hey there, We wanted to check in on this request since it has been inactive for at least 60 days. If you think this is still an important issue in the latest version of Consul or its documentation please reply with a comment here which will cause it to stay open for investigation. If there is still no activity on this issue for 30 more days, we will go ahead and close it.
Feel free to check out the community forum as well! Thank you!
valarauca
mkeeler
Overview of the Issue
Attempting to centralize a configuration for
consul agentto cut down on clutter, and I ran into some difficulty.Namely a
consul connect proxywhich fails to get a SPIFFEE cert, and never opens its local ports, is reported as alive & healthy.Reproduction Steps
Create a RAFT clusters of 5 servers. I used the same configuration for all 3. It is as follows:
Server configuration
```json { "addresses": { "http": "127.0.0.1", "https": "0.0.0.0" }, "bootstrap": false, "bootstrap_expect": 3, "ca_file": "/opt/consul/certs/ca_cert.pem", "cert_file": "/opt/consul/certs/local_cert.pem", "data_dir": "/opt/consul/data", "discard_check_output": null, "discovery_max_stale": null, "enable_script_checks": false, "enable_local_script_checks": false, "encrypt": "hLRmojjbMVq4hpIHrBRkCw==", "encrypt_verify_incoming": true, "encrypt_verify_outgoing": true, "key_file": "/opt/consul/certs/local_key.pem", "leave_on_terminate": true, "log_level": "TRACE", "log_file": "/opt/consul/consul.log", "log_rotate_bytes": 1048576, "pid_file": "/opt/consul/apigee-consul.pid", "ports": { "http": 8500, "https": 8503, "grpc": -1 }, "rejoin_after_leave": true, "retry_interval": "5s", "retry_join": [ "10.126.15.192", "10.126.15.193", "10.126.15.194", "10.126.15.195", "10.126.15.196" ], "retry_max": 200, "server": true, "services": [ { "kind": "connect-proxy", "id": "zookeeper-2181-10-126-15-192", "name": "zookeeper-2181-10-126-15-192", "port": 10000, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-2181-10-126-15-192", "destination_service_id": "zookeeper-2181-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 2181 } }, { "kind": "connect-proxy", "id": "zookeeper-2888-10-126-15-192", "name": "zookeeper-2888-10-126-15-192", "port": 10001, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-2888-10-126-15-192", "destination_service_id": "zookeeper-2888-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 2888 } }, { "kind": "connect-proxy", "id": "zookeeper-3888-10-126-15-192", "name": "zookeeper-3888-10-126-15-192", "port": 10002, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-3888-10-126-15-192", "destination_service_id": "zookeeper-3888-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 3888 } }, { "kind": "connect-proxy", "id": "cassandra-9042-10-126-15-192", "name": "cassandra-9042-10-126-15-192", "port": 10003, "enable_tag_override": false, "proxy": { "destination_service_name": "cassandra-9042-10-126-15-192", "destination_service_id": "cassandra-9042-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 9042 } }, { "kind": "connect-proxy", "id": "cassandra-9160-10-126-15-192", "name": "cassandra-9160-10-126-15-192", "port": 10004, "enable_tag_override": false, "proxy": { "destination_service_name": "cassandra-9160-10-126-15-192", "destination_service_id": "cassandra-9160-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 9160 } }, { "kind": "connect-proxy", "id": "message-router-4527-10-126-15-192", "name": "message-router-4527-10-126-15-192", "port": 10005, "enable_tag_override": false, "proxy": { "destination_service_name": "message-router-4527-10-126-15-192", "destination_service_id": "message-router-4527-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 4527 } }, { "kind": "connect-proxy", "id": "message-processor-4528-10-126-15-192", "name": "message-processor-4528-10-126-15-192", "port": 10006, "enable_tag_override": false, "proxy": { "destination_service_name": "message-processor-4528-10-126-15-192", "destination_service_id": "message-processor-4528-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 4528 } }, { "kind": "connect-proxy", "id": "message-processor-8082-10-126-15-192", "name": "message-processor-8082-10-126-15-192", "port": 10007, "enable_tag_override": false, "proxy": { "destination_service_name": "message-processor-8082-10-126-15-192", "destination_service_id": "message-processor-8082-10-126-15-192", "local_service_address": "127.0.0.1", "local_service_port": 8082 } }, { "kind": "connect-proxy", "id": "egress-10-126-15-192", "name": "egress-10-126-15-192", "port": 10032, "enable_tag_override": false, "proxy": { "destination_service_name": "egress-10-126-15-192", "destination_service_id": "egress-10-126-15-192", "upstreams": [ { "destination_name": "qpid-4529-10-126-15-193", "local_bind_port": 10008 }, { "destination_name": "qpid-5672-10-126-15-193", "local_bind_port": 10009 }, { "destination_name": "qpid-4529-10-126-15-194", "local_bind_port": 10013 }, { "destination_name": "qpid-5672-10-126-15-194", "local_bind_port": 10014 }, { "destination_name": "zookeeper-2181-10-126-15-195", "local_bind_port": 10018 }, { "destination_name": "zookeeper-2888-10-126-15-195", "local_bind_port": 10019 }, { "destination_name": "zookeeper-3888-10-126-15-195", "local_bind_port": 10020 }, { "destination_name": "cassandra-9042-10-126-15-195", "local_bind_port": 10021 }, { "destination_name": "cassandra-9160-10-126-15-195", "local_bind_port": 10022 }, { "destination_name": "zookeeper-2181-10-126-15-196", "local_bind_port": 10024 }, { "destination_name": "zookeeper-2888-10-126-15-196", "local_bind_port": 10025 }, { "destination_name": "zookeeper-3888-10-126-15-196", "local_bind_port": 10026 }, { "destination_name": "cassandra-9042-10-126-15-196", "local_bind_port": 10027 }, { "destination_name": "cassandra-9160-10-126-15-196", "local_bind_port": 10028 }, { "destination_name": "message-processor-4528-10-126-15-196", "local_bind_port": 10030 }, { "destination_name": "message-processor-8082-10-126-15-196", "local_bind_port": 10031 } ] } }, { "kind": "connect-proxy", "id": "qpid-4529-10-126-15-193", "name": "qpid-4529-10-126-15-193", "port": 10008, "enable_tag_override": false, "proxy": { "destination_service_name": "qpid-4529-10-126-15-193", "destination_service_id": "qpid-4529-10-126-15-193", "local_service_address": "127.0.0.1", "local_service_port": 4529 } }, { "kind": "connect-proxy", "id": "qpid-5672-10-126-15-193", "name": "qpid-5672-10-126-15-193", "port": 10009, "enable_tag_override": false, "proxy": { "destination_service_name": "qpid-5672-10-126-15-193", "destination_service_id": "qpid-5672-10-126-15-193", "local_service_address": "127.0.0.1", "local_service_port": 5672 } }, { "kind": "connect-proxy", "id": "postgresql-4530-10-126-15-193", "name": "postgresql-4530-10-126-15-193", "port": 10010, "enable_tag_override": false, "proxy": { "destination_service_name": "postgresql-4530-10-126-15-193", "destination_service_id": "postgresql-4530-10-126-15-193", "local_service_address": "127.0.0.1", "local_service_port": 4530 } }, { "kind": "connect-proxy", "id": "postgresql-5432-10-126-15-193", "name": "postgresql-5432-10-126-15-193", "port": 10011, "enable_tag_override": false, "proxy": { "destination_service_name": "postgresql-5432-10-126-15-193", "destination_service_id": "postgresql-5432-10-126-15-193", "local_service_address": "127.0.0.1", "local_service_port": 5432 } }, { "kind": "connect-proxy", "id": "postgresql-8084-10-126-15-193", "name": "postgresql-8084-10-126-15-193", "port": 10012, "enable_tag_override": false, "proxy": { "destination_service_name": "postgresql-8084-10-126-15-193", "destination_service_id": "postgresql-8084-10-126-15-193", "local_service_address": "127.0.0.1", "local_service_port": 8084 } }, { "kind": "connect-proxy", "id": "egress-10-126-15-193", "name": "egress-10-126-15-193", "port": 10033, "enable_tag_override": false, "proxy": { "destination_service_name": "egress-10-126-15-193", "destination_service_id": "egress-10-126-15-193", "upstreams": [ { "destination_name": "zookeeper-2181-10-126-15-192", "local_bind_port": 10000 }, { "destination_name": "zookeeper-2888-10-126-15-192", "local_bind_port": 10001 }, { "destination_name": "zookeeper-3888-10-126-15-192", "local_bind_port": 10002 }, { "destination_name": "cassandra-9042-10-126-15-192", "local_bind_port": 10003 }, { "destination_name": "cassandra-9160-10-126-15-192", "local_bind_port": 10004 }, { "destination_name": "qpid-4529-10-126-15-194", "local_bind_port": 10013 }, { "destination_name": "qpid-5672-10-126-15-194", "local_bind_port": 10014 }, { "destination_name": "postgresql-4530-10-126-15-194", "local_bind_port": 10015 }, { "destination_name": "postgresql-5432-10-126-15-194", "local_bind_port": 10016 }, { "destination_name": "postgresql-8084-10-126-15-194", "local_bind_port": 10017 }, { "destination_name": "zookeeper-2181-10-126-15-195", "local_bind_port": 10018 }, { "destination_name": "zookeeper-2888-10-126-15-195", "local_bind_port": 10019 }, { "destination_name": "zookeeper-3888-10-126-15-195", "local_bind_port": 10020 }, { "destination_name": "cassandra-9042-10-126-15-195", "local_bind_port": 10021 }, { "destination_name": "cassandra-9160-10-126-15-195", "local_bind_port": 10022 }, { "destination_name": "zookeeper-2181-10-126-15-196", "local_bind_port": 10024 }, { "destination_name": "zookeeper-2888-10-126-15-196", "local_bind_port": 10025 }, { "destination_name": "zookeeper-3888-10-126-15-196", "local_bind_port": 10026 }, { "destination_name": "cassandra-9042-10-126-15-196", "local_bind_port": 10027 }, { "destination_name": "cassandra-9160-10-126-15-196", "local_bind_port": 10028 } ] } }, { "kind": "connect-proxy", "id": "qpid-4529-10-126-15-194", "name": "qpid-4529-10-126-15-194", "port": 10013, "enable_tag_override": false, "proxy": { "destination_service_name": "qpid-4529-10-126-15-194", "destination_service_id": "qpid-4529-10-126-15-194", "local_service_address": "127.0.0.1", "local_service_port": 4529 } }, { "kind": "connect-proxy", "id": "qpid-5672-10-126-15-194", "name": "qpid-5672-10-126-15-194", "port": 10014, "enable_tag_override": false, "proxy": { "destination_service_name": "qpid-5672-10-126-15-194", "destination_service_id": "qpid-5672-10-126-15-194", "local_service_address": "127.0.0.1", "local_service_port": 5672 } }, { "kind": "connect-proxy", "id": "postgresql-4530-10-126-15-194", "name": "postgresql-4530-10-126-15-194", "port": 10015, "enable_tag_override": false, "proxy": { "destination_service_name": "postgresql-4530-10-126-15-194", "destination_service_id": "postgresql-4530-10-126-15-194", "local_service_address": "127.0.0.1", "local_service_port": 4530 } }, { "kind": "connect-proxy", "id": "postgresql-5432-10-126-15-194", "name": "postgresql-5432-10-126-15-194", "port": 10016, "enable_tag_override": false, "proxy": { "destination_service_name": "postgresql-5432-10-126-15-194", "destination_service_id": "postgresql-5432-10-126-15-194", "local_service_address": "127.0.0.1", "local_service_port": 5432 } }, { "kind": "connect-proxy", "id": "postgresql-8084-10-126-15-194", "name": "postgresql-8084-10-126-15-194", "port": 10017, "enable_tag_override": false, "proxy": { "destination_service_name": "postgresql-8084-10-126-15-194", "destination_service_id": "postgresql-8084-10-126-15-194", "local_service_address": "127.0.0.1", "local_service_port": 8084 } }, { "kind": "connect-proxy", "id": "egress-10-126-15-194", "name": "egress-10-126-15-194", "port": 10034, "enable_tag_override": false, "proxy": { "destination_service_name": "egress-10-126-15-194", "destination_service_id": "egress-10-126-15-194", "upstreams": [ { "destination_name": "zookeeper-2181-10-126-15-192", "local_bind_port": 10000 }, { "destination_name": "zookeeper-2888-10-126-15-192", "local_bind_port": 10001 }, { "destination_name": "zookeeper-3888-10-126-15-192", "local_bind_port": 10002 }, { "destination_name": "cassandra-9042-10-126-15-192", "local_bind_port": 10003 }, { "destination_name": "cassandra-9160-10-126-15-192", "local_bind_port": 10004 }, { "destination_name": "qpid-4529-10-126-15-193", "local_bind_port": 10008 }, { "destination_name": "qpid-5672-10-126-15-193", "local_bind_port": 10009 }, { "destination_name": "postgresql-4530-10-126-15-193", "local_bind_port": 10010 }, { "destination_name": "postgresql-5432-10-126-15-193", "local_bind_port": 10011 }, { "destination_name": "postgresql-8084-10-126-15-193", "local_bind_port": 10012 }, { "destination_name": "zookeeper-2181-10-126-15-195", "local_bind_port": 10018 }, { "destination_name": "zookeeper-2888-10-126-15-195", "local_bind_port": 10019 }, { "destination_name": "zookeeper-3888-10-126-15-195", "local_bind_port": 10020 }, { "destination_name": "cassandra-9042-10-126-15-195", "local_bind_port": 10021 }, { "destination_name": "cassandra-9160-10-126-15-195", "local_bind_port": 10022 }, { "destination_name": "zookeeper-2181-10-126-15-196", "local_bind_port": 10024 }, { "destination_name": "zookeeper-2888-10-126-15-196", "local_bind_port": 10025 }, { "destination_name": "zookeeper-3888-10-126-15-196", "local_bind_port": 10026 }, { "destination_name": "cassandra-9042-10-126-15-196", "local_bind_port": 10027 }, { "destination_name": "cassandra-9160-10-126-15-196", "local_bind_port": 10028 } ] } }, { "kind": "connect-proxy", "id": "zookeeper-2181-10-126-15-195", "name": "zookeeper-2181-10-126-15-195", "port": 10018, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-2181-10-126-15-195", "destination_service_id": "zookeeper-2181-10-126-15-195", "local_service_address": "127.0.0.1", "local_service_port": 2181 } }, { "kind": "connect-proxy", "id": "zookeeper-2888-10-126-15-195", "name": "zookeeper-2888-10-126-15-195", "port": 10019, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-2888-10-126-15-195", "destination_service_id": "zookeeper-2888-10-126-15-195", "local_service_address": "127.0.0.1", "local_service_port": 2888 } }, { "kind": "connect-proxy", "id": "zookeeper-3888-10-126-15-195", "name": "zookeeper-3888-10-126-15-195", "port": 10020, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-3888-10-126-15-195", "destination_service_id": "zookeeper-3888-10-126-15-195", "local_service_address": "127.0.0.1", "local_service_port": 3888 } }, { "kind": "connect-proxy", "id": "cassandra-9042-10-126-15-195", "name": "cassandra-9042-10-126-15-195", "port": 10021, "enable_tag_override": false, "proxy": { "destination_service_name": "cassandra-9042-10-126-15-195", "destination_service_id": "cassandra-9042-10-126-15-195", "local_service_address": "127.0.0.1", "local_service_port": 9042 } }, { "kind": "connect-proxy", "id": "cassandra-9160-10-126-15-195", "name": "cassandra-9160-10-126-15-195", "port": 10022, "enable_tag_override": false, "proxy": { "destination_service_name": "cassandra-9160-10-126-15-195", "destination_service_id": "cassandra-9160-10-126-15-195", "local_service_address": "127.0.0.1", "local_service_port": 9160 } }, { "kind": "connect-proxy", "id": "ldap-10389-10-126-15-195", "name": "ldap-10389-10-126-15-195", "port": 10023, "enable_tag_override": false, "proxy": { "destination_service_name": "ldap-10389-10-126-15-195", "destination_service_id": "ldap-10389-10-126-15-195", "local_service_address": "127.0.0.1", "local_service_port": 10389 } }, { "kind": "connect-proxy", "id": "egress-10-126-15-195", "name": "egress-10-126-15-195", "port": 10035, "enable_tag_override": false, "proxy": { "destination_service_name": "egress-10-126-15-195", "destination_service_id": "egress-10-126-15-195", "upstreams": [ { "destination_name": "zookeeper-2181-10-126-15-192", "local_bind_port": 10000 }, { "destination_name": "zookeeper-2888-10-126-15-192", "local_bind_port": 10001 }, { "destination_name": "zookeeper-3888-10-126-15-192", "local_bind_port": 10002 }, { "destination_name": "cassandra-9042-10-126-15-192", "local_bind_port": 10003 }, { "destination_name": "cassandra-9160-10-126-15-192", "local_bind_port": 10004 }, { "destination_name": "message-router-4527-10-126-15-192", "local_bind_port": 10005 }, { "destination_name": "message-processor-4528-10-126-15-192", "local_bind_port": 10006 }, { "destination_name": "message-processor-8082-10-126-15-192", "local_bind_port": 10007 }, { "destination_name": "qpid-4529-10-126-15-193", "local_bind_port": 10008 }, { "destination_name": "qpid-5672-10-126-15-193", "local_bind_port": 10009 }, { "destination_name": "postgresql-4530-10-126-15-193", "local_bind_port": 10010 }, { "destination_name": "postgresql-5432-10-126-15-193", "local_bind_port": 10011 }, { "destination_name": "postgresql-8084-10-126-15-193", "local_bind_port": 10012 }, { "destination_name": "qpid-4529-10-126-15-194", "local_bind_port": 10013 }, { "destination_name": "qpid-5672-10-126-15-194", "local_bind_port": 10014 }, { "destination_name": "postgresql-4530-10-126-15-194", "local_bind_port": 10015 }, { "destination_name": "postgresql-5432-10-126-15-194", "local_bind_port": 10016 }, { "destination_name": "postgresql-8084-10-126-15-194", "local_bind_port": 10017 }, { "destination_name": "zookeeper-2181-10-126-15-196", "local_bind_port": 10024 }, { "destination_name": "zookeeper-2888-10-126-15-196", "local_bind_port": 10025 }, { "destination_name": "zookeeper-3888-10-126-15-196", "local_bind_port": 10026 }, { "destination_name": "cassandra-9042-10-126-15-196", "local_bind_port": 10027 }, { "destination_name": "cassandra-9160-10-126-15-196", "local_bind_port": 10028 }, { "destination_name": "message-router-4527-10-126-15-196", "local_bind_port": 10029 }, { "destination_name": "message-processor-4528-10-126-15-196", "local_bind_port": 10030 }, { "destination_name": "message-processor-8082-10-126-15-196", "local_bind_port": 10031 } ] } }, { "kind": "connect-proxy", "id": "zookeeper-2181-10-126-15-196", "name": "zookeeper-2181-10-126-15-196", "port": 10024, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-2181-10-126-15-196", "destination_service_id": "zookeeper-2181-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 2181 } }, { "kind": "connect-proxy", "id": "zookeeper-2888-10-126-15-196", "name": "zookeeper-2888-10-126-15-196", "port": 10025, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-2888-10-126-15-196", "destination_service_id": "zookeeper-2888-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 2888 } }, { "kind": "connect-proxy", "id": "zookeeper-3888-10-126-15-196", "name": "zookeeper-3888-10-126-15-196", "port": 10026, "enable_tag_override": false, "proxy": { "destination_service_name": "zookeeper-3888-10-126-15-196", "destination_service_id": "zookeeper-3888-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 3888 } }, { "kind": "connect-proxy", "id": "cassandra-9042-10-126-15-196", "name": "cassandra-9042-10-126-15-196", "port": 10027, "enable_tag_override": false, "proxy": { "destination_service_name": "cassandra-9042-10-126-15-196", "destination_service_id": "cassandra-9042-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 9042 } }, { "kind": "connect-proxy", "id": "cassandra-9160-10-126-15-196", "name": "cassandra-9160-10-126-15-196", "port": 10028, "enable_tag_override": false, "proxy": { "destination_service_name": "cassandra-9160-10-126-15-196", "destination_service_id": "cassandra-9160-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 9160 } }, { "kind": "connect-proxy", "id": "message-router-4527-10-126-15-196", "name": "message-router-4527-10-126-15-196", "port": 10029, "enable_tag_override": false, "proxy": { "destination_service_name": "message-router-4527-10-126-15-196", "destination_service_id": "message-router-4527-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 4527 } }, { "kind": "connect-proxy", "id": "message-processor-4528-10-126-15-196", "name": "message-processor-4528-10-126-15-196", "port": 10030, "enable_tag_override": false, "proxy": { "destination_service_name": "message-processor-4528-10-126-15-196", "destination_service_id": "message-processor-4528-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 4528 } }, { "kind": "connect-proxy", "id": "message-processor-8082-10-126-15-196", "name": "message-processor-8082-10-126-15-196", "port": 10031, "enable_tag_override": false, "proxy": { "destination_service_name": "message-processor-8082-10-126-15-196", "destination_service_id": "message-processor-8082-10-126-15-196", "local_service_address": "127.0.0.1", "local_service_port": 8082 } }, { "kind": "connect-proxy", "id": "egress-10-126-15-196", "name": "egress-10-126-15-196", "port": 10036, "enable_tag_override": false, "proxy": { "destination_service_name": "egress-10-126-15-196", "destination_service_id": "egress-10-126-15-196", "upstreams": [ { "destination_name": "zookeeper-2181-10-126-15-192", "local_bind_port": 10000 }, { "destination_name": "zookeeper-2888-10-126-15-192", "local_bind_port": 10001 }, { "destination_name": "zookeeper-3888-10-126-15-192", "local_bind_port": 10002 }, { "destination_name": "cassandra-9042-10-126-15-192", "local_bind_port": 10003 }, { "destination_name": "cassandra-9160-10-126-15-192", "local_bind_port": 10004 }, { "destination_name": "message-processor-4528-10-126-15-192", "local_bind_port": 10006 }, { "destination_name": "message-processor-8082-10-126-15-192", "local_bind_port": 10007 }, { "destination_name": "qpid-4529-10-126-15-193", "local_bind_port": 10008 }, { "destination_name": "qpid-5672-10-126-15-193", "local_bind_port": 10009 }, { "destination_name": "qpid-4529-10-126-15-194", "local_bind_port": 10013 }, { "destination_name": "qpid-5672-10-126-15-194", "local_bind_port": 10014 }, { "destination_name": "zookeeper-2181-10-126-15-195", "local_bind_port": 10018 }, { "destination_name": "zookeeper-2888-10-126-15-195", "local_bind_port": 10019 }, { "destination_name": "zookeeper-3888-10-126-15-195", "local_bind_port": 10020 }, { "destination_name": "cassandra-9042-10-126-15-195", "local_bind_port": 10021 }, { "destination_name": "cassandra-9160-10-126-15-195", "local_bind_port": 10022 } ] } } ], "skip_leave_on_interrupt": false, "start_join": [ "10.126.15.192", "10.126.15.193", "10.126.15.194", "10.126.15.195", "10.126.15.196" ], "verify_incoming": true, "verify_incoming_https": true, "verify_incoming_rpc": true, "verify_outgoing": true, "connect": { "ca_config": { "private_key": "-----BEGIN EC PRIVATE KEY----- lol -----END EC PRIVATE KEY-----\n", "root_cert": "-----BEGIN CERTIFICATE----- lol -----END CERTIFICATE-----\n", "csr_max_per_second": 100, "csr_max_concurrent": 1000, "leaf_cert_ttl": "3h0m0s" }, "ca_provider": "consul" } } ```I am fairly certain the configuration is erroneous, but that isn't the bug report.
I start my
consul proxyvia system.systemd unit file
``` [Unit] Description=local proxy for zookeeper:2181 Wants=network.target After=consul.service [Service] ExecStart=/opt/consul/bin/consul connect proxy -register -service zookeeper-2181-10-126-15-196 -service-addr 127.0.0.1:2181 -listen :10024 -log-level TRACE User=consul Group=consul StandardOutput=journal StandardError=journal KillSignal=SIGTERM [Install] WantedBy=multi-user.target ```After starting I'll perform the following verification steps
consul operator raft list-peers
```sh consul operator raft list-peers Node ID Address State Voter RaftProtocol prc-test-2-871 8d8c7c78-9f9a-efdc-d96b-3e8d8170695a 10.126.15.192:8300 follower true 3 prc-test-4-871 c5bb660b-cc82-0078-67a8-70b0c6b7d289 10.126.15.193:8300 leader true 3 prc-test-1-871 66863776-8591-3944-51bc-e96645f6b3de 10.126.15.196:8300 follower true 3 prc-test-3-871 0f6740f5-a958-f594-422e-a949a34d5401 10.126.15.194:8300 follower true 3 prc-test-0-871 9589ae27-8b95-f977-6cee-d351e776ad3c 10.126.15.195:8300 follower true 3 ```api stuff
```sh curl http://127.0.0.1:8500/v1/catalog/service/zookeeper-2181-10-126-15-192-proxy?consistent=true ``` which returns ```json [{"Node":{"ID":"8d8c7c78-9f9a-efdc-d96b-3e8d8170695a","Node":"prc-test-2-871","Address":"10.126.15.192","Datacenter":"dc1","TaggedAddresses":{"lan":"10.126.15.192","wan":"10.126.15.192"},"Meta":{"consul-network-segment":""},"CreateIndex":5,"ModifyIndex":9},"Service":{"Kind":"connect-proxy","ID":"zookeeper-2181-10-126-15-192-proxy","Service":"zookeeper-2181-10-126-15-192-proxy","Tags":[],"Address":"","Meta":null,"Port":10000,"Weights":{"Passing":1,"Warning":1},"EnableTagOverride":false,"ProxyDestination":"","Proxy":{"DestinationServiceName":"zookeeper-2181-10-126-15-192"},"Connect":{},"CreateIndex":225,"ModifyIndex":225},"Checks":[{"Node":"prc-test-2-871","CheckID":"serfHealth","Name":"Serf Health Status","Status":"passing","Notes":"","Output":"Agent alive and reachable","ServiceID":"","ServiceName":"","ServiceTags":[],"Definition":{},"CreateIndex":5,"ModifyIndex":5},{"Node":"prc-test-2-871","CheckID":"zookeeper-2181-10-126-15-192-proxy-ttl","Name":"proxy heartbeat","Status":"passing","Notes":"Built-in proxy will heartbeat this check.","Output":"","ServiceID":"zookeeper-2181-10-126-15-192-proxy","ServiceName":"zookeeper-2181-10-126-15-192-proxy","ServiceTags":[],"Definition":{},"CreateIndex":225,"ModifyIndex":225}]}] ```Details
So far my service mesh looks solid.
But when I check with
lsofthe proxy isn't listening:lsofoutput
``` lsof -n -P -iTCP -sTCP:LISTEN COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME master 1107 root 13u IPv4 16921 0t0 TCP 127.0.0.1:25 (LISTEN) master 1107 root 14u IPv6 16922 0t0 TCP [::1]:25 (LISTEN) sshd 2651 root 3u IPv4 23198 0t0 TCP *:22 (LISTEN) sshd 2651 root 4u IPv6 23200 0t0 TCP *:22 (LISTEN) consul 18001 consul 3u IPv6 151523 0t0 TCP *:8300 (LISTEN) consul 18001 consul 7u IPv6 151525 0t0 TCP *:8302 (LISTEN) consul 18001 consul 10u IPv6 151528 0t0 TCP *:8301 (LISTEN) consul 18001 consul 15u IPv4 151531 0t0 TCP 127.0.0.1:8600 (LISTEN) consul 18001 consul 16u IPv4 151533 0t0 TCP 127.0.0.1:8500 (LISTEN) consul 18001 consul 17u IPv6 151534 0t0 TCP *:8503 (LISTEN) ```Also when I look at the proxies logs
proxy logs
``` Jul 13 01:23:52 prc-test-2-871 consul[27981]: 2019/07/13 01:23:52 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:23:22 prc-test-2-871 consul[27981]: 2019/07/13 01:23:22 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:22:52 prc-test-2-871 consul[27981]: 2019/07/13 01:22:52 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:22:26 prc-test-2-871 consul[27981]: 2019/07/13 01:22:26 [ERR] consul.watch: Watch (type: connect_leaf) errored: Unexpected response code: 500 (rpc e Jul 13 01:22:26 prc-test-2-871 consul[27981]: 2019/07/13 01:22:26 [ERR] consul.watch: Watch (type: connect_roots) errored: Unexpected response code: 500 (rpc Jul 13 01:22:22 prc-test-2-871 consul[27981]: 2019/07/13 01:22:22 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:21:52 prc-test-2-871 consul[27981]: 2019/07/13 01:21:52 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:21:22 prc-test-2-871 consul[27981]: 2019/07/13 01:21:22 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:20:52 prc-test-2-871 consul[27981]: 2019/07/13 01:20:52 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:20:22 prc-test-2-871 consul[27981]: 2019/07/13 01:20:22 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:19:52 prc-test-2-871 consul[27981]: 2019/07/13 01:19:52 [DEBUG] proxy: service already registered, not re-registering Jul 13 01:19:26 prc-test-2-871 consul[27981]: 2019/07/13 01:19:26 [ERR] consul.watch: Watch (type: connect_leaf) errored: Unexpected response code: 500 (rpc e Jul 13 01:19:26 prc-test-2-871 consul[27981]: 2019/07/13 01:19:26 [ERR] consul.watch: Watch (type: connect_roots) errored: Unexpected response code: 500 (rpc Jul 13 01:19:22 prc-test-2-871 consul[27981]: 2019/07/13 01:19:22 [DEBUG] proxy: service already registered, not re-registering ```It appears that it hasn't received a leaf-certificate.
Bug Report
If the proxy doesn't have a SPIFEE cert, and isn't listening. Why it is classified as healthy?
Platform Details
consul - v1.5.1 amd6 Linux centos7.X