consul-connect-inject-init: "unable to find registered connect-proxy service"

[pankaj-dahiya-devops]

Overview of the Issue:

I am trying to use Consul service mesh (connect).

I am using a freshly deployed k8s cluster, a new consul-helm deployment and I try to deploy any other service with service mesh integration. The consul-connect-inject-init container fails and crash loops, so the pod never completes the init phase.

Reproduction Steps helm install consul hashicorp/consul --set connectInject.enabled=true --set connectInject.default=true.

Once consul is running, deploy any service. In my case a simple keycloak with postgresql manifest, but I have tried others with the same behaviour.

ubuntu@ip-172-31-37-34:~/pankaj$ kubectl get po NAME READY STATUS RESTARTS AGE pankaj-consul-server-1 0/1 Pending 0 15m pankaj-consul-server-2 0/1 Pending 0 15m pankaj-consul-kstjl 0/1 Running 0 15m pankaj-consul-webhook-cert-manager-5f5d9cf784-wjh5x 1/1 Running 0 15m pankaj-consul-controller-67968c745-hs24s 1/1 Running 0 15m pankaj-consul-connect-injector-webhook-deployment-5868b8d6mm5z4 1/1 Running 0 15m pankaj-consul-connect-injector-webhook-deployment-5868b8d67qzvr 1/1 Running 0 15m pankaj-keycloak-0 0/2 Init:0/3 0 15m pankaj-consul-server-0 0/1 Running 0 15m pankaj-postgresql-0 0/2 Init:1/2 5 15m

ubuntu@ip-172-31-37-34:~/pankaj$ kubectl logs pankaj-postgresql-0 consul-connect-inject-init 2021-10-06T18:43:53.093Z [INFO] Registered service has been detected: service=pankaj-postgresql-headless 2021-10-06T18:43:54.095Z [INFO] Registered service has been detected: service=pankaj-postgresql 2021-10-06T18:43:54.095Z [INFO] Registered service has been detected: service=pankaj-postgresql-headless 2021-10-06T18:43:55.096Z [INFO] Registered service has been detected: service=pankaj-postgresql 2021-10-06T18:43:55.096Z [INFO] Registered service has been detected: service=pankaj-postgresql-headless 2021-10-06T18:43:56.097Z [INFO] Registered service has been detected: service=pankaj-postgresql 2021-10-06T18:43:56.097Z [INFO] Registered service has been detected: service=pankaj-postgresql-headless 2021-10-06T18:43:56.097Z [ERROR] Timed out waiting for service registration: error="unable to find registered connect-proxy service"

[lkysow]

Can you show kubectl describe pod pankaj-consul-server-1. Both those server pods need to be running before anything will work.

If you're just doing testing then you can use a single server:

server:
  replicas: 1

[pankaj-dahiya-devops]

ubuntu@ip-172-31-37-34:~/pankaj$ kubectl describe pod pankaj-consul-server-1

Name:           pankaj-consul-server-1
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=consul
                chart=consul-helm
                component=server
                controller-revision-hash=pankaj-consul-server-59f47fbcd6
                hasDNS=true
                release=pankaj
                statefulset.kubernetes.io/pod-name=pankaj-consul-server-1
Annotations:    consul.hashicorp.com/config-checksum: 914926d24d7bf77220ba40bb3625bc3103f9ddceb7042745eb0454d507be5706
                consul.hashicorp.com/connect-inject: false
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/pankaj-consul-server
Containers:
  consul:
    Image:       hashicorp/consul:1.10.2
    Ports:       8500/TCP, 8301/TCP, 8301/UDP, 8302/TCP, 8302/UDP, 8300/TCP, 8600/TCP, 8600/UDP
    Host Ports:  0/TCP, 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/UDP
    Command:
      /bin/sh
      -ec
      CONSUL_FULLNAME="pankaj-consul"

      mkdir -p /consul/extra-config
      cp /consul/config/extra-from-values.json /consul/extra-config/extra-from-values.json
      [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /consul/extra-config/extra-from-values.json
      [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /consul/extra-config/extra-from-values.json
      [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /consul/extra-config/extra-from-values.json

      exec /usr/local/bin/docker-entrypoint.sh consul agent \
        -advertise="${ADVERTISE_IP}" \
        -bind=0.0.0.0 \
        -bootstrap-expect=3 \
        -client=0.0.0.0 \
        -config-dir=/consul/config \
        -datacenter=dc1 \
        -data-dir=/consul/data \
        -domain=consul \
        -hcl="connect { enabled = true }" \
        -ui \
        -retry-join="${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
        -retry-join="${CONSUL_FULLNAME}-server-1.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
        -retry-join="${CONSUL_FULLNAME}-server-2.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
        -serf-lan-port=8301 \
        -config-file=/consul/extra-config/extra-from-values.json \
        -server

    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:      100m
      memory:   100Mi
    Readiness:  exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader \
2>/dev/null | grep -E '".+"'
] delay=5s timeout=5s period=3s #success=1 #failure=2
    Environment:
      ADVERTISE_IP:               (v1:status.podIP)
      HOST_IP:                    (v1:status.hostIP)
      POD_IP:                     (v1:status.podIP)
      NAMESPACE:                 default (v1:metadata.namespace)
      CONSUL_DISABLE_PERM_MGMT:  true
    Mounts:
      /consul/config from config (rw)
      /consul/data from data-default (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ppd2q (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  data-default:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-default-pankaj-consul-server-1
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pankaj-consul-server-config
    Optional:  false
  kube-api-access-ppd2q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  20m   default-scheduler  0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity rules, 1 node(s) didn't match pod anti-affinity rules.
  Warning  FailedScheduling  20m   default-scheduler  0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity rules, 1 node(s) didn't match pod anti-affinity rules.

[lkysow]

I think you don't have enough nodes in your cluster. See my previous comment if it's a test cluster.

[pankaj-dahiya-devops]

@lkysow As advised by you, I have reduced the replicas count for server and connectInject to 1, now all my consul pods are ready but still same issue!

`ubuntu@ip-172-31-37-34:~/pankaj$ kubectl get po NAME READY STATUS RESTARTS AGE pankaj-consul-webhook-cert-manager-5f5d9cf784-wjh5x 1/1 Running 0 25m pankaj-consul-controller-67968c745-hs24s 1/1 Running 0 25m pankaj-consul-connect-injector-webhook-deployment-5868b8d6mm5z4 1/1 Running 0 25m pankaj-keycloak-0 0/2 Init:0/3 0 24m pankaj-postgresql-0 0/2 Init:CrashLoopBackOff 6 24m pankaj-consul-server-0 1/1 Running 0 32s pankaj-consul-drsqz 1/1 Running 0 22s

ubuntu@ip-172-31-37-34:~/pankaj$ kubectl describe pod pankaj-consul-server-0 Name: pankaj-consul-server-0 Namespace: default Priority: 0 Node: ip-172-31-37-34/172.31.37.34 Start Time: Wed, 06 Oct 2021 18:54:21 +0000 Labels: app=consul chart=consul-helm component=server controller-revision-hash=pankaj-consul-server-54f4d9d687 hasDNS=true release=pankaj statefulset.kubernetes.io/pod-name=pankaj-consul-server-0 Annotations: consul.hashicorp.com/config-checksum: 914926d24d7bf77220ba40bb3625bc3103f9ddceb7042745eb0454d507be5706 consul.hashicorp.com/connect-inject: false Status: Running IP: 10.42.0.40 IPs: IP: 10.42.0.40 Controlled By: StatefulSet/pankaj-consul-server Containers: consul: Container ID: containerd://e3c9fdf93032beeb204eed787abcae3ecfb7d1fd682c9fbbb1bc5a6fccb2327b Image: hashicorp/consul:1.10.2 Image ID: docker.io/hashicorp/consul@sha256:8c6d8777e211920a0d939c723ab5b0bf5855ac88f5d3df0b5072601e4b973899 Ports: 8500/TCP, 8301/TCP, 8301/UDP, 8302/TCP, 8302/UDP, 8300/TCP, 8600/TCP, 8600/UDP Host Ports: 0/TCP, 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/UDP Command: /bin/sh -ec CONSUL_FULLNAME="pankaj-consul"

  mkdir -p /consul/extra-config
  cp /consul/config/extra-from-values.json /consul/extra-config/extra-from-values.json
  [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /consul/extra-config/extra-from-values.json
  [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /consul/extra-config/extra-from-values.json
  [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /consul/extra-config/extra-from-values.json

  exec /usr/local/bin/docker-entrypoint.sh consul agent \
    -advertise="${ADVERTISE_IP}" \
    -bind=0.0.0.0 \
    -bootstrap-expect=1 \
    -client=0.0.0.0 \
    -config-dir=/consul/config \
    -datacenter=dc1 \
    -data-dir=/consul/data \
    -domain=consul \
    -hcl="connect { enabled = true }" \
    -ui \
    -retry-join="${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
    -serf-lan-port=8301 \
    -config-file=/consul/extra-config/extra-from-values.json \
    -server

State:          Running
  Started:      Wed, 06 Oct 2021 18:54:22 +0000
Ready:          True
Restart Count:  0
Limits:
  cpu:     100m
  memory:  100Mi
Requests:
  cpu:      100m
  memory:   100Mi
Readiness:  exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader \

2>/dev/null

grep -E '".+"' ] delay=5s timeout=5s period=3s #success=1 #failure=2 Environment: ADVERTISE_IP: (v1:status.podIP) HOST_IP: (v1:status.hostIP) POD_IP: (v1:status.podIP) NAMESPACE: default (v1:metadata.namespace) CONSUL_DISABLE_PERM_MGMT: true Mounts: /consul/config from config (rw) /consul/data from data-default (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jv4zs (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: data-default: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-default-pankaj-consul-server-0 ReadOnly: false config: Type: ConfigMap (a volume populated by a ConfigMap) Name: pankaj-consul-server-config Optional: false kube-api-access-jv4zs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 69s default-scheduler Successfully assigned default/pankaj-consul-server-0 to ip-172-31-37-34 Normal Pulled 69s kubelet Container image "hashicorp/consul:1.10.2" already present on machine Normal Created 69s kubelet Created container consul Normal Started 68s kubelet Started container consul Warning Unhealthy 57s (x3 over 63s) kubelet Readiness probe failed:

`

[pankaj-dahiya-devops]

@lkysow are you able to replicate the same and able to find a solution?

[pankaj-dahiya-devops]

@lkysow Can you please change the label of this ticket from "waiting-on-response" to something like a "bug" so that any other contributors may also look at this! FYI, I am still unable to find a resolution for the same.

[lkysow]

Hi can you please follow our bug template and provide all the data we need to investigate:

1. Helm values
2. Helm versions, Kubernetes versions
3. The YAML for your service
4. The output of kubectl get endpoints
5. The logs of the connect inject deployment.

[pankaj-dahiya-devops]

ubuntu@ip-172-31-37-34:~$ helm version version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"clean", GoVersion:"go1.16.5"}

ubuntu@ip-172-31-37-34:~$ kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4+k3s1", GitCommit:"3e250fdbab72d88f7e6aae57446023a0567ffc97", GitTreeState:"clean", BuildDate:"2021-08-19T19:09:53Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4+k3s1", GitCommit:"3e250fdbab72d88f7e6aae57446023a0567ffc97", GitTreeState:"clean", BuildDate:"2021-08-19T19:09:53Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

Helm values : --set connectInject.enabled=true --set connectInject.default=true.

ubuntu@ip-172-31-37-34:~$ kubectl get endpoints NAME ENDPOINTS AGE kubernetes 172.31.37.34:6443 3d8h pankaj-consul-controller-webhook 10.42.0.32:9443 46h pankaj-keycloak-headless 46h pankaj-keycloak-http 46h pankaj-consul-connect-injector-svc 10.42.0.33:8080 46h pankaj-consul-server 10.42.0.40:8600,10.42.0.40:8300,10.42.0.40:8600 + 5 more... 46h pankaj-consul-ui 10.42.0.40:8500 46h pankaj-consul-dns 10.42.0.40:8600,10.42.0.41:8600,10.42.0.40:8600 + 1 more... 46h pankaj-postgresql 46h pankaj-postgresql-headless 10.42.0.37:5432 46h ubuntu@ip-172-31-37-34:~$ kubectl get po NAME READY STATUS RESTARTS AGE pankaj-consul-webhook-cert-manager-5f5d9cf784-wjh5x 1/1 Running 0 46h pankaj-consul-controller-67968c745-hs24s 1/1 Running 0 46h pankaj-consul-connect-injector-webhook-deployment-5868b8d6mm5z4 1/1 Running 0 46h pankaj-keycloak-0 0/2 Init:0/3 0 46h pankaj-consul-server-0 1/1 Running 0 46h pankaj-consul-drsqz 1/1 Running 0 46h pankaj-consul-sync-catalog-5c475c86f-sd6qk 1/1 Running 0 32h pankaj-postgresql-0 0/2 Init:CrashLoopBackOff 393 46h

ubuntu@ip-172-31-37-34:~/pankaj$ kubectl logs pankaj-consul-connect-injector-webhook-deployment-5868b8d6s9nbf 2021-10-08T16:57:03.956Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "0.0.0.0:9444"} 2021-10-08T16:57:03.957Z INFO controller-runtime.webhook registering webhook {"path": "/mutate"} 2021-10-08T16:57:03.957Z INFO attempting to acquire leader lease default/consul-controller-lock...

2021-10-08T16:57:03.957Z INFO starting metrics server {"path": "/metrics"} 2021-10-08T16:57:03.957Z INFO controller-runtime.webhook.webhooks starting webhook server 2021-10-08T16:57:03.957Z INFO controller-runtime.certwatcher Updated current TLS certificate 2021-10-08T16:57:03.957Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 8080} 2021-10-08T16:57:03.957Z INFO controller-runtime.certwatcher Starting certificate watcher 2021-10-08T16:57:16.457Z INFO handler.connect received pod {"name": "pankaj-postgresql-0", "ns": "default"} 2021-10-08T16:57:16.457Z INFO handler.connect received pod {"name": "pankaj-keycloak-0", "ns": "default"} 2021-10-08T16:57:21.982Z INFO successfully acquired lease default/consul-controller-lock

2021-10-08T16:57:21.983Z INFO controller.endpoints Starting EventSource {"reconciler group": "", "reconciler kind": "Endpoints", "source": "kind source: /, Kind="} 2021-10-08T16:57:21.983Z INFO controller.endpoints Starting EventSource {"reconciler group": "", "reconciler kind": "Endpoints", "source": "kind source: /, Kind="} 2021-10-08T16:57:21.983Z INFO controller.endpoints Starting Controller {"reconciler group": "", "reconciler kind": "Endpoints"} 2021-10-08T16:57:22.155Z INFO controller.endpoints received update for Consul client pod {"name": "pankaj-consul-57qgc"} 2021-10-08T16:57:22.155Z INFO controller.endpoints ignoring Consul client pod because it's not ready {"name": "pankaj-consul-57qgc"} 2021-10-08T16:57:22.255Z INFO controller.endpoints Starting workers {"reconciler group": "", "reconciler kind": "Endpoints", "worker count": 1} 2021-10-08T16:57:22.255Z INFO controller.endpoints retrieved {"name": "pankaj-keycloak-headless", "ns": "default"} 2021-10-08T16:57:22.257Z INFO controller.endpoints retrieved {"name": "pankaj-consul-ui", "ns": "default"} 2021-10-08T16:57:22.261Z INFO controller.endpoints retrieved {"name": "kubernetes", "ns": "default"} 2021-10-08T16:57:22.262Z INFO controller.endpoints retrieved {"name": "pankaj-consul-dns", "ns": "default"} 2021-10-08T16:57:22.263Z INFO controller.endpoints retrieved {"name": "pankaj-keycloak-http", "ns": "default"} 2021-10-08T16:57:22.264Z INFO controller.endpoints retrieved {"name": "pankaj-consul-controller-webhook", "ns": "default"} 2021-10-08T16:57:22.265Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:22.755Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:22.761Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:22.762Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.762Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.763Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.763Z INFO controller.endpoints retrieved {"name": "pankaj-consul-connect-injector-svc", "ns": "default"} 2021-10-08T16:57:22.765Z INFO controller.endpoints retrieved {"name": "pankaj-consul-server", "ns": "default"} 2021-10-08T16:57:22.766Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:22.766Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:22.766Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:22.771Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:22.854Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.855Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.856Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.856Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:22.857Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:22.954Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:22.955Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.955Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.956Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.957Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:22.957Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:22.957Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:22.961Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:22.961Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.961Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.962Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:22.967Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:22.967Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.054Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:23.055Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.055Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.056Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.056Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:23.056Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:23.057Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.061Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:23.062Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.062Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.063Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.077Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:23.077Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.084Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:23.085Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.085Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.156Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.156Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:23.156Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:23.156Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.161Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:23.162Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.162Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.163Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.197Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:23.198Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.256Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:23.257Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.257Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.258Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.258Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:23.258Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:23.259Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.262Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:23.263Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.263Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.264Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.339Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:23.340Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.343Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:23.354Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.354Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.356Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.357Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:23.357Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:23.357Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.360Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:23.361Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.361Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.362Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.517Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:23.518Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.523Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:23.525Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.525Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.526Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.526Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:23.527Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:23.527Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.554Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:23.556Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.556Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.557Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.847Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:23.847Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.850Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:23.851Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.851Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.852Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.878Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:23.879Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:23.879Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:23.882Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:23.883Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.883Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:23.884Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:24.493Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:24.493Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:24.497Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:24.498Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:24.498Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:24.499Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:24.525Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:24.525Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:24.526Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:24.529Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:24.530Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:24.530Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:24.531Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:25.779Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:25.780Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:25.783Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:25.784Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:25.784Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:25.785Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:25.812Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:25.812Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:25.812Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:25.815Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:25.816Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:25.816Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:25.817Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:26.730Z INFO handler.connect received pod {"name": "pankaj-keycloak-0", "ns": "default"} 2021-10-08T16:57:28.346Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:28.346Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:28.351Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:28.351Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:28.351Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:28.352Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:28.378Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:28.378Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:28.378Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:28.382Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:28.382Z ERROR controller.endpoints failed to register proxy service {"name": "pankaj-postgresql-headless-sidecar-proxy", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} github.com/hashicorp/consul-k8s/control-plane/connect-inject.(EndpointsController).Reconcile /home/circleci/project/project/control-plane/connect-inject/endpoints_controller.go:170 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:28.382Z ERROR controller.endpoints failed to register services or health check {"name": "pankaj-postgresql-headless", "ns": "default", "error": "Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:28.383Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "pankaj-postgresql-headless", "namespace": "default", "error": "1 error occurred:\n\t Unexpected response code: 500 (could not retrieve initial service_defaults config for service \"pankaj-postgresql-0-pankaj-postgresql-headless-sidecar-proxy\": No known Consul servers)\n\n"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214 2021-10-08T16:57:33.473Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:33.474Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:33.478Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:33.563Z INFO controller.endpoints updating health check status for service {"name": "pankaj-postgresql", "reason": "Pod \"default/pankaj-postgresql-0\" is not ready", "status": "critical"} 2021-10-08T16:57:33.963Z INFO controller.endpoints updating health check {"id": "default/pankaj-postgresql-0-pankaj-postgresql/kubernetes-health-check"} 2021-10-08T16:57:34.061Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:34.061Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:34.061Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:34.066Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:34.160Z INFO controller.endpoints updating health check status for service {"name": "pankaj-postgresql-headless", "reason": "Kubernetes health checks passing", "status": "passing"} 2021-10-08T16:57:34.355Z INFO controller.endpoints updating health check {"id": "default/pankaj-postgresql-0-pankaj-postgresql-headless/kubernetes-health-check"} 2021-10-08T16:57:34.969Z INFO controller.endpoints received update for Consul client pod {"name": "pankaj-consul-57qgc"} 2021-10-08T16:57:34.970Z INFO controller.endpoints received update for Consul client pod {"name": "pankaj-consul-57qgc"} 2021-10-08T16:57:34.970Z INFO controller.endpoints retrieved {"name": "pankaj-consul-ui", "ns": "default"} 2021-10-08T16:57:34.972Z INFO controller.endpoints retrieved {"name": "pankaj-consul-dns", "ns": "default"} 2021-10-08T16:57:34.973Z INFO controller.endpoints retrieved {"name": "pankaj-consul-controller-webhook", "ns": "default"} 2021-10-08T16:57:34.974Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql", "ns": "default"} 2021-10-08T16:57:34.975Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql", "id": "pankaj-postgresql-0-pankaj-postgresql", "agentIP": "172.31.37.34"} 2021-10-08T16:57:34.978Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-sidecar-proxy"} 2021-10-08T16:57:35.062Z INFO controller.endpoints updating health check status for service {"name": "pankaj-postgresql", "reason": "Pod \"default/pankaj-postgresql-0\" is not ready", "status": "critical"} 2021-10-08T16:57:35.065Z INFO controller.endpoints retrieved {"name": "pankaj-consul-connect-injector-svc", "ns": "default"} 2021-10-08T16:57:35.066Z INFO controller.endpoints retrieved {"name": "pankaj-consul-server", "ns": "default"} 2021-10-08T16:57:35.067Z INFO controller.endpoints retrieved {"name": "pankaj-postgresql-headless", "ns": "default"} 2021-10-08T16:57:35.067Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-postgresql-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:35.067Z INFO controller.endpoints registering service with Consul {"name": "pankaj-postgresql-headless", "id": "pankaj-postgresql-0-pankaj-postgresql-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:35.160Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-postgresql-headless-sidecar-proxy"} 2021-10-08T16:57:35.171Z INFO controller.endpoints updating health check status for service {"name": "pankaj-postgresql-headless", "reason": "Kubernetes health checks passing", "status": "passing"} 2021-10-08T16:57:35.174Z INFO controller.endpoints retrieved {"name": "pankaj-consul-dns", "ns": "default"} 2021-10-08T16:57:47.237Z INFO handler.connect received pod {"name": "pankaj-keycloak-0", "ns": "default"} 2021-10-08T16:57:48.385Z INFO controller.endpoints retrieved {"name": "pankaj-keycloak-http", "ns": "default"} 2021-10-08T16:57:48.386Z INFO controller.endpoints registering service with Consul {"name": "pankaj-keycloak-http", "id": "pankaj-keycloak-0-pankaj-keycloak-http", "agentIP": "172.31.37.34"} 2021-10-08T16:57:48.394Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-keycloak-http-sidecar-proxy"} 2021-10-08T16:57:48.659Z INFO controller.endpoints updating health check status for service {"name": "pankaj-keycloak-http", "reason": "Pod \"default/pankaj-keycloak-0\" is not ready", "status": "critical"} 2021-10-08T16:57:48.755Z INFO controller.endpoints updating health check {"id": "default/pankaj-keycloak-0-pankaj-keycloak-http/kubernetes-health-check"} 2021-10-08T16:57:48.763Z INFO controller.endpoints retrieved {"name": "pankaj-keycloak-headless", "ns": "default"} 2021-10-08T16:57:48.763Z INFO controller.endpoints skipping syncing service cluster IP to Consul {"name": "pankaj-keycloak-headless", "ns": "default", "ip": "None"} 2021-10-08T16:57:48.763Z INFO controller.endpoints registering service with Consul {"name": "pankaj-keycloak-headless", "id": "pankaj-keycloak-0-pankaj-keycloak-headless", "agentIP": "172.31.37.34"} 2021-10-08T16:57:48.772Z INFO controller.endpoints registering proxy service with Consul {"name": "pankaj-keycloak-headless-sidecar-proxy"} 2021-10-08T16:57:48.868Z INFO controller.endpoints updating health check status for service {"name": "pankaj-keycloak-headless", "reason": "Pod \"default/pankaj-keycloak-0\" is not ready", "status": "critical"} 2021-10-08T16:57:49.158Z INFO controller.endpoints updating health check {"id": "default/pankaj-keycloak-0-pankaj-keycloak-headless/kubernetes-health-check"}

[lkysow]

Thanks for the logs! Can you provide the YAML for your services or something similar so we could try and reproduce this ourselves?

[pankaj-dahiya-devops]

Which service YAML file do you need? I am using the original chart of consul-k8s and keycloak! I have enabled injector and reduced replica count to 1 in value.yaml

[lkysow]

The keycloak install YAML.

[pankaj-dahiya-devops]

I am using the official keycloak helm chart without any changes: https://github.com/codecentric/helm-charts/tree/master/charts/keycloak

FYI, I have tried the same with few other applications too, all giving the same error.!

[pankaj-dahiya-devops]

Hi @lkysow, I think this is a critical bug if it is a bug. It's been more than 4 days, are you able to reproduce the same?

[pankaj-dahiya-devops]

@lkysow are you able to find something? I can see the tag of the ticket is still waiting-on-response, do you need any more input from my side?

[kschoche]

Hi @pankaj-dahiya! I gave this a try on my end and I can see the same behaviour as you're reporting. The reason that connect-init is stuck in a crash loop is that keycloak registers two Kubernetes services [keycloak-headless, keycloak-http] and we do not currently support a model where multiple kube services can reference the same pods in Consul. I was able to work around it and get the keycloak-http service and pod to come online by deleting the keycloak-headless kube service and restarting the keycloak pod. I unfortunately do not know enough about keycloak to know what affect this will have on its behaviour as I don't know how to use keycloak. You may want to give that a try and see if it works for you. Also as a general note I noticed that even the keycloak-http service has 3 service ports and we currently only support a single service port in the mesh, so you may need to utilize https://www.consul.io/docs/k8s/connect#consul-hashicorp-com-transparent-proxy-exclude-inbound-ports && https://www.consul.io/docs/k8s/connect#consul-hashicorp-com-transparent-proxy-exclude-outbound-ports for the administrative ports, and these ports would not be utilizing the service mesh.

We do have support for multi-port services as well as managing situations where multiple kube services reference the same pods in our roadmap.

[pankaj-dahiya-devops]

Hi @kschoche , rather than deleting a service of a pod, is there a way to exclude it , just like we can do for ports ?

[kschoche]

Hi @pankaj-dahiya - Unfortunately this is not possible in the current architecture. In theory an annotation to the k8s service object could be added and we could look at that when trying to manage consul service registration, however that would probably not be an ideal configuration anyways because you would have some ports of your service on the service mesh and some not on the service mesh.

[pankaj-dahiya-devops]

So does that mean, we should drop consul for at least this time, because the service mesh have such limitation which can't be overridden soon!

[pankaj-dahiya-devops]

@kschoche , you have specified above that consul can't register services if multiple services are pointing to a single pod, but we can see that the vault have multiple services with multiple ports, but its service is still registered!

[kschoche]

Hi @pankaj-dahiya! Could you clarify your last comment a little bit please? I'm a little confused about mentioning vault for context of this discussion.

To clarify my statements about the current support of multi-port mesh services in Consul, keycloak is deploying two types of kube services which are problematic in our current architecture :

1. multiple k8s services that reference the same pod, iirc the services are named keycloak-http and keycloak-headless.
2. keycloak-http has 3 service ports (more than 1).

On point (1): The consul-k8s endpoints-controller does indeed attempt to register both kube services with Consul, this is an unfortunate artifact of the current design and probably should be handled differently so that it does not even attempt it. However, when the keycloak pod attempts to start it will only have a single envoy proxy attached via the consul webhook and this will service only a single application port. The connect-init init container will be unable to determine which consul or k8s service it is supposed to configure the envoy proxy for, so it will fail to start.

For point (2): Consul currently only exposes a single port per sidecar proxy, thus, traffic can only flow into a single port of an upstream service since the port is exposed by a single Envoy process running as a sidecar. Consul also does not natively support routing traffic from a downstream service to multiple ports for a single upstream service.

Unfortunately we do not currently support multi-port service mesh applications, or having multiple kube/consul services reference the same pods. We do plan to include this support in the future and it is on our roadmap.

[david-yu]

Closing as this is related to multi-port services which is being tracked here: https://github.com/hashicorp/consul/issues/5388. As far as the case where multiple k8s services select the same pod, we are releasing a feature that allows you to ignore services via a consul.hashicorp.com/service-ignore label: https://github.com/hashicorp/consul-k8s/pull/858

[mister2d]

@kschoche , you have specified above that consul can't register services if multiple services are pointing to a single pod, but we can see that the vault have multiple services with multiple ports, but its service is still registered!

Hi @pankaj-dahiya! Could you clarify your last comment a little bit please? I'm a little confused about mentioning vault for context of this discussion.

I believe vault was mentioned because Vault can register its services with Consul having multiple service on multiple ports, but not keycloak. For that matter, I am stuck trying to deploy a mariadb cluster with registered Consul services because it registers three services: mariadb, mariadb-headless, and mariadb-galera-metrics. Why would it work for Vault and not anything else (as far as I can tell)?