Consul + Kubernetes + Ocelot + Connect injector

[gianniskt]

Hello,

I have installed the helm chart successfully, and trying to use the connect-inject as clarified here: https://www.consul.io/docs/k8s/connect#installation-and-configuration Using these values:

connectInject:
   enabled: true
   default: true

I am using Ocelot as API Gateway with this configuration on ocelot.json:

 "ServiceDiscoveryProvider": {
      "Scheme": "https",
      "Host": "<consul-service-name>",
      "Port": 8501,
      "Token": "<consul-token>",
      "Type": "Consul"
    }

Without connect-inject everything is fine, the services are registering on Consul, and the requests are forwarded as expected. By enabling connect-inject, I receive the error from POD Logs:

error: a container name must be specified for pod microservice1, choose one of: [app1 envoy-sidecar] or one of the init containers: [copy-consul-bin consul-connect-inject-init]

Also these are the logs from consul-connect-inject:

INFO    controller.endpoints    registering proxy service with Consul   {"name": "service1-sidecar-proxy"}
INFO    controller.endpoints    updating health check status for service        {"name": "service1", "reason": "Pod \"docker-repository/pod1\" is not ready", "status": "critical"}
[ERROR] Timed out waiting for service registration: error="unable to find registered connect-proxy service"

I haven' t tried to set annotations on Deployments like "consul.hashicorp.com/connect-service-upstreams: service1:80" as I suppose is not needed, due to "default: true". Is this a bug related to issue #764 , or something else that I am missing on the configuration?

I read somewhere that Kubernetes Services have to be Headless to integrate with Service Mesh (Consul Connect). Is this a possible solution?

[kschoche]

Hi @gianniskt! Thanks for filing this issue! The first error you mentioned is just a standard kubernetes message, you'll just need to specify a container for which you want to see logs: kubectl logs -f <your-pod-name> -c app1. The fact that you're seeing envoy-sidecar in that log messages means that it is connect-injected.

For your second error, generally the reason the pod is unable to find Consul services is that you've not created a Kube Service for your application. This is required. I believe if you'd let that pod error for another minute or two it would've printed an error message suggesting that you do so, and to check the connect webhook deployment logs for additional info! If you've created a Kube service for your connect application already, then the next place to look is the webhook deployment logs: ex: kubectl logs -f kyle-consul-consul-connect-injector-webhook-deployment-84d5dbq5 .. keeping in mind that there are 2 replicas so one will not be populated.

If you look at the logs of the webhook it may provide additional information.

Oh, to answer your last question, your service does not need to be Headless, but it does need to exist!

[gianniskt]

Hello @kschoche

Thanks for your answers. Yes my service for the deployment exists. This is the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      namespace: test
      labels:
        app: app1
    spec:
      containers:
      - name: app1
        image: ${DOCKER-REGISTRY-}app1:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 80

And this is the service definition for the Deployment:

apiVersion: v1
kind: Service
metadata:
  name: app1-service
  namespace: test
spec:
  ports:
    - port: 80
  selector:
      app: app1

If I disable connect-inject, everything is fine and my service is accessible through "http://app1-service-test.service.consul" and Ocelot route the appropriate request to that address. Also at the UI a service with name "app1-service-test" is created, as I have "addK8SNamespaceSuffix: true" at Consul Installation.

With Connect Inject enabled, the UI shows the same service but the Check id is "test/app1-697664bd65-5f29g-app1-service/kubernetes-health-check" and health checks are failing with output: Pod "test/app1-697664bd65-5f29g" is not ready

The logs from connect-inject deployment are showing this:

INFO    controller.endpoints    retrieved       {"name": "app1-service", "ns": "test"}
INFO    controller.endpoints    registering service with Consul {"name": "app1", "id": "app1-697664bd65-5f29g-app1-service", "agentIP": "10.23.45.44"}
INFO    controller.endpoints    registering proxy service with Consul   {"name": "app1-service-sidecar-proxy"}
INFO    controller.endpoints    updating health check status for service        {"name": "app1-service", "reason": "Pod \"test/app1-697664bd65-5f29g\" is not ready", "status": "critical"}
INFO    controller.endpoints    updating health check   {"id": "test/app1-697664bd65-5f29g-app1-service/kubernetes-health-check"}

Furthermore the POD logs from all containers are showing this output:

[INFO]  Registered service has been detected: service=app1
[INFO]  Registered service has been detected: service=app1-sidecar-proxy
[INFO]  Connect initialization completed
invalid character '\r' in string literal
Error from server (BadRequest): container "app1" in pod "app1-697664bd65-5f29g" is waiting to start: PodInitializing
Unable to find registered services; retrying
Check to ensure a Kubernetes service has been created for this application. If your pod is not starting also check the connect-inject deployment logs.

Just to mention that Consul is installed to different namespace (default) from my application & services (test), but I don't think that it's a problem.

I am not sure if the problem is at the configuration of Ocelot.json or the Service Definition, or maybe a bug of the Envoy Proxy.