Unable to find pod with katalog-sync annotation (katalog-sync.wish.com/service-names)

[ylohnitram]

time="2020-03-11T10:45:36Z" level=error msg="error registering with katalog-sync-daemon: Unknown rpc error: code = Unknown desc = Unable to find pod with katalog-sync annotation (katalog-sync.wish.com/service-names): /api/v1/namespaces/default/pods/hw-7c88d77f6-99c2m"

deployment.yaml:

apiVersion: apps/v1 kind: Deployment metadata: name: hw labels: app: hw annotations: katalog-sync.wish.com/sidecar: "katalog-sync-sidecar" katalog-sync.wish.com/service-names: "hw" katalog-sync.wish.com/service-port: '8080' spec: replicas: 1 selector: matchLabels: app: hw template: metadata: labels: app: hw annotations: "katalog-sync.wish.com/sidecar": "katalog-sync-sidecar" "katalog-sync.wish.com/service-names": "hw" "katalog-sync.wish.com/service-port": '8080'

katalog-sync.wish.com/service-port-servicename: '12345'

    #katalog-sync.wish.com/service-meta: 'a:1,b:2'
    #katalog-sync.wish.com/service-meta-servicename: 'b:1,c:2'
    #katalog-sync.wish.com/service-tags: a,b
    "katalog-sync.wish.com/sync-interval": 2s
spec:
  terminationGracePeriodSeconds: 1
  containers:
  - name: hw
    image: "... IMAGE ..."
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
    livenessProbe:
      httpGet:
        path: "/live"
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
    readinessProbe:
      httpGet:
        path: "/ready"
        port: 8080
      periodSeconds: 5
  - command:
    - "/bin/katalog-sync-sidecar"
    args:
    - "--katalog-sync-daemon=$(HOST_IP):8501"
    - "--namespace=$(MY_POD_NAMESPACE)"
    - "--pod-name=$(MY_POD_NAME)"
    - "--container-name=katalog-sync-sidecar"
    - "--bind-address=:8888"
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: MY_POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    image: "... IMAGE ..."
    imagePullPolicy: Always
    name: katalog-sync-sidecar
    readinessProbe:
      httpGet:
        path: "/ready"
        port: 8080
      initialDelaySeconds: 1
      periodSeconds: 5

[jacksontj]

Curious as this looks like the example configuration (although the indentation got eaten by github). Could you re-paste in a code block (using ` 3 times)-- as it looks like it should work.

[jacksontj]

I'm going to close this out for now as we haven't heard back in a while. If you are still seeing the same issue please feel free to re-open :)

[gempesaw]

i got this error as well, but it was a slightly different culprit. as part of our hardened AWS EKS image, we disable the kubelet readonly port that is typically open on 10255, so we have to use the read/write https port with authentication on https://localhost:10250.

https://github.com/awslabs/amazon-eks-ami/issues/128

        env:
        - name: KUBELET_API
          value: 'https://$(HOST_IP):10250/pods'

(also i didn't use hostNetwork: true so that's why i'm using $(HOST_IP) injected via downwardAPI. for hostNetwork: true, it should be fine for https://localhost:10250/pods)

---

i did have some trouble debugging at first, cuz i read the docs wrong and didn't pass kubelet address?

instead i passed the normal like .. https://kubernetes.default:443/pods kubernetes API default namespace Service address. but of course that is not as helpful, because it's not the node-local pods that you get from the kubelet. and also /pods is not a valid URL in the kubernetes API lol

but the trick was, neither the sidecar nor the daemonset indicated that i did anything wrong, other than the error message unable to find pod...