Expose / pod never starts fails on readinessProbe

[echo-devnull]

I created a kubernetes service that listens on port 222 which I want to "reverse tunnel" to my local machine.

kubectl create svc clusterip gitlab --tcp=222:222

Then I expose it via:

localizer expose default/gitlab

But the pod that gets created, never actually gets to "running" state:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m45s                default-scheduler  Successfully assigned default/localizer-gitlab-kjqpt to lgs-lnx7624
  Normal   Pulled     2m44s                kubelet            Container image "linuxserver/openssh-server" already present on machine
  Normal   Created    2m44s                kubelet            Created container default
  Normal   Started    2m44s                kubelet            Started container default
  Warning  Unhealthy  5s (x20 over 2m44s)  kubelet            Readiness probe failed: dial tcp 172.29.4.101:2222: connect: connection refused

The yaml:

  apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 58d1352a3f8739d41f8589dd1e1dd31a15517060b79687e1cee386f9a45e7454
    cni.projectcalico.org/podIP: 172.29.4.101/32
    cni.projectcalico.org/podIPs: 172.29.4.101/32
    kubernetes.io/psp: default-psp
    localizer.jaredallard.github.com/objects: '[]'
  labels:
    app: gitlab
    localizer.jaredallard.github.com/exposed: "true"
  name: localizer-gitlab-kjqpt
  namespace: default
spec:
  containers:
  - env:
    - name: PASSWORD_ACCESS
      value: "true"
    - name: USER_PASSWORD
      value: supersecretpassword
    - name: USER_NAME
      value: outreach
    - name: DOCKER_MODS
      value: linuxserver/mods:openssh-server-ssh-tunnel
    image: linuxserver/openssh-server
    imagePullPolicy: IfNotPresent
    name: default
    ports:
    - containerPort: 222
    readinessProbe:
      tcpSocket:
        port: 2222
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-9kgvq
      readOnly: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: OnFailure
  serviceAccountName: default
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-9kgvq
    projected:
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              fieldPath: metadata.namespace
            path: namespace

And the logging from the running localizer:

INFO[0023] created pod localizer-gitlab-kjqpt            component=exposer service=default/gitlab
INFO[0023] waiting for remote pod to be ready ...        component=exposer service=default/gitlab
INFO[0360] pkg/mod/github.com/jaredallard/client-go@v0.25.12-jaredallard.1/tools/cache/reflector.go:169: watch of *v1.Deployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 11; INTERNAL_ERROR; received from peer") has prevented the request from succeeding  caller="reflector.go:347"
INFO[0361] pkg/mod/github.com/jaredallard/client-go@v0.25.12-jaredallard.1/tools/cache/reflector.go:169: watch of *v1.StatefulSet ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 17; INTERNAL_ERROR; received from peer") has prevented the request from succeeding  caller="reflector.go:347"

[jaredallard]

Does the pod have any notable logs? It sounds like the SSH server inside of the pod never became happy, for whatever reason.

[echo-devnull]

No all the logging is in the original post I'm afraid. The pod itself did not create any logging.

[jaredallard]

That's... bizarre. Makes me think something went wrong with the image being downloaded. Can you try again but ensure that your nodes don't have an existing linuxserver/openssh-server image present? That should download the latest image and hopefully work 😓

[echo-devnull]

I've got the time to look into this again ;-)

So, the expose (reverse tunnel) pod that get's created does have a single log entry:

[mod-init] Attempting to run Docker Modification Logic

But does not move beyond that, or ever get to the ready status. According to describe the pod itself is running, but the Readiness probe failed. It is trying to connect to "2222" ? That seems wrong to me. Should it not be "22" ?

Nevermind, it seems to indeed be the default number: https://docs.linuxserver.io/images/docker-openssh-server

[jaredallard]

@echo-devnull Is there some sort of firewall being used? I believe that part of the linuxserver openssh image downloads a git repo at runtime, so I wonder if it's something related to that? Or, DNS lookup issues. Spitballing ideas. I haven't been able to repro this so I don't know what else to suggest 😭