All runners stuck in "Init"

[patrickblackjr]

Checks

• [X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• [X] I am using charts that are officially provided

Controller Version

0.8.3

Deployment Method

ArgoCD

Checks

• [X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
• [X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Unsure of how to reproduce as this seemed to occur randomly without any changes from my end.

Describe the bug

I have dozens of pods stuck in "Init" status for multiple hours at a time. No errors in either controller pods logs.

Describe the expected behavior

Run as normal

Additional Context

githubConfigUrl: mygithub
        githubConfigSecret: github-arc-secret
        controllerServiceAccount:
          namespace: arc
          name: gha-arc-controller-gha-rs-controller
        template:
          spec:
            initContainers:
              - name: init-dind-externals
                image: us-docker.pkg.dev/custom:latest
                command:
                  [
                    "cp",
                    "-r",
                    "-v",
                    "/home/runner/externals/.",
                    "/home/runner/tmpDir/",
                  ]
                volumeMounts:
                  - name: dind-externals
                    mountPath: /home/runner/tmpDir
            containers:
              - name: runner
                image: us-docker.pkg.dev/custom:latest
                command: ["/home/runner/run.sh"]
                env:
                  - name: DOCKER_HOST
                    value: unix:///run/docker/docker.sock
                volumeMounts:
                  - name: work
                    mountPath: /home/runner/_work
                  - name: dind-sock
                    mountPath: /run/docker
                    readOnly: true
              - name: dind
                image: docker:24.0.9-dind@sha256:51a2a0a9f8d15d455584ea054a602ea58917a78c34df2261b0695a60f0a6ae61
                # FIX: https://github.com/actions/actions-runner-controller/issues/3159
                # CAUSE: https://github.com/docker-library/docker/issues/463
                args:
                  - dockerd
                  - --host=unix:///run/docker/docker.sock
                  - --group=$(DOCKER_GROUP_GID)
                env:
                  - name: DOCKER_GROUP_GID
                    value: "123"
                  - name: DOCKER_IPTABLES_LEGACY
                    value: "1"
                securityContext:
                  privileged: true
                volumeMounts:
                  - name: work
                    mountPath: /home/runner/_work
                  - name: dind-sock
                    mountPath: /run/docker
                  - name: dind-externals
                    mountPath: /home/runner/externals
                  - name: daemon-json
                    mountPath: /etc/docker/daemon.json
                    subPath: daemon.json
                    readOnly: true
            volumes:
              - name: work
                emptyDir: {}
              - name: dind-sock
                emptyDir: {}
              - name: dind-externals
                emptyDir: {}
              - name: daemon-json
                configMap:
                  name: docker-daemon-config

Controller Logs

https://gist.github.com/patrickblackjr/c8894d94dbfd797585535108df07c04b

Runner Pod Logs

https://gist.github.com/patrickblackjr/c8894d94dbfd797585535108df07c04b

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @patrickblackjr, could you please run kubectl describe on them to see what is the problem?