One of the job steps hangs and fail without any log messages

omerlh commented 1 year ago

Checks

[X] I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
[X] I'm not using a custom entrypoint in my runner image

Controller Version

v0.27.0

Helm Chart Version

0.22.0

CertManager Version

v1.9.1

Deployment Method

Helm

cert-manager installation

Using the official helm chart - https://charts.jetstack.io

Checks

[X] This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
[X] I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
[X] My actions-runner-controller version (v0.x.y) does support the feature
[X] I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
[X] I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: default-runner
spec:
  template:
    spec:
      organization: goledge
      image: <our custom image, see dockerfile bellow>
      dockerdWithinRunnerContainer: true
      labels:
        - self-hosted
        - linux
      priorityClassName: low

FROM summerwind/actions-runner-dind@sha256:7c7a5dd8839644de4149da9150a19910930928bc9651ee40eec904567ae348ad

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - \
  && echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list \
  && sudo apt-get install ca-certificates curl gnupg lsb-release \
  && sudo mkdir -p /etc/apt/keyrings \
  && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list \
  && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
  && sudo apt update && sudo apt install --no-install-recommends yarn docker-compose-plugin default-jdk amazon-ecr-credential-helper -y \
  && sudo apt-get install -y \
  && java -version \
  && wget https://github.com/bazelbuild/bazelisk/releases/download/v1.14.0/bazelisk-linux-amd64 -O /tmp/bazelisk && sudo mv /tmp/bazelisk /usr/local/bin/bazelisk && sudo chmod +x /usr/local/bin/bazelisk && sudo ln -s /usr/local/bin/bazelisk /usr/local/bin/bazel  \
  && bazelisk --version && bazel --version


### To Reproduce

```markdown
I am not sure how TBH

Describe the bug

Recently we noticed job failing without any clear indication about the why:

This is the hang, it usually fails after job timeout time. There are no logs anywhere, so I have no idea why it is failing

Describe the expected behavior

Job runs without failures and not hanging

Whole Controller Logs

https://gist.github.com/omerlh/0e958a197c0fc4ea1be83814fb17f444

Whole Runner Pod Logs

https://gist.github.com/omerlh/0e958a197c0fc4ea1be83814fb17f444

Additional Context

No response

github-actions[bot] commented 1 year ago

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

omerlh commented 1 year ago

We suspect this might be related to Karpenter removing a node when the runner pod is running on, and lack of PDB does not block this operation. I saw the chart does not have one, maybe worth adding?

actions / actions-runner-controller