`gha-runner-scale-set` chart comment about `containerMode` is wrong and misleading

DaazKu commented 5 months ago

Checks

[X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
[X] I am using charts that are officially provided

Controller Version

0.9.1

Deployment Method

Helm

Checks

[X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
[X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

- Read these lines: https://github.com/actions/actions-runner-controller/blob/49490c4421aa8d58a8eb375fe7a539bdfe28b7a6/charts/gha-runner-scale-set/values.yaml#L76-L77
- Specify `template.spec` while keeping `containerMode.type=kubernetes`
- Everything works

Describe the bug

If any customization is required for dind or kubernetes mode, containerMode should remain empty, and configuration should be applied to the template.

This is plain not true.

If you want kubernetes mode you have to specify containerMode.type=kubernetes otherwise you won't get the RBAC setup for you. You are also pretty much forced to customize the template on EKS with because otherwise things do not work by default. (See: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied) Or if you want to add other environment variables to the runner..

Going into the chart you also see that the template is honoured even if containerMode.type is set.

Describe the expected behavior

Proper comment that is not misleading. Link to relevant documentation if necessary.

Additional Context

values.yml that works in EKS for containerMode.type=kubernetes

    controllerServiceAccount:
      namespace: ...
      name: ...
    runnerScaleSetName: ...
    githubConfigUrl: ...
    maxRunners: 1
    containerMode:
      type: kubernetes
      kubernetesModeWorkVolumeClaim:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "ephemeral-storage"
        resources:
          requests:
            storage: 5Gi
    template:
      spec:
        # See https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied
        securityContext:
          fsGroup: 123
        containers:
          - name: runner
            image: ghcr.io/actions/actions-runner:latest
            command: ["/home/runner/run.sh"]

Controller Logs

N/A

Runner Pod Logs

N/A

github-actions[bot] commented 5 months ago

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

hlascelles commented 5 months ago

Agree, the documentation is wrong for kubernetes on that front. containerMode.type: kubernetes needs to be set.

We also had to add this:

          # We must add an init container to change the ownership of the _work directory
          # https://docs.github.com/en/enterprise-server@3.9/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied
          initContainers:
          - name: kube-init
            image: ghcr.io/actions/actions-runner:latest
            command: ["sudo", "chown", "-R", "1001:123", "/home/runner/_work"]
            volumeMounts:
            - name: work
              mountPath: /home/runner/_work

A working setup for EKS

For those that follow and want to run GitHub Actions Runners on self hosted EKS with your own custom image as the runner, this worked for us. You must prepare the Secret and install two charts.

# Chart: git@github.com:actions/actions-runner-controller
# git ref: 4357525445b0b77388af4e1f171b5b7bd9b116a4
# Path: charts/gha-runner-scale-set-controller
    values:
      namespace: github

# Chart: git@github.com:actions/actions-runner-controller
# git ref: 4357525445b0b77388af4e1f171b5b7bd9b116a4
# Path: charts/gha-runner-scale-set
    values:
      githubConfigUrl: https://github.com/myorg/myrepo
      # Create this k8s Secret and put the three values in it with keys: `github_app_id`, `github_app_installation_id`, `github_app_private_key`.
      githubConfigSecret: github-actions-runner-scale-set-secret
      controllerServiceAccount:
        namespace: github
        # Name must line up with the above chart release name eg `github-actions-scale-set-controller`. Install the above chart and see what SA name it makes.
        name: github-actions-scale-set-controller-gha-rs-controller
      minRunners: 1
      maxRunners: 4
      containerMode:
        # Needed, even if the docs say it isn't.
        type: kubernetes
        # Here is how your make the runner pods have a custom IAM Role, so they can (eg) contact real resources in you AWS account.
        kubernetesModeServiceAccount:
          annotations:
            # https://github.com/actions/actions-runner-controller/blob/98854ef9c018141d7386657322da351e11029da2/charts/gha-runner-scale-set/tests/values_kubernetes_mode_service_account_annotations.yaml#L4
            eks.amazonaws.com/role-arn: arn:aws:iam::XXXXX:role/my-ci-role
      template:
        spec:
          # We must add an init container to change the ownership of the _work directory
          # https://docs.github.com/en/enterprise-server@3.9/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied
          initContainers:
          - name: kube-init
            image: ghcr.io/actions/actions-runner:latest
            command: ["sudo", "chown", "-R", "1001:123", "/home/runner/_work"]
            volumeMounts:
            - name: work
              mountPath: /home/runner/_work

          # We have to fully override the containers simply to set our own "image"
          containers:
          - name: runner
            # This image is used as the runner image.
            # Note it cannot be "your image `FROM ubunutu` or similar, it must be based off the one in https://github.com/actions/runner/blob/main/images/Dockerfile
            # Or you can build your own and try and include all the items from that build.
            image: "XXXXXX.dkr.ecr.eu-west-1.amazonaws.com/my-image:123456789"
            command:
            - /home/runner/run.sh
            env:
            - name: ACTIONS_RUNNER_CONTAINER_HOOKS
              value: /home/runner/k8s/index.js
            - name: ACTIONS_RUNNER_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
              value: "false"
            volumeMounts:
            - name: work
              mountPath: /home/runner/_work
          volumes:
          - name: work
            ephemeral:
              volumeClaimTemplate:
                spec:
                  accessModes: [ "ReadWriteOnce" ]
                  # Critical change here compared to the docs. EKS does not support "local-storage" by default.
                  storageClassName: "gp2"
                  resources:
                    requests:
                      storage: 25Gi

The above solved errors we were seeing like:

Error: HttpError: HTTP request failed
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

noamgreen commented 2 months ago

HI , did you found workaround to this issue "An error occurred (AccessDenied) when calling"

cuse you cant use your roles and you must use the new brilliant way the pods are cant use the IRSA and can get aws access ... i cant assume role , we all knew for assume role we need to have role ( pre-define) that have sts approve ...

:-(

hlascelles commented 2 months ago

Sorry, no, we never saw that error!

actions / actions-runner-controller