actions / actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runners
Apache License 2.0
4.57k stars 1.08k forks source link

Ability to customize the dind image in gha-runner-scale-set #3709

Open victor-cetinic opened 1 month ago

victor-cetinic commented 1 month ago

What would you like added?

I have been trying to customise the gha-runner-scale-set so that i can set the docker registry mirror globally for all the runners using the Docker DIND approach.

Most instructions point to modifying the registry settings of the docker service by adding a command line like : --registry-mirror=https://mirror.gcr.io (further options - https://cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images )

The issue im facing is that these need to be configured on the dind container, and not the runner container., and when i created a template like this


template:
  spec: 
    nodeSelector:
      cloud.google.com/gke-preemptible: "true"
    initContainers:
      - name: dockerconfigwriter
        image: alpine
        command:
          - sh
          - -c
          - |
            echo "{\"registry-mirrors\": [ \"https://mirror.gcr.io\"] }" > /home/runner/.docker/daemon.json
        volumeMounts:
          - name: docker-config-volume
            mountPath: /home/runner/.docker
    containers:
      - name: runner
        command:
          - /home/runner/run.sh
        image:
          ghcr.io/actions/actions-runner:latest
        env:
          - name: DOCKER_HOST
            value: unix:///var/run/docker.sock
          - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
            value: \"120\"
        resources:
          limits:
            cpu: 6
            memory: 5Gi
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: docker-config-volume
            mountPath: /etc/docker
      - name: dind
        image: docker:dind
        args:
          - dockerd
          - --host=unix:///var/run/docker.sock
          - --group=$(DOCKER_GROUP_GID)
          - --registry-mirror=https://mirror.gcr.io
        env:
          - name: DOCKER_GROUP_GID
            value: \"123\"
        securityContext:
          privileged: true
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: dind-externals
            mountPath: /home/runner/externals
          - name: docker-config-volume
            mountPath: /etc/docker
    volumes:
      - name: docker-config-volume
        emptyDir: { }

I expected to be able to customise the dind container, however, it does not get included in the template, it simply uses the default pre-defined in the template, so none of these customisations are considered in the final template, consequently , the settings are not applied.

Output from dry run

Below you can see the final template, that contains the customisations i made to the runner, however, the dind container is ignored so my mirror registry settings are not applied.

  containers:
      - name: runner

        command: 
          - /home/runner/run.sh
        image: 
          ghcr.io/actions/actions-runner:latest
        resources: 
          limits:
            cpu: 6
            memory: 5Gi
        env:
          - 
            name: DOCKER_HOST
            value: unix:///var/run/docker.sock
          - 
            name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
            value: \"120f\"
        volumeMounts:
          - 
            mountPath: /home/runner/_work
            name: work
          - 
            mountPath: /var/run
            name: dind-sock
          - 
            mountPath: /etc/docker
            name: docker-config-volume
      - name: dind
        image: docker:dind
        args:
          - dockerd
          - --host=unix:///var/run/docker.sock
          - --group=$(DOCKER_GROUP_GID)
        env:
          - name: DOCKER_GROUP_GID
            value: "123"
        securityContext:
          privileged: true
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: dind-externals
            mountPath: /home/runner/externals

Why is this needed?

The issue for me is I hit limits going to docker registry, and having a single way to do this for all runners, makes more sense and would apply for all rather than each workflow having to do something

Additional context

Add any other context or screenshots about the feature request here.

github-actions[bot] commented 1 month ago

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

Alpenovsky commented 1 week ago

I had the same problem. I use FluxCD to create Helm release and I used the postRenderers functionality to overwrite (as a workaround) the dind image or dind args:


apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: dev-runners
spec:
  releaseName: dev-runners
  chartRef:
    kind: OCIRepository
    name: dev-runners
  postRenderers:
    - kustomize:
        patches:
          - target:
              kind: AutoscalingRunnerSet
              name: dev-runners
            patch: |
              - op: replace
                path: /spec/template/spec/containers/1/image
                value: 24.0.9-dind
[...]