Closed guillaumevillemont closed 9 months ago
Hello! Thank you for filing an issue.
The maintainers will triage your issue shortly.
In the meantime, please take a look at the troubleshooting guide for bug reports.
If this is a feature request, please review our contribution guidelines.
Hey @guillaumevillemont,
Thank you for detailed description of the problem! Let's transfer this issue to the container hook repo because it is a more appropriate place. The ARC is responsible up to the point of spinning up the runner, and the actual hook implementation can be modified in a way to achieve your goal :relaxed:
we are also experiencing the exact same issues, on our side it seems we are facing CPU issues rather than memory issues. Do you have a workaround?
Closing since the new release is out :relaxed:
Closing since the new release is out ☺️
If I understand correctly the fix is only applicable when using a single node cluster or "readwritemany", which are a very rare situation. In EKS "readwritemany" is not supported, and our cluster is not single node.
Is there something im missing?
You are right, the fix is not general. The issue is that the runner needs to pass assets using volumes to the hook in order for a job to be executed. The way it is done is by using volumes. If you only have a ReadWriteOnce
volume, that means that the job pod needs to land on the same node where the runner is. There is a great proposal from @Wielewout here to lean more on the scheduler by "suggesting" the node where the runner is by using labels. That will still mean that job pods are scheduled on the same node where the runner is, but it will be in a pending state until there are enough resources for the job pod to start. But the downside is that your job may execute much longer than it needs to execute.
So until the solution is accepted, if you only have ReadWriteOnce
volumes, unfortunately, you would have to find the solution that fits your cluster configuration...
Checks
Controller Version
0.6.1
Deployment Method
Helm
Checks
To Reproduce
Runner template
```yaml template: metadata: labels: app: myarc spec: initContainers: - name: init-k8s-volume-permissions image: ghcr.io/actions/actions-runner:latest command: ["/bin/sh", "-c"] args: - | sudo chown -R 1001:123 /home/runner/_work volumeMounts: - name: work mountPath: /home/runner/_work containers: - name: runner image: ghcr.io/actions/actions-runner:latest command: ["/home/runner/run.sh"] env: - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER value: "true" - name: ACTIONS_RUNNER_CONTAINER_HOOKS value: /home/runner/k8s/index.js - name: ACTIONS_RUNNER_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE value: /home/runner/pod-templates/default.yaml securityContext: runAsUser: 1001 runAsGroup: 123 resources: requests: cpu: 500m memory: 512Mi limits: memory: 512Mi volumeMounts: - name: work mountPath: /home/runner/_work - name: pod-templates mountPath: /home/runner/pod-templates readOnly: true volumes: - name: pod-templates configMap: name: pod-templates ```Setting
minRunners: 0
andmaxRunners: 5
also help highlight this issue in my example.Make sure you are using an autoscaling nodepool (that can go to very few or even zero nodes)
Make sure the runner has large memory/cpu requests, for the workflow pod (using PodTemplate)
PodTemplate ConfigMap
```yaml apiVersion: v1 kind: ConfigMap metadata: name: pod-templates namespace: gh-arc data: default.yaml: | --- apiVersion: v1 kind: PodTemplate metadata: name: runner-pod-template namespace: gh-arc labels: app: runner-pod-template spec: securityContext: runAsUser: 1001 runAsGroup: 123 containers: - name: $job resources: requests: cpu: 1000m memory: 8Gi limits: memory: 8Gi ```strategy:
to make sure they all spawn nearly at the same time), andcontainers:
to make sure it creates a-workflow
pod.Actions CI
```yaml name: GitHub Actions Test run-name: Test on: [push] jobs: foo: runs-on: myarc container: debian strategy: matrix: package: - 'common' - 'utils' - 'ui' - 'billing' steps: - run: echo "Running for ${{ matrix.package }}" - name: Check out repository code uses: actions/checkout@v3 - run: sleep 300 ```Describe the bug
The rs-controller receives the pipeline request and scale up the runnerset to 4 runners.
Each runner pods get scheduled to one node. Since they only require 500Mi memory each, they all fit in that 16Go memory node.
After initializing, they each spawn a
-workflow
pod next to them. Kubernetes now tries to schedule 4 pod that each requires 8Go memory on that same single node. It fails due to OOM.On Github Actions you can see your job failed at the "Initialize containers" step, with :
Describe the expected behavior
The
-workflow
pod should be scheduled by kube-scheduler on a different node.I think the container-hook does not rely on kube-scheduler and all workflow pods are necessarily spawn next to its runner pod. Maybe due to a constraint I haven't seen (volumes maybe?)
I've seen a lot of people setting resources requests to the container but I'm failing to see how that can solve my issue since the job is actually run on the workflow pod.
Additional Context
All configs are in the reproduction steps
We are mostly trying to reduce our cost on kubernetes nodes and don't find acceptable to have a few large nodes idle, in case a CI pipeline get triggers. Hence the requirement to scale the nodepool to zero and set runner minimum to zero too.
We also provide multiple runner by size, so our developpers can juste pick the desirable runner size with tags like
memory-xl
,cpu-s
, etc.Controller Logs
Runner Pod Logs