gha-runner-scale-set-controller - helm chart doesn't support specifying resource limits

mikeclayton commented 1 year ago

I'm very excited about the Preview release of the GHA Runner Scale Set Controller and I'm currently trying to configure it on our DEV AKS cluster. I've hit a small speed-bump that I figured I'd report back though. I've got a workaround for the time being, but it would be great to get a fix rolled into the base package...

What would you like added?

At present there doesn't appear to be a way to specify resource limits for the ReplicaSet in the gha-runner-scale-set-controller Helm chart.

This is currently preventing us from deploying to an Azure Kubernetes Cluster because we're using the built-in "ASC Default" Policy with the "Containers: CPU and memory limits should be enforced" parameter set to "deny".

The result is that trying to install the helm chart fails, which this event:

> kubectl describe replicaset gha-runner-scale-set-controller -n my-namespace

... snip ...
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                From                   Message
  ----     ------        ----               ----                   -------
  Warning  FailedCreate  87s (x6 over 84m)  replicaset-controller  Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-2875548f79e717d03827] container <manager> has no resource limits

Specifically, the error message "Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits] container has no resource limits"

A workaround is to allow-list the namespace containing the ReplicaSet, but ideally resource limits this would be configurable in the Helm chart "values.yaml" file.

Why is this needed?

To provide built-in compatibility with corporate governance of Azure Kubernetes Cluster instances that have "ASC Default" policies set to deny creation of ReplicaSets with no resource limits configured.

Additional context

Here's a screenshot of the parameters that can be used to add the GHA Runner Scale Set Controller to an allow list when the "deny" policy is configured, but this may not be ideal / allowed in some environments...

github-actions[bot] commented 1 year ago

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

mikeclayton commented 1 year ago

Actually, I've just found the

template:
  spec:
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]

in the values.yaml - I'm presuming that'll be the place to add the resource limits.

I'll close this issue for now and open another one if I still can't work it out :-).

mikeclayton commented 1 year ago

I've had a chance to properly look at the helm charts now, and they don't seem to support specifying the resource limits. Ideally, whe I'd like to be able to do is something like this in the controller and runner values.yaml:

    spec:
      containers:
      - name: ...
        resources:
          limits:
            memory: 256Mi
            cpu: 100m

I think I can patch this with tools like kustomize before it gets deployed to the cluster, but it would be nice if it were supported in the helm chart...

mikeclayton commented 1 year ago

Right, never mind - found it for real this time...

https://github.com/actions/actions-runner-controller/blob/91c8991835016f8c6568f101d4a28185baec3dcc/charts/gha-runner-scale-set-controller/values.yaml#L55

Closing this issue (again)...

actions / actions-runner-controller