Runner Scale Set: "No gha-rs-controller deployment found" when rendering Helm chart

alecor191 commented 1 year ago

Checks

[X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
[X] I am using charts that are officially provided

Controller Version

0.6.1

Deployment Method

Helm

Checks

[X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
[X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Follow Configuring Runner Scale Set but instead of helm install use helm template

Describe the bug

Running helm template using the parameters defined in the Configuring Runner Scale Set doc fails as follows:

❯ helm template $installationName --namespace $namespace --set githubConfigUrl=$githubConfigUrl --set githubConfigSecret.github_token=$githubPat oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set:0.6.1

Digest: sha256:d0a4e067e15a2c616c6c2d049e98d9dc8e8aadb11ac6625cd01ee3ca30db8caa

Error: execution error at (gha-runner-scale-set/templates/manager_role_binding.yaml:17:11): No gha-rs-controller deployment found using label (app.kubernetes.io/part-of=gha-rs-controller). Consider setting controllerServiceAccount.name in values.yaml to be explicit if you think the discovery is wrong.

However, helm install succeeds (notice that I'm using the exact same parameters as above):

❯ helm install $installationName --namespace $namespace --set githubConfigUrl=$githubConfigUrl --set githubConfigSecret.github_token=$githubPat oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set:0.6.1
Digest: sha256:d0a4e067e15a2c616c6c2d049e98d9dc8e8aadb11ac6625cd01ee3ca30db8caa
NAME: arc-runner-set
LAST DEPLOYED: Fri Nov  3 14:16:58 2023
NAMESPACE: apps
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing gha-runner-scale-set.

Describe the expected behavior

helm template should succeed, like helm install does. It should be possible to render the chart without error.

Additional Context

The reason why this is a blocking issue for us, is that we don't run Helm directly, but we use a IaC framework (Pulumi) that runs Helm under the hood. Pulumi fails with the same error as mentioned above, essentially preventing us from using Runner Scale Sets in our IaC setup.

Only after some troubleshooting, we found that also helm template fails with the same error. I.e. it seems that the Helm chart performs some checks that don't work when it is not being installed on a cluster.

Controller Logs

N/A. Nothing is deployed to K8S, this is a local operation.

Runner Pod Logs

N/A. Nothing is deployed to K8S, this is a local operation.

github-actions[bot] commented 1 year ago

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

nikola-jokic commented 1 year ago

Hey @alecor191,

The error message hints at the problem. While running a template, you don't have a service account configured, and the helm chart tries to look it up. To avoid this issue, you can specify a service account inside values.yaml file

alecor191 commented 1 year ago

Thanks @nikola-jokic for your super-fast response!

I understand that I can set it explicitly (I did and it worked fine). However, shouldn't helm template work if helm install works without requiring additional settings?

nikola-jokic commented 1 year ago

No problem! :relaxed: I think it is well summarized in this comment. You can try using --dry-run in order to allow lookup to be executed.

DPatrickBoyd commented 1 year ago

Thanks @nikola-jokic for your super-fast response!

I understand that I can set it explicitly (I did and it worked fine). However, shouldn't helm template work if helm install works without requiring additional settings?

@alecor191 were you able to do this in a values.yaml file or were you manually adding the controllersServiceAccount name to the helm template step? I am running into something similar with argoCD

controllerServiceAccount:
  namespace: arc-system
  name: arc-system-gha-rs-controller

setting my values.yaml to something like this didn't fix it.

alecor191 commented 1 year ago

@DPatrickBoyd yes, I was able to add the controllerServiceAccount section to the values file passed to the Helm command. I noticed that the controller Helm chart creates a Service Account named actions-runner-controller-gha-rs-controller in the namespace where I installed the controller. It also has the label app.kubernetes.io/part-of=gha-rs-controller set, that helm template complained about not being able to find. So I just used those values:

controllerServiceAccount:
  namespace: <controller-namespace>
  name: actions-runner-controller-gha-rs-controller

mahmud2011 commented 11 months ago

@DPatrickBoyd yes, I was able to add the controllerServiceAccount section to the values file passed to the Helm command. I noticed that the controller Helm chart creates a Service Account named actions-runner-controller-gha-rs-controller in the namespace where I installed the controller. It also has the label app.kubernetes.io/part-of=gha-rs-controller set, that helm template complained about not being able to find. So I just used those values:
controllerServiceAccount:
  namespace: <controller-namespace>
  name: actions-runner-controller-gha-rs-controller

@alecor191 thanks. It resolved my issue. But I am currently facing one problem with the secret. My secret is in arc-runners namespace. If I deploy directly with the helm install following the documentation It's working fine. But as I am trying to deploy with argo I am getting following error:

...
 "logger":"AutoscalingRunnerSet",
 "message":"Failed to initialize Actions service client for creating a new runner scale set",
 "autoscalingrunnerset":
   {"name":"runner-scale-set-app",
    "namespace":"arc-runners"},
 "error":"failed to find GitHub config 
 secret: secrets \"gihtub-token\" is forbidden: User \"system:serviceaccount:arc-systems:runner-scale-set-controller-app-gha-rs-controller\" 
 cannot get resource \"secrets\" in API group \"\" in the namespace \"arc-runners\"",
 ...

nikola-jokic commented 11 months ago

Hey @mahmud2011,

Do you have a finalizer on your service account (actions.github.com/cleanup-protection). It does seem to me that we should either patch the service account provided to have the finalizer, or document it in the values.yaml. Please, let us know if that solves your problem :relaxed:

mahmud2011 commented 11 months ago

Hey @mahmud2011,

Do you have a finalizer on your service account (actions.github.com/cleanup-protection). It does seem to me that we should either patch the service account provided to have the finalizer, or document it in the values.yaml. Please, let us know if that solves your problem ☺️

No, I didn't have the finalizer. I have manually added it but still the same.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: ...
  creationTimestamp: '2023-12-07T13:23:25Z'
  finalizers:
    - actions.github.com/cleanup-protection
  labels:
    app.kubernetes.io/instance: runner-scale-set-controller-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gha-rs-controller
    app.kubernetes.io/namespace: arc-systems
    app.kubernetes.io/part-of: gha-rs-controller
    app.kubernetes.io/version: 0.7.0
    argocd.argoproj.io/instance: runner-scale-set-controller-app
    helm.sh/chart: gha-rs-controller-0.7.0
  name: runner-scale-set-controller-app-gha-rs-controller
  namespace: arc-systems
  resourceVersion: '1309739725'
  uid: f0edb4bf-3d2c-46a3-8599-a650c44b5622

log:

{"severity":"info","ts":"2023-12-07T13:26:11Z","logger":"AutoscalingRunnerSet","message":"Creating a new runner scale set","autoscalingrunnerset":{"name":"runner-scale-set-app","namespace":"arc-runners"}}

{"severity":"error","ts":"2023-12-07T13:26:11Z","logger":"AutoscalingRunnerSet","message":"Failed to initialize Actions service client for creating a new runner scale set","autoscalingrunnerset":{"name":"runner-scale-set-app","namespace":"arc-runners"},"error":"failed to find GitHub config secret: secrets \"gihtub-token\" is forbidden: User \"system:serviceaccount:arc-systems:runner-scale-set-controller-app-gha-rs-controller\" cannot get resource \"secrets\" in API group \"\" in the namespace \"arc-runners\"","stacktrace":...}

{"severity":"error","ts":"2023-12-07T13:26:11Z","message":"Reconciler error","controller":"autoscalingrunnerset","controllerGroup":"actions.github.com","controllerKind":"AutoscalingRunnerSet","AutoscalingRunnerSet":{"name":"runner-scale-set-app","namespace":"arc-runners"},"namespace":"arc-runners","name":"runner-scale-set-app","reconcileID":"93e680b9-83e4-4e16-9518-da3140e5141c","error":"failed to find GitHub config secret: secrets \"gihtub-token\" is forbidden: User \"system:serviceaccount:arc-systems:runner-scale-set-controller-app-gha-rs-controller\" cannot get resource \"secrets\" in API group \"\" in the namespace \"arc-runners\"","stacktrace":...}

nikola-jokic commented 11 months ago

After adding a finalizer, have you tried re-installing the same scale set, and after it is completely up, deleting it?

mahmud2011 commented 10 months ago

The issue was not related to finalizer. Just added the actions.github.com in apiGroups of my rbac. Resolved.

mconigliaro commented 10 months ago

I'm having the same problem when deploying with Argo CD. I think it must be related to the fact that instead of a helm install, Argo CD renders all the manifests and applies them its own way. Anyway, setting the controllerServiceAccount options got the installation to work, but now I'm seeing this same error about accessing secrets:

2024-01-05T23:53:24Z ERROR Reconciler error {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-doop","namespace":"arc-systems"}, "namespace": "arc-systems", "name": "arc-doop", "reconcileID": "84ec6789-7f67-430f-98aa-df8f35f18b7a", "error": "failed to find GitHub config secret: secrets \"arc-doop-gha-rs-github-secret\" is forbidden: User \"system:serviceaccount:arc-runners:arc-gha-rs-controller\" cannot get resource \"secrets\" in API group \"\" in the namespace \"arc-systems\""}

What/where is this argocd-application-controller.yaml @mahmud2011 is talking about?

Update: OK, I think I fixed my issue:

controllerServiceAccount:
  namespace: <namespace of controller>
  name: <release name of controller>-gha-rs-controller

rakeshramakrishnan-hbk commented 9 months ago

I am also facing the same issue with argocd, where it does a helm template and then applies. However, setting the controllerServiceAccount helm value did not help. Argocd seems to apply the helm chart without any values first, and then would apply the values file - so it is stuck at the first part of finding the controller.

mahmud2011 commented 9 months ago

I am also facing the same issue with argocd, where it does a helm template and then applies. However, setting the controllerServiceAccount helm value did not help. Argocd seems to apply the helm chart without any values first, and then would apply the values file - so it is stuck at the first part of finding the controller.

Did you check the service account name and it's namespace?

object88 commented 7 months ago

Just to chime in, this also happens with helm diff. Adding the controllerServiceAccount fixes it. It seems like this should be a required set of properties, rather than optional.

AurimasNav commented 1 month ago

Just to chime in, this also happens with helm diff. Adding the controllerServiceAccount fixes it. It seems like this should be a required set of properties, rather than optional.

I agree, it seem like this was setup with the assumption that controller and scale set will bet set-up in the same namespace, but given the recommendation to use separate namespaces this parameter should be mandatory.

Came here for solution for failing argocd deployment.

actions / actions-runner-controller