ERROR AutoscalingRunnerSet Failed to get runner scale set from Actions service

[avgKol]

Checks

• [X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• [X] I am using charts that are officially provided

Controller Version

0.6.1

Deployment Method

Helm

Checks

• [X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
• [X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install Actions Runner Controller using Helm
2. Install/configure a runner scale set using Helm

Describe the bug

The controller pod logs this error and the listener pod doesn't get created:

ERROR AutoscalingRunnerSet Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorgcom/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

Describe the expected behavior

Listener pod should be created.

Additional Context

Enterprise GitHub

Controller Logs

2023-11-14T06:04:24Z    INFO    Update strategy set to: {"updateStrategy": "immediate"}
2023-11-14T06:04:24Z    INFO    AutoscalingListener image pull policy changed   {"ImagePullPolicy": "IfNotPresent"}
2023-11-14T06:04:24Z    INFO    AutoscalingListener logging parameters changed  {"LogLevel": "debug", "LogFormat": "text"}
2023-11-14T06:04:24Z    INFO    Registering scale set metrics
2023-11-14T06:04:24Z    INFO    starting manager
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.AutoscalingRunnerSet"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunnerSet"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.AutoscalingListener"}
2023-11-14T06:04:24Z    INFO    Starting Controller {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "source": "kind source: *v1alpha1.EphemeralRunner"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "source": "kind source: *v1.Pod"}
2023-11-14T06:04:24Z    INFO    Starting Controller {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1alpha1.AutoscalingListener"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.Pod"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.ServiceAccount"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.Role"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.RoleBinding"}
2023-11-14T06:04:24Z    INFO    Starting Controller {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunnerSet"}
2023-11-14T06:04:24Z    INFO    Starting EventSource    {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunner"}
2023-11-14T06:04:24Z    INFO    Starting Controller {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet"}
2023-11-14T06:04:24Z    INFO    Starting workers    {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "worker count": 1}
2023-11-14T06:04:24Z    INFO    Starting workers    {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "worker count": 1}
2023-11-14T06:04:24Z    INFO    Starting workers    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "worker count": 1}
2023-11-14T06:04:24Z    INFO    Starting workers    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "worker count": 1}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removed finalizer from GitHub secret    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Waiting for dependent resources to be deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    GitHub secret has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role binding    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removed finalizer from manager role binding {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Waiting for dependent resources to be deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    GitHub secret has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role binding    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Manager role binding has already been deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removed finalizer from manager role {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Waiting for dependent resources to be deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    GitHub secret has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role binding    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Manager role binding has already been deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Manager role has already been deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Removing finalizer  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:04:24Z    INFO    AutoscalingRunnerSet    Successfully removed finalizer after cleanup    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:00Z    INFO    AutoscalingRunnerSet    Adding finalizer    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:00Z    INFO    AutoscalingRunnerSet    Successfully added finalizer    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:00Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:00Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:00Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:00Z    INFO    actions-clients creating new client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:00Z    INFO    actions-clients successfully created new client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:00Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-14T06:06:00Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}
2023-11-14T06:06:15Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:06:15Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "033566de-92db-4448-8875-2470b1bd7704", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:06:15Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:15Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:15Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:15Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:15Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-14T06:06:15Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}
2023-11-14T06:06:30Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:06:30Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "f1ae7222-e389-4cd4-83b5-dbe51c68bdc6", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:06:30Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:30Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:30Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:30Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:30Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-14T06:06:30Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}
2023-11-14T06:06:46Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:06:46Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "3d20ad60-4956-46c9-9287-e484e6ef287a", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:06:46Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:46Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:06:46Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:46Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:06:46Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-14T06:06:46Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}
2023-11-14T06:07:01Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:07:01Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "e7a7420e-aa8e-45b1-bf7b-5853bf6283c0", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:07:01Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:07:01Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:07:01Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:07:01Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:07:01Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-14T06:07:01Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}
2023-11-14T06:07:16Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:07:16Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "1dc02df8-5266-4e78-9f02-326dfdc65159", "error": "failed to get runner registration token on refresh: Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": POST https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token giving up after 5 attempt(s): Post \"https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-14T06:07:16Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:07:16Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-14T06:07:16Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:07:16Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-14T06:07:16Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-14T06:07:16Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}

Runner Pod Logs

N/A

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[dond00m]

Seeing the same error under same conditions. I'll add that I tried using the TLS settings available in the AutoScalingRunnerSet spec but there is no way to provide the same data to the controller.

githubServerTLS:
  certificateFrom:
    configMapKeyRef:
      name: config-map-name
      key: ca.crt
  runnerMountPath: /usr/local/share/ca-certificates/

[dond00m]

@avgKol I was able to work this out. Looks like the TLS config in the runner set is the correct config to use. I created the PEM formatted value of our cert chain in a config map and was able to register with our Github instance. The key and format need to line up to the following - https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#custom-tls-certificates

[avgKol]

Thank you for your help . I successfully added a configmap in the arc-systems and arc-runners namespaces. However, I have encountered a new error.

2023-11-15T19:07:08Z    INFO    AutoscalingListener image pull policy changed   {"ImagePullPolicy": "IfNotPresent"}
2023-11-15T19:07:08Z    INFO    AutoscalingListener logging parameters changed  {"LogLevel": "debug", "LogFormat": "text"}
2023-11-15T19:07:08Z    INFO    Registering scale set metrics
2023-11-15T19:07:08Z    INFO    starting manager
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1alpha1.AutoscalingListener"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunnerSet"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "source": "kind source: *v1alpha1.EphemeralRunner"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.AutoscalingRunnerSet"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunnerSet"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.Pod"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.AutoscalingListener"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.ServiceAccount"}
2023-11-15T19:07:08Z    INFO    Starting Controller {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.Role"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.RoleBinding"}
2023-11-15T19:07:08Z    INFO    Starting Controller {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "source": "kind source: *v1.Pod"}
2023-11-15T19:07:08Z    INFO    Starting Controller {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner"}
2023-11-15T19:07:08Z    INFO    Starting EventSource    {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunner"}
2023-11-15T19:07:08Z    INFO    Starting Controller {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet"}
2023-11-15T19:07:08Z    INFO    Starting workers    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "worker count": 1}
2023-11-15T19:07:08Z    INFO    Starting workers    {"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "worker count": 1}
2023-11-15T19:07:08Z    INFO    Starting workers    {"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "worker count": 1}
2023-11-15T19:07:08Z    INFO    Starting workers    {"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "worker count": 1}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removed finalizer from GitHub secret    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Waiting for dependent resources to be deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    GitHub secret has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role binding    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removed finalizer from manager role binding {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Waiting for dependent resources to be deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    GitHub secret has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role binding    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Manager role binding has already been deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removed finalizer from manager role {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Waiting for dependent resources to be deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Deleting resources  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up the listener    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Listener is deleted {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Cleaning up ephemeral runner sets   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    All ephemeral runner sets are deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode service account    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-binding-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-role-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Skipping cleaning up kubernetes mode role binding   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "reason": "annotation key \"actions.github.com/cleanup-kubernetes-mode-service-account-name\" not present"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from no permission service account   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    No permission service account has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-no-permission"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from GitHub secret   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    GitHub secret has already been deleted  {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-github-secret"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role binding    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Manager role binding has already been deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer from manager role    {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Manager role has already been deleted   {"autoscalingrunnerset": "arc-runners/arc-runner-set", "name": "arc-runner-set-gha-rs-manager"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Removing finalizer  {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:07:08Z    INFO    AutoscalingRunnerSet    Successfully removed finalizer after cleanup    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:35Z    INFO    AutoscalingRunnerSet    Adding finalizer    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:35Z    INFO    AutoscalingRunnerSet    Successfully added finalizer    {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:35Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:35Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:35Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:35Z    INFO    actions-clients creating new client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:35Z    INFO    actions-clients successfully created new client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:35Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.myorg.com/Infrastructure/arc"}
2023-11-15T19:19:35Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://github.myorg.com/api/v3/repos/Infrastructure/arc/actions/runners/registration-token"}
2023-11-15T19:19:37Z    INFO    actions-clients getting Actions tenant URL and JWT  {"registrationURL": "https://github.myorg.com/api/v3/actions/runner-registration"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "d680b677-384c-4b3b-939d-fce733e5d9de", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "69316877-f778-4d43-9b1a-4eee39a77e6f", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "44138c72-dbc7-4061-9df0-b9aa454b6a46", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "489bfc28-12c3-4c7a-8d80-151dcab1d903", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "71f48ff5-5923-4448-a0d1-e69faff63a3f", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "0ba3c6d6-f1c3-4f1a-9f45-de0856f321ef", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:37Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:37Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:37Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "02124cc0-7bbc-4d23-a2bf-712d8c897c2a", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:38Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:38Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:38Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:38Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:38Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:38Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "54287065-01a3-417f-8b0b-32b044df2a15", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:38Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:38Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:38Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:38Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:39Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:39Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "d5123d12-a10b-490a-a401-421b2a4c11c5", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:40Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:40Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:40Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:40Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:40Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:40Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "ff866749-00ba-42e1-bb70-8d8b91af180b", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:42Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:42Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:42Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:42Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:42Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:42Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "61f101f9-1cf0-4549-8488-81bda8483b6d", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:48Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:48Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:48Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:48Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:48Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:48Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "1989839f-757b-40cf-9875-5a16b4fd1f32", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:58Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:58Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:19:58Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:58Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:19:58Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:19:58Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "5ffa053e-b8ee-486d-9d65-a945ec044360", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:20:18Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:20:18Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:20:18Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:20:18Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:20:18Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:20:18Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "3adef46c-1788-419e-81a8-b514ff632978", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:20:59Z    INFO    AutoscalingRunnerSet    Runner scale set id annotation does not exist. Creating a new runner scale set. {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:20:59Z    INFO    AutoscalingRunnerSet    Creating a new runner scale set {"autoscalingrunnerset": "arc-runners/arc-runner-set"}
2023-11-15T19:20:59Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:20:59Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.myorg.com/Infrastructure/arc", "namespace": "arc-runners"}
2023-11-15T19:21:00Z    ERROR   AutoscalingRunnerSet    Failed to get runner scale set from Actions service {"autoscalingrunnerset": "arc-runners/arc-runner-set", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:435
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
    github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-11-15T19:21:00Z    ERROR   Reconciler error    {"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "AutoscalingRunnerSet": {"name":"arc-runner-set","namespace":"arc-runners"}, "namespace": "arc-runners", "name": "arc-runner-set", "reconcileID": "7bc586eb-0832-4500-9d49-368f5d4438cd", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235```

[nikola-jokic]

Hey @avgKol,

What version of Enterprise Server are you using? Scale sets are available on 3.9+. Can you please provide values.yaml file as well? Please make sure to remove anything that should not be exposed :relaxed:

[nikola-jokic]

Closing this issue for now until we hear back from you.

[ameyp]

Is there a way to run the controller without a scale set? I'm okay with having a fixed number of runners rather than have them scale up and down.

[carsonmclean]

Is there a way to run the controller without a scale set? I'm okay with having a fixed number of runners rather than have them scale up and down.

Also curious on this! If we are still on GHES 3.7, I think we can only use the old summerwind ARC, correct?

[nikola-jokic]

@carsonmclean correct. APIs that are being used by scale sets are only available starting at GHES 3.9+