search

actions / actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runners
Apache License 2.0
4.41k stars 1.04k forks source link

Can we configure default ghcr.io/actions/actions-runner:latest with the custom one that is equivalent to GitHub hosted runner #3433

Closed gpadmesh closed 2 months ago

gpadmesh commented 2 months ago

Checks

Controller Version

0.6.1

Deployment Method

Helm

Checks

To Reproduce

We deployed arc system and runner scale set

helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller 

helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

This install works like a charm but the problem is the base image has minimum packages but nowhere compared to GitHub-hosted runner docker images.

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]

We tried composing a docker image from this https://github.com/actions/actions-runner-controller/blob/master/runner/actions-runner-dind.ubuntu-22.04.dockerfile or any other ubuntu dockerfile. The docker image completes the build successfully. However, when we specify in the values.yaml the docker image the docker container starts but we have this error message

Defaulted container "runner" out of: runner, dind, init-dind-externals (init) Waiting for docker to be ready.

We tried with no dind but the container is unable to spin up

ubuntu-latest-tjmzl-runner-55vld 0/1 ContainerCreating 0 2m6s


### Describe the bug

We expect on own docker ubuntu image would work with ARC

### Describe the expected behavior

We expect on own docker ubuntu image would work with ARC

### Additional Context

```yaml
## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://github.com/myorg"

## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token
githubConfigSecret: 
  ### GitHub Apps Configuration
  ## NOTE: IDs MUST be strings, use quotes
  #github_app_id: ""
  #github_app_installation_id: ""
  #github_app_private_key: |

  ### GitHub PAT Configuration
  github_token: "my_token"
## If you have a pre-define Kubernetes secret in the same namespace the gha-runner-scale-set is going to deploy,
## you can also reference it via `githubConfigSecret: pre-defined-secret`.
## You need to make sure your predefined secret has all the required secret data set properly.
##   For a pre-defined secret using GitHub PAT, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_token='ghp_your_pat'
##   For a pre-defined secret using GitHub App, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_app_id=123456 --from-literal=github_app_installation_id=654321 --from-literal=github_app_private_key='-----BEGIN CERTIFICATE-----*******'
# githubConfigSecret: pre-defined-secret

## proxy can be used to define proxy settings that will be used by the
## controller, the listener and the runner of this scale set.
#
# proxy:
#   http:
#     url: http://proxy.com:1234
#     credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
#   https:
#     url: http://proxy.com:1234
#     credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
#   noProxy:
#     - example.com
#     - example.org

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
#maxRunners: 5

## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
minRunners: 1

# runnerGroup: "default"
runnerGroup: "test"

## name of the runner scale set to create.  Defaults to the helm release name
# runnerScaleSetName: ""

## A self-signed CA certificate for communication with the GitHub server can be
## provided using a config map key selector. If `runnerMountPath` is set, for
## each runner pod ARC will:
## - create a `github-server-tls-cert` volume containing the certificate
##   specified in `certificateFrom`
## - mount that volume on path `runnerMountPath`/{certificate name}
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
##   2.303.0 this will instruct the runner to reload certificates on the host)
##
## If any of the above had already been set by the user in the runner pod
## template, ARC will observe those and not overwrite them.
## Example configuration:
#
# githubServerTLS:
#   certificateFrom:
#     configMapKeyRef:
#       name: config-map-name
#       key: ca.crt
#   runnerMountPath: /usr/local/share/ca-certificates/

## Container mode is an object that provides out-of-box configuration
## for dind and kubernetes mode. Template will be modified as documented under the
## template object.
##
## If any customization is required for dind or kubernetes mode, containerMode should remain
## empty, and configuration should be applied to the template.
 containerMode:
   type: "dind"
# containerMode:
#   type: "dind"  ## type can be set to dind or kubernetes
#   ## the following is required when containerMode.type=kubernetes
#   kubernetesModeWorkVolumeClaim:
#     accessModes: ["ReadWriteOnce"]
#     # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
#     storageClassName: "dynamic-blob-storage"
#     resources:
#       requests:
#         storage: 1Gi
#   kubernetesModeServiceAccount:
#     annotations:

## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
#   spec:
#     containers:
#     # Use this section to append additional configuration to the listener container.
#     # If you change the name of the container, the configuration will not be applied to the listener,
#     # and it will be treated as a side-car container.
#     - name: listener
#       securityContext:
#         runAsUser: 1000
#     # Use this section to add the configuration of a side-car container.
#     # Comment it out or remove it if you don't need it.
#     # Spec for this container will be applied as is without any modifications.
#     - name: side-car
#       image: example-sidecar

## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
  ## template.spec will be modified if you change the container mode
  ## with containerMode.type=dind, we will populate the template.spec with following pod spec
  ######################################################################################################
  ## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
  ## template:
  ##   spec:
  ##     containers:
  ##     - name: runner
  ##       image: ghcr.io/actions/actions-runner:latest
  ##       command: ["/home/runner/run.sh"]
  ##       env:
  ##         - name: ACTIONS_RUNNER_CONTAINER_HOOKS
  ##           value: /home/runner/k8s/index.js
  ##         - name: ACTIONS_RUNNER_POD_NAME
  ##           valueFrom:
  ##             fieldRef:
  ##               fieldPath: metadata.name
  ##         - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
  ##           value: "true"
  ##       volumeMounts:
  ##         - name: work
  ##           mountPath: /home/runner/_work
  ##     volumes:
  ##       - name: work
  ##         ephemeral:
  ##           volumeClaimTemplate:
  ##             spec:
  ##               accessModes: [ "ReadWriteOnce" ]
  ##               storageClassName: "local-path"
  ##               resources:
  ##                 requests:
  ##                   storage: 1Gi
   spec:
     containers:
       - name: runner
         image: ghcr.io/corunner-ubuntu22:latest
         command: ["/home/runner/run.sh"]
##         resources:
##          limits:
##            cpu: 500m
##            memory: 512Mi

## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
# controllerServiceAccount:
#   namespace: arc-system
#   name: test-arc-gha-runner-scale-set-controller

Controller Logs

: 0}
2024-04-12T16:05:03Z    INFO    EphemeralRunnerSet  Scaling comparison  {"ephemeralrunnerset": {"name":"ubuntu-latest-tjmzl","namespace":"ubuntu-latest"}, "current": 1, "desired": 1}
2024-04-12T16:05:03Z    INFO    EphemeralRunner EphemeralRunner pod is deleted and status is updated with failure count {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:03Z    INFO    EphemeralRunner Ephemeral runner container failed   {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "exitCode": 127}
2024-04-12T16:05:03Z    INFO    EphemeralRunner Updating ephemeral runner status to track the failure count {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:03Z    INFO    EphemeralRunner EphemeralRunner pod is deleted and status is updated with failure count {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:04Z    INFO    EphemeralRunner Ephemeral runner container failed   {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "exitCode": 127}
2024-04-12T16:05:04Z    INFO    EphemeralRunner Updating ephemeral runner status to track the failure count {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:04Z    INFO    EphemeralRunner EphemeralRunner pod is deleted and status is updated with failure count {"ephmeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:05Z    INFO    EphemeralRunner Ephemeral runner container failed   {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "exitCode": 127}
2024-04-12T16:05:05Z    INFO    EphemeralRunner Updating ephemeral runner status to track the failure count {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:05Z    INFO    EphemeralRunner EphemeralRunner pod is deleted and status is updated with failure count {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:05Z    INFO    EphemeralRunner EphemeralRunner has failed more than 5 times. Marking it as failed  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:05Z    INFO    EphemeralRunner Updating ephemeral runner status to Failed  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:05Z    INFO    EphemeralRunner Removing the runner from the service    {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:05Z    INFO    EphemeralRunnerSet  Ephemeral runner counts {"ephemeralrunnerset": {"name":"ubuntu-latest-tjmzl","namespace":"ubuntu-latest"}, "pending": 0, "running": 0, "finished": 0, "failed": 1, "deleting": 0}
2024-04-12T16:05:05Z    INFO    EphemeralRunnerSet  Scaling comparison  {"ephemeralrunnerset": {"name":"ubuntu-latest-tjmzl","namespace":"ubuntu-latest"}, "current": 1, "desired": 1}
2024-04-12T16:05:05Z    INFO    EphemeralRunnerSet  Updating status with current runners count  {"ephemeralrunnerset": {"name":"ubuntu-latest-tjmzl","namespace":"ubuntu-latest"}, "count": 1}
2024-04-12T16:05:05Z    INFO    actions-clients retrieve actions client {"githubConfigURL": "https://github.com/org", "namespace": "ubuntu-latest"}
2024-04-12T16:05:05Z    INFO    actions-clients using cache client  {"githubConfigURL": "https://github.com/org", "namespace": "ubuntu-latest"}
2024-04-12T16:05:05Z    INFO    EphemeralRunner Removing runner from the service    {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "runnerId": 462}
2024-04-12T16:05:05Z    INFO    actions-clients refreshing token    {"githubConfigUrl": "https://github.com/org"}
2024-04-12T16:05:05Z    INFO    actions-clients getting runner registration token   {"registrationTokenURL": "https://api.github.com/orgs/org/actions/runners/registration-token"}
2024-04-12T16:05:05Z    INFO    EphemeralRunnerSet  Ephemeral runner counts {"ephemeralrunnerset": {"name":"ubuntu-latest-tjmzl","namespace":"ubuntu-latest"}, "pending": 0, "running": 0, "finished": 0, "failed": 1, "deleting": 0}
2024-04-12T16:05:05Z    INFO    EphemeralRunnerSet  Scaling comparison  {"ephemeralrunnerset": {"name":"ubuntu-latest-tjmzl","namespace":"ubuntu-latest"}, "current": 1, "desired": 1}
2024-04-12T16:05:05Z    INFO    AutoscalingRunnerSet    Find existing ephemeral runner set  {"autoscalingrunnerset": {"name":"ubuntu-latest","namespace":"ubuntu-latest"}, "name": "ubuntu-latest-tjmzl", "specHash": "7479d7d848"}
2024-04-12T16:05:06Z    INFO    actions-clients getting Actions tenant URL and JWT  {"registrationURL": "https://api.github.com/actions/runner-registration"}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Removed runner from the service {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "runnerId": 462}
2024-04-12T16:05:06Z    INFO    EphemeralRunner EphemeralRunner is marked as Failed and deleted from the service    {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Cleaning up resources after after ephemeral runner termination  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "phase": "Failed"}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Cleaning up the runner pod  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Pod is deleted  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Cleaning up the runner jitconfig secret {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Deleting the jitconfig secret   {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Waiting for ephemeral runner owned resources to be deleted  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Cleaning up resources after after ephemeral runner termination  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "phase": "Failed"}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Cleaning up the runner pod  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Pod is deleted  {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Cleaning up the runner jitconfig secret {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner Secret is deleted   {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}}
2024-04-12T16:05:06Z    INFO    EphemeralRunner EphemeralRunner has already finished. Stopping reconciliation and waiting for EphemeralRunnerSet to clean it up {"ephemeralrunner": {"name":"ubuntu-latest-tjmzl-runner-55vld","namespace":"ubuntu-latest"}, "phase": "Failed"}

Runner Pod Logs

Defaulted container "runner" out of: runner, dind, init-dind-externals (init)
Waiting for docker to be ready.
github-actions[bot] commented 2 months ago

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

sravula84 commented 2 months ago

we are also facing the same issue almost from past 5 months ? any suggestions highly appreciated

nikola-jokic commented 2 months ago

Hey @gpadmesh,

You do need to mount the docker socket from the host in order to run it, and you need to start the docker daemon. This is a configuration issue, it is not related to ARC itself. ARC will only spin up the runner described by the template field.