actions / actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runners
Apache License 2.0
4.4k stars 1.04k forks source link

Unable to update runner packages with proxy enabled #3521

Closed erypolovina closed 1 month ago

erypolovina commented 1 month ago

Checks

Controller Version

0.9.0

Deployment Method

Helm

Checks

To Reproduce

Enable the proxy variable on `gha-runner-scale-set/values.yaml` file

proxy:
  http:
    url: http://<my.proxy.endpoing>:3128
    # credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
  https:
    url: http://<my.proxy.endpoint>:3128
    # credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
  noProxy:
    - X.X.X.X/16

We have disabled external internet access for our Runners and are required to use a proxy to reach external endpoints (ex. ubuntu updates, github...)

When the runners spin up, if you try to run sudo apt update it will hang in our case. But if you connect to the runner and set the proxy environment variables on root, the command will then work.

root@gha-scaleset-small-n6njd-runner-7ldsp:~# export https_proxy=http://<my.proxy.endpoint>:3128
root@gha-scaleset-small-n6njd-runner-7ldsp:~# export http_proxy=http://<my.proxy.endpoint>:3128
root@gha-scaleset-small-n6njd-runner-7ldsp:~# sudo apt update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:3 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.7 kB]

Additionally, if you check the environment variables for runner user and root user in the runners with

env | grep proxy

We can see that the Proxy variables are set for the runner and not root user.


### Describe the bug

Can not perform `sudo apt update` command on a runner due to the environment variables for the proxy not being correctly set. It is only set on runner user but does not account for running root commands.

### Describe the expected behavior

Running commands that require use of a  proxy as sudo such as `sudo apt update` should be working as expected.

### Additional Context

```yaml
githubConfigUrl: ${github_enterprise_server_url}
githubConfigSecret:
  github_token: ${github_token}

proxy:
  http:
    url: http://<my.proxy.endpoint>:3128
  https:
    url: http://<my.proxy.endpoint>:3128
  noProxy:
    - X.X.X.X/16

maxRunners: 25
minRunners: 5

runnerGroup: ${runner_group}
runnerScaleSetName: ${runner_scale_set_name}

listenerTemplate:
  metadata:
    labels:
      app: ${k8s_labels}
  spec:
    containers:
    - name: listener
      securityContext:
        runAsUser: 1000
    nodeSelector:
      dedicated: cicd
    tolerations:
      - effect: NoSchedule
        key: dedicated
        operator: Equal
        value: cicd

template:
  metadata:
    labels:
      app: ${k8s_labels}
  spec:
    initContainers:
    - name: init-dind-externals
      image: ghcr.io/actions/actions-runner:latest
      command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
      volumeMounts:
        - name: dind-externals
          mountPath: /home/runner/tmpDir
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: DOCKER_HOST
          value: unix:///var/run/docker.sock
      resources:
        limits:
          memory: ${requests_mem}
        requests:
          memory: ${requests_mem}
          cpu: ${requests_cpu}
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /var/run
        - name: docker-config-volume
          mountPath: /home/runner/.docker # docker actions write to this dir and mounting secret is readonly, need this to allow writing
        - name: artifactory-credentials
          mountPath: /home/runner/.docker/config.json
          subPath: config.json
    - name: dind
      image: docker:dind
      args:
        - dockerd
        - --host=unix:///var/run/docker.sock
        - --group=$(DOCKER_GROUP_GID)
      env:
        - name: DOCKER_GROUP_GID
          value: "123"
      resources:
        limits:
          memory: ${requests_mem}
        requests:
          memory: ${requests_mem}
          cpu: ${requests_cpu}
      securityContext:
        privileged: true
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /var/run
        - name: dind-externals
          mountPath: /home/runner/externals
    volumes:
      - name: artifactory-credentials
        secret:
          items:
            - key: .dockerconfigjson
              path: config.json
          secretName: ${artifactory_config_secret_name}
      - name: docker-config-volume
        emptyDir: {}
      - name: work
        emptyDir: {}
      - name: dind-sock
        emptyDir: {}
      - name: dind-externals
        emptyDir: {}
    nodeSelector:
      dedicated: cicd
    tolerations:
      - effect: NoSchedule
        key: dedicated
        operator: Equal
        value: cicd

### Controller Logs

```shell
https://gist.github.com/erypolovina/019c38fa59d595a6f4a8e2c3e9530c80

Runner Pod Logs

https://gist.github.com/erypolovina/43dbeb6545763f147418fdca9ef80b30
nikola-jokic commented 1 month ago

Hey @erypolovina,

I don't think this is an issue with ARC, but more importantly, I think this is working as expected. When you run sudo, environment variables are not preserved by default. I think you would need to run something like sudo -E apt-get update.