search

actions / actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runners
Apache License 2.0
4.56k stars 1.07k forks source link

Ability to configure DIND bridge network #1652

Open FleetAdmiralButter opened 2 years ago

FleetAdmiralButter commented 2 years ago

Is your feature request related to a problem? Please describe.

The default Docker bridge network, 172.17.0.0/16, overlaps with our VPC network. This breaks network connectivity within the DIND container.

Describe the solution you'd like Similar to the dockerMTU parameter, could there be a parameter that could be used to override the default Docker bridge network?

Describe alternatives you've considered Alternatively, could the controller implement a way to specify a different DOCKER_HOST other than localhost? I'm able to get a separate DIND deployment running by manually setting the bridge network, but there doesn't seem a way to make ARC connect to it.

mumoshu commented 2 years ago

@FleetAdmiralButter Hey! This does seem like a valid feature request. Would you mind submitting a PR for that? Something that implemented in a way similar to dockerMTU would work(An ARC_ prefixed envvar passed to the runner pod's docker container, which invokes a conditional block of code that updates the docker daemon.json so that docker uses the specific bip and the CIDR?)

ctr26 commented 2 years ago

I think this is happening to me too.

mumoshu commented 2 years ago

@ctr26 Hey! Thanks for chiming in- please feel free to submit a pull request 😄 ARC's runner pod's docker sidecar container uses this script to dynamically generate the docker daemon.json. You might be able to enhance it to configure the bridge network.

https://github.com/actions-runner-controller/actions-runner-controller/blob/36e95dad477e72cd98c1414817d56697bffd5d4e/runner/startup.sh#L27-L36

ctr26 commented 2 years ago

I tried to get around this by using containerMode: kubernetes but that doesn't seem to work with secrets?

ctr26 commented 2 years ago 
# runnerdeployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: ***
spec:
  replicas: 1
  # containerMode: kubernetes
  template:
    spec:
      dockerMTU: 1400
      repository: **********
      labels:
        - ******
      ephemeral: false

This doesn't actually change the MTU for me

docker0   Link encap:Ethernet  HWaddr 02:42:1C:A0:EF:2A  
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)`
jlees-Gss commented 2 years ago

As a work around we turned off docker in the RunnerDeployment since we ran into the same problem. We were even looking to bake our own runner image with different daemon.json. dockerEnabled: false You do have a long start time while it searches the side car (roughly 5 mins), that's the downside

Good feature request though

ctr26 commented 2 years ago

I ended up disabling docker too, which causes its own problems but it's fine

mumoshu commented 2 years ago

@ctr26 Currently, dockerMTU isn't enough for setting it all up. You might also need https://github.com/actions-runner-controller/actions-runner-controller/pull/1201.

I tried to get around this by using containerMode: kubernetes but that doesn't seem to work with secrets?

Which secret are you referring, K8s secret or GitHub Actions secret? I thought it works with both though.

ctr26 commented 2 years ago

GitHub actions secrets were failing, I'll check again though.

mumoshu commented 2 years ago

You do have a long start time while it searches the side car (roughly 5 mins), that's the downside

@jlees-Gss Thanks. But where do that 5 minutes come from? 🤔

I thought dockerEnabled: false would just disable the whole docker sidecar along with the check to see the dockerd is up and running. If it really delays the runner startup for 5 minutes just for dockerEnabled: false, it might be a bug in ARC.

mumoshu commented 2 years ago

@ctr26 Thanks. Interesting! It would be great if you could share related logs or errors. If it's coming from workflow jobs, it might be an issue in actions/runner.

toast-gear commented 2 years ago 
# runnerdeployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: ***
spec:
  replicas: 1
  # containerMode: kubernetes
  template:
    spec:
      dockerMTU: 1400
      repository: **********
      labels:
        - ******
      ephemeral: false

This doesn't actually change the MTU for me

docker0   Link encap:Ethernet  HWaddr 02:42:1C:A0:EF:2A  
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)`

@ctr26 can you try with the dind image please

ctr26 commented 2 years ago

That was a typo above, I did try it with the DIND image and the correct volume mounts, the MTU even changed but I still had the same problem.

mumoshu commented 2 years ago

@ctr26 I believe you still need https://github.com/actions-runner-controller/actions-runner-controller/pull/1201 to propagate the MTU change to all the actions and service containers created by actions/runner.

jlees-Gss commented 2 years ago

You do have a long start time while it searches the side car (roughly 5 mins), that's the downside

@jlees-Gss Thanks. But where do that 5 minutes come from? 🤔

I thought dockerEnabled: false would just disable the whole docker sidecar along with the check to see the dockerd is up and running. If it really delays the runner startup for 5 minutes just for dockerEnabled: false, it might be a bug in ARC.

So according to the log I think it first has to timeout looking for the daemon. My understanding at least

2022-08-21 00:43:39.752  DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2022-08-21 00:43:39.753  DEBUG --- Waiting until Docker is available or the timeout is reached
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
}
√ Connected to GitHub

Current runner version: '2.295.0'
2022-08-21 00:45:42Z: Listening for Jobs

Unless I've missed something in the RunnerDeployment that needs changing.

..
ephemeral: true
# true (default) = A privileged docker sidecar container is included in the runner pod.
# false = A docker sidecar container is not included in the runner pod and you can't use docker.
# If set to false, there are no privileged container and you cannot use docker.
dockerEnabled: false
# false (default) = Docker support is provided by a sidecar container deployed in the runner pod.
# true = No docker sidecar container is deployed in the runner pod but docker can be used within the runner container instead. The image summerwind/actions-runner-dind is used by default.
dockerdWithinRunnerContainer: true
..

EDIT: I found the DISABLE_WAIT_FOR_DOCKER field

resolves start up time

mumoshu commented 2 years ago

@jlees-Gss Thanks! Another solution to your problem is to "unset" dockerdWithinRunnerContainer: true and change the image to a non-dind variant, because those both basically forces the runner to start the dind daemon anyway. You should prevent it from starting the dind daemon in the first place. (You don't need dind and that's why you've set dockerEnabled: false earlier, right?

jlees-Gss commented 2 years ago

@mumoshu Thanks for the advice We didn't like the compromises so we built our own daemon.json configmap to inject

We then verified by kubectl exec -it into the pod and was able to use dockerd ps

We're still in the process of verifying our dockerfile workflows are working as should bee Will confirm ASAP

I'll share the code just now

Create configmap of new daemon.json

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: <runner namespace>
  name: <runner name>-worker-docker-daemon
  labels:
    app: <runner name>-worker
data:
  daemon.json: |
    {
      "bip": "192.168.169.5/24",
      "fixed-cidr": "192.168.169.5/25",
      "default-address-pools":[ {"base":"192.168.170.5/24","size":28} ]
    }

Attach configmap to runnerdeployment as volume

ephemeral: true
dockerEnabled: true
dockerdWithinRunnerContainer: true
workDir: /home/runner/work
volumes:
  - name: <runner name>-worker-docker-daemon
    configMap:
      name: <runner name>-worker-docker-daemon
volumeMounts:
  - name: <runner name>-worker-docker-daemon
    mountPath: /etc/docker/daemon.json
    subPath: daemon.json
    readOnly: true

Just for your information this problem was occurring in AWS for us

dputnamfr commented 1 year ago

@jlees-Gss I feel like you've got a really close workaround here. I'm in a similar boat, although docker doesn't seem to be live inside my runner container.

Would you be willing to share the whole yaml? I'm in the same boat you are injecting daemon.json via configmap, and I'm ending up with the repeated message Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. I could DISABLE_WAIT_FOR_DOCKER as you did earlier, but I'd actually like the docker daemon inside the runner to run as opposed to simply fail out.

@mumoshu I'm going through the code, but I'm afraid I don't see the part of the code in the runner that STARTS docker... do you have any info on this?

dputnamfr commented 1 year ago

It appears to be possible to use the dockerVolumeMounts option along with a custom configmap as defined by @jlees-Gss above to override the daemon.json in the dind pod, meaning you don't have to jam the docker daemon into the runner.

This works (on AWS) for me without needing to disable waiting for docker or utilizing dockerdWithinRunnerContainer

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: poc
  name: daemon-json
data:
  daemon.json: |
    {
      "bip": "192.168.169.5/24",
      "fixed-cidr": "192.168.169.5/25",
      "default-address-pools":[ {"base":"192.168.170.5/24","size":28} ]
    }

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: poc
spec:

  template:
    spec:
      volumes:
      - name: daemon-json
        configMap:
          name: daemon-json
      dockerVolumeMounts:
      - name: daemon-json
        mountPath: /etc/docker/daemon.json
        subPath: daemon.json
        readOnly: true
nebed commented 1 year ago

@dputnamfr your solution works for me on Azure, AKS cluster as well