Open FleetAdmiralButter opened 2 years ago
@FleetAdmiralButter Hey! This does seem like a valid feature request.
Would you mind submitting a PR for that? Something that implemented in a way similar to dockerMTU would work(An ARC_ prefixed envvar passed to the runner pod's docker
container, which invokes a conditional block of code that updates the docker daemon.json
so that docker uses the specific bip and the CIDR?)
I think this is happening to me too.
@ctr26 Hey! Thanks for chiming in- please feel free to submit a pull request 😄
ARC's runner pod's docker
sidecar container uses this script to dynamically generate the docker daemon.json
. You might be able to enhance it to configure the bridge network.
I tried to get around this by using containerMode: kubernetes
but that doesn't seem to work with secrets?
# runnerdeployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: ***
spec:
replicas: 1
# containerMode: kubernetes
template:
spec:
dockerMTU: 1400
repository: **********
labels:
- ******
ephemeral: false
This doesn't actually change the MTU for me
docker0 Link encap:Ethernet HWaddr 02:42:1C:A0:EF:2A
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)`
As a work around we turned off docker in the RunnerDeployment since we ran into the same problem.
We were even looking to bake our own runner image with different daemon.json.
dockerEnabled: false
You do have a long start time while it searches the side car (roughly 5 mins), that's the downside
Good feature request though
I ended up disabling docker too, which causes its own problems but it's fine
@ctr26 Currently, dockerMTU
isn't enough for setting it all up. You might also need https://github.com/actions-runner-controller/actions-runner-controller/pull/1201.
I tried to get around this by using containerMode: kubernetes but that doesn't seem to work with secrets?
Which secret are you referring, K8s secret or GitHub Actions secret? I thought it works with both though.
GitHub actions secrets were failing, I'll check again though.
You do have a long start time while it searches the side car (roughly 5 mins), that's the downside
@jlees-Gss Thanks. But where do that 5 minutes come from? 🤔
I thought dockerEnabled: false
would just disable the whole docker sidecar along with the check to see the dockerd is up and running. If it really delays the runner startup for 5 minutes just for dockerEnabled: false
, it might be a bug in ARC.
@ctr26 Thanks. Interesting! It would be great if you could share related logs or errors. If it's coming from workflow jobs, it might be an issue in actions/runner
.
# runnerdeployment.yaml apiVersion: actions.summerwind.dev/v1alpha1 kind: RunnerDeployment metadata: name: *** spec: replicas: 1 # containerMode: kubernetes template: spec: dockerMTU: 1400 repository: ********** labels: - ****** ephemeral: false
This doesn't actually change the MTU for me
docker0 Link encap:Ethernet HWaddr 02:42:1C:A0:EF:2A inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)`
@ctr26 can you try with the dind image please
That was a typo above, I did try it with the DIND image and the correct volume mounts, the MTU even changed but I still had the same problem.
@ctr26 I believe you still need https://github.com/actions-runner-controller/actions-runner-controller/pull/1201 to propagate the MTU change to all the actions and service containers created by actions/runner
.
You do have a long start time while it searches the side car (roughly 5 mins), that's the downside
@jlees-Gss Thanks. But where do that 5 minutes come from? 🤔
I thought
dockerEnabled: false
would just disable the whole docker sidecar along with the check to see the dockerd is up and running. If it really delays the runner startup for 5 minutes just fordockerEnabled: false
, it might be a bug in ARC.
So according to the log I think it first has to timeout looking for the daemon. My understanding at least
2022-08-21 00:43:39.752 DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2022-08-21 00:43:39.753 DEBUG --- Waiting until Docker is available or the timeout is reached
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
}
√ Connected to GitHub
Current runner version: '2.295.0'
2022-08-21 00:45:42Z: Listening for Jobs
Unless I've missed something in the RunnerDeployment that needs changing.
..
ephemeral: true
# true (default) = A privileged docker sidecar container is included in the runner pod.
# false = A docker sidecar container is not included in the runner pod and you can't use docker.
# If set to false, there are no privileged container and you cannot use docker.
dockerEnabled: false
# false (default) = Docker support is provided by a sidecar container deployed in the runner pod.
# true = No docker sidecar container is deployed in the runner pod but docker can be used within the runner container instead. The image summerwind/actions-runner-dind is used by default.
dockerdWithinRunnerContainer: true
..
EDIT: I found the DISABLE_WAIT_FOR_DOCKER field
resolves start up time
@jlees-Gss Thanks! Another solution to your problem is to "unset" dockerdWithinRunnerContainer: true
and change the image
to a non-dind variant, because those both basically forces the runner to start the dind daemon anyway. You should prevent it from starting the dind daemon in the first place. (You don't need dind and that's why you've set dockerEnabled: false
earlier, right?
@mumoshu Thanks for the advice We didn't like the compromises so we built our own daemon.json configmap to inject
We then verified by kubectl exec -it into the pod and was able to use dockerd ps
We're still in the process of verifying our dockerfile workflows are working as should bee Will confirm ASAP
I'll share the code just now
Create configmap of new daemon.json
---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: <runner namespace>
name: <runner name>-worker-docker-daemon
labels:
app: <runner name>-worker
data:
daemon.json: |
{
"bip": "192.168.169.5/24",
"fixed-cidr": "192.168.169.5/25",
"default-address-pools":[ {"base":"192.168.170.5/24","size":28} ]
}
Attach configmap to runnerdeployment as volume
ephemeral: true
dockerEnabled: true
dockerdWithinRunnerContainer: true
workDir: /home/runner/work
volumes:
- name: <runner name>-worker-docker-daemon
configMap:
name: <runner name>-worker-docker-daemon
volumeMounts:
- name: <runner name>-worker-docker-daemon
mountPath: /etc/docker/daemon.json
subPath: daemon.json
readOnly: true
Just for your information this problem was occurring in AWS for us
@jlees-Gss I feel like you've got a really close workaround here. I'm in a similar boat, although docker doesn't seem to be live inside my runner container.
Would you be willing to share the whole yaml? I'm in the same boat you are injecting daemon.json via configmap, and I'm ending up with the repeated message Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
. I could DISABLE_WAIT_FOR_DOCKER as you did earlier, but I'd actually like the docker daemon inside the runner to run as opposed to simply fail out.
@mumoshu I'm going through the code, but I'm afraid I don't see the part of the code in the runner that STARTS docker... do you have any info on this?
It appears to be possible to use the dockerVolumeMounts
option along with a custom configmap as defined by @jlees-Gss above to override the daemon.json in the dind pod, meaning you don't have to jam the docker daemon into the runner.
This works (on AWS) for me without needing to disable waiting for docker or utilizing dockerdWithinRunnerContainer
apiVersion: v1
kind: ConfigMap
metadata:
namespace: poc
name: daemon-json
data:
daemon.json: |
{
"bip": "192.168.169.5/24",
"fixed-cidr": "192.168.169.5/25",
"default-address-pools":[ {"base":"192.168.170.5/24","size":28} ]
}
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: poc
spec:
template:
spec:
volumes:
- name: daemon-json
configMap:
name: daemon-json
dockerVolumeMounts:
- name: daemon-json
mountPath: /etc/docker/daemon.json
subPath: daemon.json
readOnly: true
@dputnamfr your solution works for me on Azure, AKS cluster as well
Is your feature request related to a problem? Please describe.
The default Docker bridge network, 172.17.0.0/16, overlaps with our VPC network. This breaks network connectivity within the DIND container.
Describe the solution you'd like Similar to the dockerMTU parameter, could there be a parameter that could be used to override the default Docker bridge network?
Describe alternatives you've considered Alternatively, could the controller implement a way to specify a different DOCKER_HOST other than localhost? I'm able to get a separate DIND deployment running by manually setting the bridge network, but there doesn't seem a way to make ARC connect to it.