Open brianjmurrell opened 1 month ago
systemctl enable --user --now podman.socket podman.service sudo ln -sf /run/user/$EUID/podman/podman.sock /var/run/docker.sock
Didn't seem to work for me.
Ultimately, shouldn't the docker socket path (i.e. $DOCKER_HOST
if specified) that is being used by act
to start the container that the runner is in[1] be the same socket path that is used by docker
commands inside the runner?
[1] The path that is specified as the very first line of act
's output: time="2024-07-10T09:58:27-04:00" level=info msg="Using docker host 'unix:///run/user/1001/podman/podman.sock', and daemon socket 'unix:///run/user/1001/podman/podman.sock'"
which in my case comes from my $DOCKER_HOST
env. variable.
ln -sf
is too weak to survive a docker -v <host-path>:/var/run/docker.sock
, something like mount --bind is betterYes a broken symlink let act's auto detected to abort searching it's hacky list of paths.
Ultimately, shouldn't the docker socket path (i.e.
$DOCKER_HOST
if specified) that is being used byact
to start the container that the runner is in[1] be the same socket path that is used bydocker
commands inside the runner?
Hmm, please note due to the immense amount of log content for the case DOCKER_HOST unset in this issue there is not enough information for the valid DOCKER_HOST=unix:///run/user/1001/podman/podman.sock
case that is expected to work
ln -sf
is too weak to survive adocker -v <host-path>:/var/run/docker.sock
, something like mount --bind is better
In fact ln -sf
(i.e. the workaround above) is working. But only if I set SELinux to permissive (which is a bad thing).
Yes a broken symlink let act's auto detected to abort searching it's hacky list of paths.
I suppose it's searching it's hacky list of paths ought to avoid broken symlinks and move on to the next candidate. But is act
actually using it's hacky list of paths at this point? What is failing here is a docker
command initiated from the workflow in the runner. This is just regular docker
here isn't it? It would use $DOCKER_HOST
if it were propagated into the runner namespace wouldn't it? Of course that assumes that the path that $DOCKER_HOST
points at is mounted into the runner also doesn't it?
Hmm, please note due to the immense amount of log content for the case DOCKER_HOST unset in this issue there is not enough information for the valid
DOCKER_HOST=unix:///run/user/1001/podman/podman.sock
case that is expected to work
So are you asking for more information? I'd be happy to provide whatever is necessary. I can tell you that DOCKER_HOST
is unset in runner. That seems like the minimum requirement for all of this to work doesn't it?
This is just regular docker here isn't it?
No it's the go client library of docker to send api requests a lot of docker cli behavior doesn't apply, all docker commands shown in the log are all symbolical and no real command
I'm not using SELinux, since I use ubuntu and have a permissive developer system.
This can indeed make things much worse
I think you misunderstood, DOCKER_HOST
should be enough for act if it is a unix://
protocol to use podman of any socket location that is desired.
Also act does never add any bind mount selinux labels, so I doubt act works well
This is just regular docker here isn't it?
No it's the go client library of docker to send api requests a lot of docker cli behavior doesn't apply, all docker commands shown in the log are all symbolical and no real command
To be perfectly clear, what is failing here is a step like this in my workflow file:
- name: …
continue-on-error: true
run: docker build --file Dockerfile
--tag build
.
and/or:
- name: …
continue-on-error: true
run: |
docker run --name build-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.distro }}
--user build
-v "$PWD":"$PWD" -w "$PWD"
--privileged=true
-e FULLNAME="$FULLNAME"
make chrootbuild"
Are you telling me that the go client library and/or the act
runner are intercepting these docker
commands in the run:
statement and turning them into API requests?
I'm not using SELinux, since I use ubuntu and have a permissive developer system.
Well, "developer system" is not really any reason to make SELinux permissive (on distros that support SELinux). In fact it's a great way to develop software that then breaks on end-user enforcing systems.
I think you misunderstood,
DOCKER_HOST
should be enough for act if it is aunix://
protocol to use podman of any socket location that is desired.
The use of $DOCKER_HOST outside of the runner, to tell act
where to find the socket works fine. It's the problem of then using the docker
command inside the workflow on the runner that is having the problem.
Yes it helps to provide more context, like you did now. Information is all and nothing.
Are you telling me that the go client library and/or the act runner are intercepting these docker commands in the run: statement and turning them into API requests?
No, you didn't tell me that this is inside a user defined run step, how should I guess that? All errors posted here were about act's log output coming from the go client library used inside of act
Your problem is of the following nature (act_runner is a Gitea Actions runner for an open source github alternative) Except that I'm writing about dind images, in your cases is the podman daemon outside of the workflow container
If I'm writing nowadays about dind I no longer think that a dind for act_runner image gives you a true dind experience
For example in a true dind runner I expect that I could do docker run --rm -v $PWD:/tmp/mnt ubuntu:latest ls /tmp/mnt > within a run step and have my workflow working dir mounted in the sub container of my workflow
Those dind runner images however would mount paths from the act_runner container (potential harmful if this socket > get's bind mounted), because dockerd lives there. Did take years to change my mind. What do you expect from dind? For kubernetes are those act_runner dind images needed, but otherwise you have to deal > with the same issues as without dind or avoid bind mounts everywhere
You need dind for this to work, but act doesn't have such a feature as of now
ln -sf
is too weak to survive adocker -v <host-path>:/var/run/docker.sock
, something like mount --bind is betterIn fact
ln -sf
(i.e. the workaround above) is working. But only if I set SELinux to permissive (which is a bad thing).
I doubt docker info works inside the workflow if you do that without pointing with DOCKER_HOST to a unix socket that is not a symlink After bind mount you have a dead symlink in the container
Keep in mind avoiding errors is not the same as making things work
Yes it helps to provide more context, like you did now. Information is all and nothing.
Are you telling me that the go client library and/or the act runner are intercepting these docker commands in the run: statement and turning them into API requests?
No, you didn't tell me that this is inside a user defined run step, how should I guess that?
The title of this issue is Setting docker socket inside runner. Unless there is something I am missing, inside [the] runner implies the use of docker
commands inside the runner. I'm not sure how else docker would be used inside the runner.
I can use podman, beside the container nesting problem that you just introduced
Yes, I am using podman just fine, modulo wanting to use docker
in a run:
step due to the assumption of binding /var/run/docker.sock
into the runner container. It seems to me that act
should be binding whatever socket it is using into the container, not a hard-coded (or however it's determined) /var/run/docker.sock
.
It seems to me that
act
should be binding whatever socket it is using into the container, not a hard-coded (or however it's determined)/var/run/docker.sock
.
But act does exactly that for me, please expand the collapsed log in https://github.com/nektos/act/issues/2393#issuecomment-2221174801
.
DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock ~/Downloads/act_Linux_x86_64/act -v -W w.yml
[check docs/build] [DEBUG] Common container.HostConfig ==> &{Binds:[/run/user/1000/podman/podman.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:host PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:
OomKillDisable: PidsLimit: Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[{Type:volume Source:act-check-docs-build-207843b3f40f24fbdcf3e9acdca6e0dfbc66756b2616c30371c7a1815189603f-env Target:/var/run/act ReadOnly:false Consistency: BindOptions: VolumeOptions: TmpfsOptions: ClusterOptions: } {Type:volume Source:act-check-docs-build-207843b3f40f24fbdcf3e9acdca6e0dfbc66756b2616c30371c7a1815189603f Target:/tmp/test ReadOnly:false Consistency: BindOptions: VolumeOptions: TmpfsOptions: ClusterOptions: } {Type:volume Source:act-toolcache Target:/opt/hostedtoolcache ReadOnly:false Consistency: BindOptions: VolumeOptions: TmpfsOptions: ClusterOptions: }] MaskedPaths:[] ReadonlyPaths:[] Init: }
Only the dest socket /var/run/docker.sock
is hardcoded as of now, with the option to turn off that mount
However it doesn't solve your problem, because the socket bind mount is the cause of your problem
C4Context
title System Context diagram for Internet Banking System
Boundary(system, "Host System") {
System(podmansock, "podman", "Allows to run containers")
Boundary(podcontainers, "Podman Containers") {
System(jobContainer, "Job Container")
System(dockerruncommand, "Docker Run Command")
}
System(act, "act", "Allows to run workflows")
}
Rel(act, podmansock, "")
Rel(act, jobContainer, "")
Rel(jobContainer, dockerruncommand, "Broken Bind Mounts!!!")
Rel(jobContainer, podmansock, "Broken Bind Mounts!!!")
The job container now tries to mount the path of the $PWD of the container from your host system to the "Docker Run Command" container.
You want to mount the path of the $PWD of the container from your container to the "Docker Run Command" container.
Only if podman is part of jobcontainer and "Docker Run Command" is spawned inside of jobcontainer everything is working like with VM and no container.
And so does this become an issue particularly when you try to bind paths inside the runner into the docker container you start in the runner? I.e. doing something like:
- name: Run in Docker
run: |
set -eux
rm -rf result
mkdir -p result
touch result/flag
docker run --name build-"$unique"-${{ matrix.distro }} \
"${docker_args[@]}" \
-v "$PWD":"$PWD" -w "$PWD" \
-v "$PWD"/result:/var/tmp/result \
build bash -c "if [ ! -e /var/tmp/result/flag ]; then \
echo \"/var/tmp/result/flag does not exist\" \
exit 1 \
fi"
If so, any viable work-around?
this become an issue particularly when you try to bind paths inside the runner into the docker container you start in the runner?
Yes exactly like that is always broken.
-P ubuntu-latest=-self-hosted
docker cp
(you may need to pipe a tar file via -
, not shure how the cli transfers files by default in this setup)I have been busy this week for learning in my university, I'm actually maintaining two tools to run GitHub Actions Locally one using the official actions/runner based on .net (that also supports Azure Pipelines) and this one
This issue mentioned here happens on the official runner as well, if they run both a job container via docker/podman, while running both tools directly on a VM both act and the official runner has no problem at all
In https://github.com/nektos/act/issues/2393 is now a --dind
flag to get a priviledged dind container per job.
The following docker run -v $PWD:$PWD --rm alpine:latest ls $PWD
works in that scenario
With --reuse
as well you can speed up a debug cycle by not purging the dind container and let it run between restarts of act.
I'm mounting the same folder of the job container has as mounts, so both have the same container paths for these mounts.
So /opt/hostedtoolcache, github.workspace and the directory of the extracted actions can be bind mounted as if you would use a VM.
Problems with this setup (Fedora 39 + Podman)
After some digging, I've identified several issues:
/var/run/docker.sock
is a broken link to/run/podman/podman.sock
. The file/run/podman/podman.sock
doesn't exist because by default Podman works in daemonless mode, so there's no daemon and therefore no socket. For the file/run/podman/podman.sock
to exist, I've had to start/enable the systemd service for Podman (on Fedora:podman.socket
&podman.service
)./var/run/docker.sock
, but not for/run/podman/podman.sock
. To solve this, I've decided to enable the systemd service as an unprivileged user rather than as root, and set the environment variableDOCKER_HOST
to point to/run/user/$EUID/podman/podman.sock
.act
seems to use theDOCKER_HOST
environment variable for some steps of the process, but not all of them. When I setDOCKER_HOST=unix:///run/user/$EUID/podman/podman.sock
, pulling the Docker image works, but creating the container still fails. In the end, I've decided to overwrite the symbolic link at/var/run/docker.sock
to point to/run/user/$EUID/podman/podman.sock
.How to diagnose the root cause
Here are the steps I've used to find out the root cause:
Check that the socket files exist and have the right permissions:
Create the socket if it doesn't exist:
Check that your user can read the sockets:
Enable the socket/service as an unprivileged user instead of root:
Verify again using the user socket:
Overwrite the symbolic link at
/var/run/docker.sock
ifact
/podman
is still trying to access/var/run/docker.sock
instead of using theDOCKER_HOST
environment variable:and check again:
Solution for Fedora 39 + Podman
In the end, these are the commands that I've used to fix the problem on Fedora 39 using Podman:
Notice that this is an acceptable workaround for me, since I'm the only user who uses
act
&podman
on my computer, but it may not work if multiple unprivileged users need to useact
&podman
on the same computer, because the user socket only has permissions for that specific user.Originally posted by @jonancm in https://github.com/nektos/act/issues/1744#issuecomment-1873335752