pat-s
commented
9 months ago Can you prove that it's related to selinux? I.e. by changing the selinux level and testing whether it works/fails with doing so?
Open birkenfunk opened 9 months ago
pat-s
commented
9 months ago Can you prove that it's related to selinux? I.e. by changing the selinux level and testing whether it works/fails with doing so?
birkenfunk
commented
9 months ago I already fixed it together with @6543 and after disabeling SeLinux it worked without any problems
To Fix it I had to execute the commands:
sudo ausearch -c 'woodpecker-agen' --raw | audit2allow -M my-woodpeckeragen
sudo semodule -X 300 -i my-woodpeckeragen.ppAfter doing so Woodpecker worked without any problems
So maybe you could add a documentation for SeLinux.
mscherer
commented
9 months ago That's a rather crude fix, without the logs or the content of the generated policy (or simply the setup, eg, what domain was used for woodpecker-agent, etc), I would advise against adding that to the documentation as is.
birkenfunk
commented
9 months ago Here is my docker-compose.
# docker-compose.yml
version: '3'
services:
woodpecker-server:
image: woodpeckerci/woodpecker-server:next
ports:
- 8000:8000
volumes:
- woodpecker-server-data:/var/lib/woodpecker/
env_file:
- .env
environment:
- WOODPECKER_OPEN=false
- WOODPECKER_HOST=${WOODPECKER_HOST}
- WOODPECKER_GITEA=true
- WOODPECKER_GITEA_URL=${WOODPECKER_GITEA_URL}
- WOODPECKER_GITEA_CLIENT=${WOODPECKER_GITEA_CLIENT}
- WOODPECKER_GITEA_SECRET=${WOODPECKER_GITEA_SECRET}
- WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
- WOODPECKER_ADMIN=Birkenfunk
- WOODPECKER_GRPC_SECURE=true
- WOODPECKER_GRPC_VERIFY=true
# - WOODPECKER_LOG_LEVEL=trace
woodpecker-agent:
image: woodpeckerci/woodpecker-agent:next
command: agent
restart: always
depends_on:
- woodpecker-server
volumes:
- woodpecker-agent-config:/etc/woodpecker
- /var/run/docker.sock:/var/run/docker.sock
env_file:
- .env
environment:
- WOODPECKER_SERVER=woodpecker-server:9000
- WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
- WOODPECKER_BACKEND=docker
- WOODPECKER_MAX_WORKFLOWS=3
# - WOODPECKER_LOG_LEVEL=trace
volumes:
woodpecker-server-data:
woodpecker-agent-config:This is the Log
woodpecker-server_1 | {"level":"info","time":"2023-10-31T10:32:32Z","message":"LogLevel = info"}
woodpecker-server_1 | {"level":"info","time":"2023-10-31T10:32:32Z","message":"Starting Woodpecker server with version 'next-387637bb4e'"}
woodpecker-agent_1 | {"level":"info","time":"2023-10-31T10:32:32Z","message":"LogLevel = info"}
woodpecker-agent_1 | {"level":"info","time":"2023-10-31T10:32:32Z","message":"Starting Woodpecker agent with version 'next-387637bb4e' and backend 'docker' using platform 'linux/amd64' running up to 3 pipelines in parallel"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/wp_24711997-84cd-46c2-9967-a0fe7a177c1f/kill?signal=9\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not kill container 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_clone'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Delete \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/wp_24711997-84cd-46c2-9967-a0fe7a177c1f?v=1\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not remove container 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_clone'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/wp_28a0207d-365b-4163-b64b-69fab1e78965/kill?signal=9\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not kill container 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_stage_0'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Delete \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/wp_28a0207d-365b-4163-b64b-69fab1e78965?v=1\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not remove container 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_stage_0'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/wp_4042522c-b1b1-42e1-a32c-931b7faaa1af/kill?signal=9\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not kill container 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_stage_1'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Delete \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/wp_4042522c-b1b1-42e1-a32c-931b7faaa1af?v=1\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not remove container 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_stage_1'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Delete \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/volumes/wp_01he2mnwetv0mg2p4fh5jzt1m6_0_default\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not remove volume 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_default'"}
woodpecker-agent_1 | {"level":"error","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Delete \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/wp_01he2mnwetv0mg2p4fh5jzt1m6_0_default\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"could not remove network 'wp_01he2mnwetv0mg2p4fh5jzt1m6_0_default'"}
woodpecker-agent_1 | {"level":"error","error":"rpc error: code = Unknown desc = Step finished with exit code 1, permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/volumes/create\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"grpc error: wait(): code: Unknown: rpc error: code = Unknown desc = Step finished with exit code 1, permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/volumes/create\": dial unix /var/run/docker.sock: connect: permission denied"}
woodpecker-agent_1 | {"level":"warn","repo":"Birkenfunk/TH-RO-KP-Dart-GO","pipeline":"33","id":"35","error":"rpc error: code = Unknown desc = Step finished with exit code 1, permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/volumes/create\": dial unix /var/run/docker.sock: connect: permission denied","time":"2023-10-31T10:32:43Z","message":"cancel signal received"}This is the generated file from sudo ausearch -c 'woodpecker-agen' --raw | audit2allow -M my-woodpeckeragen
module my-woodpeckeragen 1.0;
require {
type container_var_run_t;
type container_t;
type container_runtime_t;
class sock_file write;
class unix_stream_socket connectto;
}
#============= container_t ==============
allow container_t container_runtime_t:unix_stream_socket connectto;
#!!!! This avc is allowed in the current policy
allow container_t container_var_run_t:sock_file write;
6543
commented
9 months ago @mscherer if you have more knowledge about SE_Linux I would welcome your input :)
(I personally dont have it :/ )
mscherer
commented
9 months ago @6543 I do have some knowledge, but I do not think I can turn that in a easy and clean solution.
I assume the server is running some Fedora based one (or Centos/RHEL), as they are the one with a policy by default.
After digging in the policy, it seems there is no boolean to let a container speak to the docker daemon.
One solution could be semanage permissive container_t. This disable SELinux for all containers which is not great (and I think less secure than the custom policy). However, that's easier to audit (since AVCs are still in the logs) and easier to test, that's less likely to break. And that's one command, much less magic that custom policy.
Another solution is to convince upstream to add a boolean. I do not know if upstream devs would be ok with adding a boolean to let a container speak to the docker daemon. That's a rather sensitive access (once you have access to the docker daemon, you are root, one of the reason why podman exist), so I suspect the devs may not be ok with it.
I am far from being familiar with docker compose, but wouldn't adding "privileged: true" for the woodpecker-agent service be a solution ?
From a security PoV, that would give a lot of access, but if it need to speak to docker, we may not have the choice.
There is also security_opt that might be something to look at. For example, run the woodpecker-agent container as unconfined_t or staff_t would give the required access, by adding:
security_opt:
- "label=type:unconfined_t"This would limit the selinux change to just that container, and I think that's the best we can do.
A 3rd solution would be to not use the socket, but either the http access (IMHO not a great solution since you need to fiddle with TLS certs, a never ending source of annoyance) or with ssh on localhost, as explained on docker doc. But I guess there is no ssh in the woodpecker agent image, so that's not trivial.
handlebargh
commented
6 months ago I run Woodpecker with podman on a fedora server and generated a custom policy for each the server and the agent with udica. I'm not a SELinux expert so I can't guarantee that the policies are bulletproof but I tend to think that's more desirable than running the container completely unconfined. udica generates the policies from analyzing the docker inspect output of a container, While it tailores its policies, it's still a bit of a off the shelf solution.
These are the policies I generated with udica (and most probably added something mysself, I can't remember)
(block my_woodpecker_server
(blockinherit container)
(blockinherit restricted_net_container)
(allow process soundd_port_t ( tcp_socket ( name_bind )))
(allow process http_port_t ( tcp_socket ( name_bind )))
(allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
(allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
(allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
(allow process container_file_t ( sock_file ( append getattr open read write )))
)(block my_woodpecker_agent
(blockinherit container)
(blockinherit restricted_net_container)
(allow process soundd_port_t ( tcp_socket ( name_bind )))
(allow process ntop_port_t ( tcp_socket ( name_bind )))
(allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
(allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
(allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
(allow process container_file_t ( sock_file ( append getattr open read write )))
(allow process container_runtime_t ( unix_stream_socket ( connectto )))
)Note the last two allow rules in the agent policy. That's the ones that allow docker/podman socket communication.
To use the policies, load them first:
semodule -i <policy_file> /usr/share/udica/templates/{base_container.cil,net_container.cil}Then run the container with the security label:
security_opt:
- label=type:my_woodpecker_server.processTo verify the processes are running in confined mode, run ps -eZ | grep woodpecker
The output should be similar to this:
system_u:system_r:my_woodpecker_server.process:s0:c36,c1018 24989 ? 00:01:22 woodpecker-server
system_u:system_r:my_woodpecker_agent.process:s0:c119,c691 25102 ? 00:00:15 woodpecker-agentHowever, use this with caution! I can not and will guarantee the robustness of these policies. Besides I encourage everyone trying this to generate the policies by himself because different container configurations will produce different policies. So right now I wouldn't advise to add this to the documentation either.
@mscherer do you have any thoughts about this method?
mscherer
commented
5 months ago @handlebargh not much, that's also a solution that work.
The policy you generated is fine, as fine as the AVC one (even if I think that's a bit more readable and more intentional, so that's IMHO better). There isn't any obvious glaring security hole, or at least, not more than what is required by a normal usage (again, we need to be able to run arbitrary containers by speaking to a daemon that run as root and download stuff from the internet, so there isn't much to confine before breaking the system ).
If this was a PR, I would suggest that the soundd_port_t and ntop_port_t lines would be better with some comment that explain why that name (and how to change if someone decide to change the port ), but that's a upstream problem, in that it would be more readable for a neophyte if this was called port_XXXX_t and have some alias (and so not something we can solve here).
xoxys
commented
5 months ago Using privileged: true is IMO the best and cleanest way, and we should document it. https://danwalsh.livejournal.com/78373.html
When using Linux Server with SE_Linux installed SE_Linux prevents access to Docker socket.
So there is a missing documentation for SE_Linux
-> default docker-compose from docs was used