Consul docker cannot run with "no-new-privileges" option

[canhnt]

The consul:1.2.3 Docker image can run as non-root user (after #102). However in a secure environment requiring no-new-privileges Docker option, it cannot ping which is used for service discovery.

Reproducible steps:

docker run -ti -u consul --cap-add NET_RAW --cap-add NET_ADMIN  --security-opt=no-new-privileges  consul:1.2.3 sh
$ ping 8.8.8.8
ping: icmp open socket: Operation not permitted

In K8s, the no-new-privileges option is set by using allowPrivilegeEscalation=false, which is recommended in https://kubernetes.io/docs/tasks/configure-pod-container/security-context/. Running Consul for now is not possible in such K8s cluster with such hardened security settings.

[canhnt]

I have tried to fix by removing s attribute from /bin/ping and add capabilities to it, but it still failed:

RUN chmod -s /bin/ping && \
    setcap cap_net_admin,cap_net_raw=+ep /bin/ping

[mkeeler]

@canhnt The no-new-privileges option is going to prevent gaining capabilities through execution of a binary that you ran setcap on.

This site explains it pretty well: https://www.projectatomic.io/blog/2016/03/no-new-privs-docker/

In particular from that site, it says:

Processes with no_new_privs are not allowed to change uid/gid or gain any other capabilities, even if the process executes setuid binaries or executables with file capability bits set.

It would seem that no-new-privileges completely breaks all setuid and setcap enabled binary functionality. When executing the container as the 'consul' user you are effectively running with no capabilities but setuid and setcap'ed binaries can enable some privileged functionality. no-new-privileges prevents gaining any capabilities in this manner or becoming uid 0.

Therefore there is nothing that can be done. You have 2 options:

1. Execute the container without no-new-privileges
2. Execute the container as root and just drop all unnecessary capabilities (docker adds a bunch of default ones). You could try running as root with --cap-drop=ALL --cap-add NET_RAW --cap-add NET_ADMIN and then see if any other problems crop up.

[canhnt]

@mkeeler : Your comment is exactly correct. Then can we replace the service discovery by using ping with a different one, like netcat that does not need NET_RAW or NET_ADMIN capabilities, then the problem is solved?

The no-new-privileges is really important for a secure Docker environment, which prevented any executable files having +s attribute in Docker images can run as root. While we cannot scan through all files in all running Docker images, enabling allowPrivilegeEscalation=false, thus no-new-privileges is the way to go.

[mkeeler]

Yes, assuming you were previously using a script check that executed ping you could replace that with either a TCP check or a script check that executes something like nc -z -w 3 <host> <port>.

This check is fundamentally different thought in that it checks that a service is bound and listening on that port instead of the hosts networking is active.

Also of note is that depending upon the application bound to the port being checked you could cause errors in the logs for opening and then immediately closing the connection without transmitting any data.