Can't run image with vault user

[a8j8i8t8]

When I run this image with vault user I'm getting below error. unable to set CAP_SETFCAP effective capability: Operation not permitted

[sharul121]

Same here. Anybody solved this?

[FilBot3]

@a8j8i8t8 can you provide your Dockerfile or docker-compose.yml you're using?

[a8j8i8t8]

@predatorian3 I used following command to start vault on my machine. docker run -it --rm --cap-add=IPC_LOCK --user vault --name=dev-vault vault It failed with error I mentioned earlier. Cheers.

[arthurk]

Works fine on my system:

$ docker run --rm -it --cap-add=IPC_LOCK --user vault vault:1.1.3 sh
/ $ id
uid=100(vault) gid=1000(vault) groups=1000(vault)

Why doesn't the image run with the vault user by default?

[jasonodonnell]

@arthurk the Vault container is using dumb-init to handle reaping zombie processes and forward signals to all processes in the container. The Vault server process itself is owned by the vault user:

/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 {docker-entrypoi} /usr/bin/dumb-init /bin/sh /usr/local/bin/docker-entrypoint.sh server -dev
    7 vault     0:00 vault server -config=/vault/config -dev-root-token-id= -dev-listen-address=0.0.0.0:8200 -dev
   60 root      0:00 /bin/sh
   68 root      0:00 ps -ef

[varunrocks6]

By setting below env vars, I was able to run the image with vault user:

`env:

• name: "SKIP_CHOWN" value: "true"
• name: "SKIP_SETCAP" value: "true"`

[seanw2020]

Varunrocks6's suggestion disables memory locking, exposing the vault Barrier to be written to disk, which is a potential security vulnerability. As Vault Configuration says:

Disabling mlock is not recommended in production

Separately, Vault's Production Hardening doc says:

Don't Run as Root. Vault is designed to run as an unprivileged user

Well, the vault Go binary does NOT run as root, but its container does. That is a potential security vulnerability. OpenShift disallows it by default. And some security scans will fail builds because of it. Openshift's documentation says:

Some container images (examples: postgres and redis) require root access and have certain expectations about how volumes are owned. For these images, add the service account to the anyuid SCC.

But that shouldn't be necessary:

... applications, databases, load balancers, etc. shouldn't ever be run as root

In short, how can we run the vault container without root privileges?

UPDATE October 2019: If you're installing Vault in Kubernetes, it seems reasonable to run with vault user instead of root and to disable mlock because Vault's configuration documentation says:

Disabling mlock is not recommended unless the systems running Vault only use encrypted swap or do not use swap at all.

And Kubernetes nodes require Kubelet which requires disabling swap:

The Kubelet won’t run if swap is enabled, so you’ll disable it with the following command:

swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab

Sadly, setting USER as vault still causes this issue (unable to set CAP_SETFCAP effective capability: Operation not permitted), even without --cap-add=IPC_LOCK.

[nicolasbelanger]

Got it working by setting - name: "SKIP_SETCAP" value: "false" only. How bad/good is it as a compromise?

[symbiont-andrew-leinung]

Running into a similar issue, albeit this time with a read-only filesystem - you can't add linux capabilities to read-only files such as the vault binary.

Would it be possible to run the setcap command on the vault binary as part of the image build, instead of at runtime in the docker-entrypoint.sh script?

[th3n3wguy]

By setting below env vars, I was able to run the image with vault user:

env: - name: "SKIP_CHOWN" value: "true" - name: "SKIP_SETCAP" value: "true"

I'm not sure why, but this is exactly what got the Vault server running for me when running the Docker image inside of Kubernetes.

Thanks to @varunrocks6 for the info!

[amitdeshmukh]

@seanw2020 is right.

Disabling mlock is not the solution for production usage. Nor is running the vault docker container as root (which is the default behaviour).

I got around this by:

1. Installing libcap on the vault:latest image which is alpine based:

RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories RUN apk update RUN apk add libcap

2. Setting the capabilities required by vault to run as a non-root user

RUN /usr/sbin/setcap cap_ipc_lock=+ep /bin/vault

3. The above 2 steps are performed as the root user at the start of the Dockerfile, after which I am setting the container to run as the user vault

ARG USER=vault USER vault

Would love feedback on this approach, security concerns etc.

[jkevlin2]

I tried this

FROM vault:latest RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories RUN apk update RUN apk add libcap RUN /usr/sbin/setcap cap_ipc_lock=+ep /bin/vaul ARG USER=vault USER vault

and got the following error when attempting to run this as a vault agent sidecar

kubectl logs vault-agent-example vault-agent-auth /usr/local/bin/docker-entrypoint.sh: exec: line 104: agent: not found

Did I miss a step?

[hongkongkiwi]

@seanw2020 is right.

Disabling mlock is not the solution for production usage. Nor is running the vault docker container as root (which is the default behaviour).

I got around this by:

1. Installing libcap on the vault:latest image which is alpine based:

RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories RUN apk update RUN apk add libcap

1. Setting the capabilities required by vault to run as a non-root user

RUN /usr/sbin/setcap cap_ipc_lock=+ep /bin/vault

1. The above 2 steps are performed as the root user at the start of the Dockerfile, after which I am setting the container to run as the user vault

ARG USER=vault USER vault

Would love feedback on this approach, security concerns etc.

I tried this solution but I still have the same setcap error when starting the container in Docker.

unable to set CAP_SETFCAP effective capability: Operation not permitted

[wsams]

@varunrocks6 SKIP_CHOWN="true" works for me as well in Kubernetes, but I'd also like to understand why and if this should be avoided. I believe this is where it's used https://github.com/hashicorp/docker-vault/blob/master/0.X/docker-entrypoint.sh#L71

Update: I'm unclear what happened here. I was on 1.3.3 in Kubernetes with 3 replicas. After performing a Helm upgrade directly to 1.6.2 I ran into the error mentioned in the description of this issue. I only deleted one of the standby nodes. Then I changed the tag to 1.5.3 and added SKIP_CHOWN="TRUE", ran helm upgrade, deleted the same pod that was deleted before and it started fine. Then I changed the tag to 1.6.3, removed SKIP_CHOWN="TRUE", ran helm upgrade, deleted the same pod and it started fine. Then I deleted the other standby, then active node and it appears Vault 1.6.3 is running fine on Kubernetes.

[cortopy]

It seems that after reading this issue we have two choices: either run a privileged container as root or disabling mlock.

This choice is startling for a tool which is supposed to raise the security standards of a kubernetes cluster

Since there hasn't been any updates for a while I just wanted to add my cents in 2021. In the case of kubernetes, disabling mlock seems safe because:

1. What @seanw2020 already said about swap being disabled for kubelet
2. If choosing the raft storage engine, mlock is not required either

So for me vault with no root and raft seems like the best option in kubernetes

[austin-agronick]

Here is a working solution for vault 1.14.3, tested on dietpi OS running as vault user with mlock enabled: docker-compose.yml:

version: '3'
services:
  vault:
    build:
      context: .
      dockerfile: Dockerfile-vault
    container_name: hashicorp-vault
    user: "vault"
    volumes:
      - ./vault/logs:/vault/logs
      - ./vault/config/:/vault/config.d
      - ./vault/data:/vault/data
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
      VAULT_ADDR: "http://0.0.0.0:8200"
      SKIP_SETCAP: true  # handled in dockerfile
    cap_add:
      - IPC_LOCK
    command: "server -config /vault/config.d/config.hcl"

Dockerfile-vault:

FROM hashicorp/vault
RUN setcap cap_ipc_lock=+ep /bin/vault

Setup:

1. create file and empty directory: ./vault/config/config.hcl,./vault/data info on config.hcl contents: https://developer.hashicorp.com/vault/docs/configuration
2. run on host: sudo chown -R 100:100 ./vault; sudo usermod -aG users dietpi security note: users group is default group does not do anything, see https://unix.stackexchange.com/a/326782
3. run on host: sudo chmod 600 /vault/data/vault.db; docker compose up disregard errors that go away after step 3: core: security barrier not initialized core: seal configuration missing, not initialized
4. initialize vault at http://<listener "tcp" address from config.hcl>:8200/ui in your browser