hashicorp / go-secure-stdlib

Mozilla Public License 2.0
64 stars 24 forks source link

plugincontainer: Fix rootless tests #123

Open tomhjp opened 6 months ago

tomhjp commented 6 months ago

It seems like gVisor fixed a bug where a file's capabilities weren't being properly applied when creating the process: https://github.com/google/gvisor/commit/586c38d70081b13b2ed494cef48e99b93956843e

I haven't fully finished investigating yet, but I think it's the above commit that caused the tests to start failing once it got released and used in CI.

tomhjp commented 5 months ago

In case it's helpful to anyone picking this up, I wrote a lima config file to help quickly spin up a VM that is ready to test this:

default.yaml

```yaml # Based on https://github.com/lima-vm/lima/blob/9d31f2a7ee7c6a699298c66d80f26ad2d39fae76/examples/docker.yaml # A template to use Docker instead of containerd & nerdctl # $ limactl start ./docker.yaml # $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine # To run `docker` on the host (assumes docker-cli is installed): # $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock') # $ docker ... # This template requires Lima v0.8.0 or later images: # Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months. - location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-amd64.img" arch: "x86_64" digest: "sha256:054db2d88c454bb0ad8dfd8883955e3946b57d2b0bf0d023f3ade3c93cdd14e5" - location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-arm64.img" arch: "aarch64" digest: "sha256:eafa7742ce5ff109222ea313d31ea366d587b4e89b900b11d8285ae775dfe8c3" cpus: 8 memory: 16GiB mounts: - location: "~" writable: true - location: "/tmp/lima" writable: true # containerd is managed by Docker, not by Lima, so the values are set to false here. containerd: system: false user: false provision: - mode: system script: | #!/bin/bash apt-get install -y jq make - mode: system # This script defines the host.docker.internal hostname when hostResolver is disabled. # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts. # Names defined in /etc/hosts inside the VM are not resolved inside containers when # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later). script: | #!/bin/sh sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts # Install docker packages - mode: system script: | #!/bin/bash set -eux -o pipefail command -v docker >/dev/null 2>&1 && exit 0 export DEBIAN_FRONTEND=noninteractive curl -fsSL https://get.docker.com | sh # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless systemctl disable --now docker apt-get install -y uidmap dbus-user-session # Setup rootless docker - mode: user script: | #!/bin/bash set -eux -o pipefail systemctl --user start dbus dockerd-rootless-setuptool.sh install docker context use rootless # Install gVisor (runsc) - mode: user script: | mkdir -p "$HOME/bin" ( set -e ARCH="$(uname -m)" URL="https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}" wget --quiet "${URL}/runsc" "${URL}/runsc.sha512" \ "${URL}/containerd-shim-runsc-v1" "${URL}/containerd-shim-runsc-v1.sha512" sha512sum -c runsc.sha512 \ -c containerd-shim-runsc-v1.sha512 rm -f -- *.sha512 chmod a+rx runsc containerd-shim-runsc-v1 mv runsc containerd-shim-runsc-v1 "$HOME/bin" ) mkdir -p ~/.config/docker/ tee ~/.config/docker/daemon.json <> "${HOME}/.bashrc" probes: - script: | #!/bin/bash set -eux -o pipefail if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then echo >&2 "docker is not installed yet" exit 1 fi if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then echo >&2 "rootlesskit (used by rootless docker) is not running" exit 1 fi hint: See "/var/log/cloud-init-output.log". in the guest hostResolver: # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also # resolve inside containers, and not just inside the VM itself. hosts: host.docker.internal: host.lima.internal portForwards: - guestSocket: "/run/user/{{.UID}}/docker.sock" hostSocket: "{{.Dir}}/sock/docker.sock" message: | cd to the Vault repo on your host, run `lima` to enter the VM, and then: make dev # Optional: Pull all the dependencies to populate the VM's Go module cache etc. go test -v -count=1 -run="^TestExternalPluginInContainer_MountAndUnmount/rootless_runsc$" github.com/hashicorp/vault/vault ```