Chown error on layers - writing go program to run rootless buildah

a-shirshov commented 2 days ago

Description I am trying to write some go code to run rootless buildah. I was trying to make it work by myself, but it wasn't successful. I read the docs and also did some fmt.Println in source buildah code to understand what is happening. I would really appreciate any advice, because currently i am losing hope, if it is possible. Right now I am stuck at getting chown errors even though I have set ignore_chown_errors=true. In the code down below I have commented out unshare because my goal to run program without using it. In dockerfile I am also trying to give permissions to everything buildah need to make it work. And I copy my own created conf files to container.

Steps to reproduce the issue:

Prepare main.go file (below)

Build it

go build -mod=vendor -tags "containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper linux cni" -o buildahtest main.go

Prepare dockerfile (below)

Run it

podman run --rm -it $(podman build  -q .)

Describe the results you received: panic: copying system image from manifest list: writing blob: adding layer with blob "sha256:ba83bbfca9443648a883d1404b33faa0f5e096a99a2b683e3bbaee8912bca845"/""/"sha256:886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7": creating read-only layer with ID "886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7": chown /var/lib/containers/storage/vfs/dir/886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7: operation not permitted

Describe the results you expected: Builded image, program is working correctly

main.go:

package main

import (
    "context"
    "fmt"
    "os"
    "strconv"

    "github.com/containers/buildah"
    "github.com/containers/common/pkg/config"
    "github.com/containers/storage"
    "github.com/sirupsen/logrus"
)

var logger *logrus.Logger

func setLogger() {
    logger = logrus.New()
    logger.SetLevel(logrus.DebugLevel)
}

func main() {
    setLogger()
    // if buildah.InitReexec() {
    //  return
    // }
    logger.Info("Preparing rootless build environment...")
    //unshare.MaybeReexecUsingUserNamespace(false)
    conf, err := config.Default()
    if err != nil {
        logger.Errorf("Encountered an error: %s", err.Error())
        return
    }

    logger.Info("Prepared config...")

    uid := os.Geteuid()
    uidStr := strconv.Itoa(uid)
    capabilities, err := conf.Capabilities(uidStr, nil, nil)
    logger.Info("Prepared capabilities...")

    buildStoreOptions, err := storage.DefaultStoreOptions()
    logger.Info("Prepared buildStoreOptions...")

    if err != nil {
        logger.Errorf("Encountered an error: %s", err.Error())
        panic(err)
    }

    buildStore, err := storage.GetStore(buildStoreOptions)
    if err != nil {
        logger.Errorf("Encountered an error: %s", err.Error())
        panic(err)
    }
    logger.Info("Prepared build store...")

    defer func(buildStore storage.Store, force bool) {
        _, err := buildStore.Shutdown(force)
        if err != nil {
            logger.Errorf("Encountered an error during build store shutdown: %s", err.Error())
        }
    }(buildStore, false)

    fmt.Println(buildStore)
    fmt.Println(buildStoreOptions)

    builderOpts := buildah.BuilderOptions{
        FromImage:    "docker.io/golang:1.23-bullseye",
        Capabilities: capabilities,
        //GroupAdd:     []string{"keep-groups"},
        Logger:       logger,
    }
    logger.Info("Starting image build...")

    fmt.Println("BuildStore:", buildStore)

    builder, err := buildah.NewBuilder(context.TODO(), buildStore, builderOpts)
    if err != nil {
        panic(err)
    }
    logger.Info("Created builder...")

    defer func(builder *buildah.Builder) {
        err := builder.Delete()
        if err != nil {
            logger.Errorf("Encountered an error during defer: %s", err.Error())
        }
    }(builder)

    logger.Info("Setting UID:GID to 10010:10020")
    builder.SetUser("10010:10020")
    return
}

Dockerfile:

FROM registry.fedoraproject.org/fedora:latest

WORKDIR /app

ENV HOME=/app

COPY config .config

COPY buildahtest .

RUN mkdir /run/lock \
  && chmod -R 777 /run/lock

RUN mkdir /var/lib/containers \
  && chmod -R 777 /var/lib/containers

RUN groupadd appuser -g 10020 && \
useradd appuser -u 10010 -g appuser -M -d /app \
  && chown -R 10010:10020 /app \
  && chown -R 10010:10020 /app/.config \
  && chmod -R 777 /app

RUN usermod --add-subuids 10000-75535 $(whoami)
RUN usermod --add-subgids 10000-75535 $(whoami)

RUN chmod 0644 /etc/subuid && chmod 0644 /etc/subgid

USER 10010:10020

ENTRYPOINT [ "./buildahtest" ]

go.mod

module buildahtest

go 1.22.2

require (
    github.com/containers/buildah v1.37.2
    github.com/containers/common v0.60.2
    github.com/containers/storage v1.55.0
    github.com/sirupsen/logrus v1.9.3
)

require (
    dario.cat/mergo v1.0.0 // indirect
    github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
    github.com/BurntSushi/toml v1.4.0 // indirect
    github.com/Microsoft/go-winio v0.6.2 // indirect
    github.com/Microsoft/hcsshim v0.12.5 // indirect
    github.com/VividCortex/ewma v1.2.0 // indirect
    github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect
    github.com/aead/serpent v0.0.0-20160714141033-fba169763ea6 // indirect
    github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
    github.com/chzyer/readline v1.5.1 // indirect
    github.com/cilium/ebpf v0.11.0 // indirect
    github.com/containerd/cgroups/v3 v3.0.3 // indirect
    github.com/containerd/containerd v1.7.18 // indirect
    github.com/containerd/errdefs v0.1.0 // indirect
    github.com/containerd/log v0.1.0 // indirect
    github.com/containerd/stargz-snapshotter/estargz v0.15.1 // indirect
    github.com/containerd/typeurl/v2 v2.1.1 // indirect
    github.com/containernetworking/cni v1.2.3 // indirect
    github.com/containernetworking/plugins v1.5.1 // indirect
    github.com/containers/image/v5 v5.32.2 // indirect
    github.com/containers/libtrust v0.0.0-20230121012942-c1716e8a8d01 // indirect
    github.com/containers/luksy v0.0.0-20240618143119-a8846e21c08c // indirect
    github.com/containers/ocicrypt v1.2.0 // indirect
    github.com/coreos/go-systemd/v22 v22.5.0 // indirect
    github.com/cyberphone/json-canonicalization v0.0.0-20231217050601-ba74d44ecf5f // indirect
    github.com/cyphar/filepath-securejoin v0.3.1 // indirect
    github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
    github.com/disiqueira/gotree/v3 v3.0.2 // indirect
    github.com/distribution/reference v0.6.0 // indirect
    github.com/docker/distribution v2.8.3+incompatible // indirect
    github.com/docker/docker v27.1.1+incompatible // indirect
    github.com/docker/docker-credential-helpers v0.8.2 // indirect
    github.com/docker/go-connections v0.5.0 // indirect
    github.com/docker/go-units v0.5.0 // indirect
    github.com/felixge/httpsnoop v1.0.4 // indirect
    github.com/fsnotify/fsnotify v1.7.0 // indirect
    github.com/fsouza/go-dockerclient v1.11.1 // indirect
    github.com/go-jose/go-jose/v4 v4.0.2 // indirect
    github.com/go-logr/logr v1.4.2 // indirect
    github.com/go-logr/stdr v1.2.2 // indirect
    github.com/go-openapi/analysis v0.23.0 // indirect
    github.com/go-openapi/errors v0.22.0 // indirect
    github.com/go-openapi/jsonpointer v0.21.0 // indirect
    github.com/go-openapi/jsonreference v0.21.0 // indirect
    github.com/go-openapi/loads v0.22.0 // indirect
    github.com/go-openapi/runtime v0.28.0 // indirect
    github.com/go-openapi/spec v0.21.0 // indirect
    github.com/go-openapi/strfmt v0.23.0 // indirect
    github.com/go-openapi/swag v0.23.0 // indirect
    github.com/go-openapi/validate v0.24.0 // indirect
    github.com/godbus/dbus/v5 v5.1.0 // indirect
    github.com/gogo/protobuf v1.3.2 // indirect
    github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
    github.com/golang/protobuf v1.5.4 // indirect
    github.com/google/go-containerregistry v0.20.0 // indirect
    github.com/google/go-intervals v0.0.2 // indirect
    github.com/google/uuid v1.6.0 // indirect
    github.com/gorilla/mux v1.8.1 // indirect
    github.com/hashicorp/errwrap v1.1.0 // indirect
    github.com/hashicorp/go-multierror v1.1.1 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/jinzhu/copier v0.4.0 // indirect
    github.com/josharian/intern v1.0.0 // indirect
    github.com/json-iterator/go v1.1.12 // indirect
    github.com/klauspost/compress v1.17.9 // indirect
    github.com/klauspost/pgzip v1.2.6 // indirect
    github.com/letsencrypt/boulder v0.0.0-20240418210053-89b07f4543e0 // indirect
    github.com/mailru/easyjson v0.7.7 // indirect
    github.com/manifoldco/promptui v0.9.0 // indirect
    github.com/mattn/go-runewidth v0.0.16 // indirect
    github.com/mattn/go-shellwords v1.0.12 // indirect
    github.com/mattn/go-sqlite3 v1.14.22 // indirect
    github.com/miekg/pkcs11 v1.1.1 // indirect
    github.com/mistifyio/go-zfs/v3 v3.0.1 // indirect
    github.com/mitchellh/mapstructure v1.5.0 // indirect
    github.com/moby/buildkit v0.12.5 // indirect
    github.com/moby/docker-image-spec v1.3.1 // indirect
    github.com/moby/patternmatcher v0.6.0 // indirect
    github.com/moby/sys/mountinfo v0.7.2 // indirect
    github.com/moby/sys/sequential v0.5.0 // indirect
    github.com/moby/sys/user v0.2.0 // indirect
    github.com/moby/term v0.5.0 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/morikuni/aec v1.0.0 // indirect
    github.com/oklog/ulid v1.3.1 // indirect
    github.com/opencontainers/go-digest v1.0.0 // indirect
    github.com/opencontainers/image-spec v1.1.0 // indirect
    github.com/opencontainers/runc v1.1.13 // indirect
    github.com/opencontainers/runtime-spec v1.2.0 // indirect
    github.com/opencontainers/runtime-tools v0.9.1-0.20230914150019-408c51e934dc // indirect
    github.com/opencontainers/selinux v1.11.0 // indirect
    github.com/openshift/imagebuilder v1.2.14 // indirect
    github.com/ostreedev/ostree-go v0.0.0-20210805093236-719684c64e4f // indirect
    github.com/pkg/errors v0.9.1 // indirect
    github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
    github.com/proglottis/gpgme v0.1.3 // indirect
    github.com/rivo/uniseg v0.4.7 // indirect
    github.com/seccomp/libseccomp-golang v0.10.0 // indirect
    github.com/secure-systems-lab/go-securesystemslib v0.8.0 // indirect
    github.com/sigstore/fulcio v1.4.5 // indirect
    github.com/sigstore/rekor v1.3.6 // indirect
    github.com/sigstore/sigstore v1.8.4 // indirect
    github.com/spf13/cobra v1.8.1 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6 // indirect
    github.com/sylabs/sif/v2 v2.18.0 // indirect
    github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect
    github.com/tchap/go-patricia/v2 v2.3.1 // indirect
    github.com/titanous/rocacheck v0.0.0-20171023193734-afe73141d399 // indirect
    github.com/ulikunitz/xz v0.5.12 // indirect
    github.com/vbatts/tar-split v0.11.5 // indirect
    github.com/vbauerster/mpb/v8 v8.7.5 // indirect
    github.com/vishvananda/netlink v1.2.1-beta.2 // indirect
    github.com/vishvananda/netns v0.0.4 // indirect
    go.etcd.io/bbolt v1.3.10 // indirect
    go.mongodb.org/mongo-driver v1.14.0 // indirect
    go.mozilla.org/pkcs7 v0.0.0-20210826202110-33d05740a352 // indirect
    go.opencensus.io v0.24.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
    go.opentelemetry.io/otel v1.24.0 // indirect
    go.opentelemetry.io/otel/metric v1.24.0 // indirect
    go.opentelemetry.io/otel/trace v1.24.0 // indirect
    golang.org/x/crypto v0.26.0 // indirect
    golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect
    golang.org/x/mod v0.20.0 // indirect
    golang.org/x/net v0.28.0 // indirect
    golang.org/x/sync v0.8.0 // indirect
    golang.org/x/sys v0.24.0 // indirect
    golang.org/x/term v0.23.0 // indirect
    golang.org/x/text v0.17.0 // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 // indirect
    google.golang.org/grpc v1.64.1 // indirect
    google.golang.org/protobuf v1.34.1 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
    sigs.k8s.io/yaml v1.4.0 // indirect
    tags.cncf.io/container-device-interface v0.8.0 // indirect
    tags.cncf.io/container-device-interface/specs-go v0.8.0 // indirect
)

Storage.conf

[storage]
driver = "vfs"
runroot = ""
graphroot = "/var/lib/containers/storage"
rootless_storage_path = "$HOME/.local/share/containers/storage"
transient_store = true
[storage.options.vfs]
ignore_chown_errors="true"
[storage.options.overlay]
ignore_chown_errors="true"

Program logs:

INFO[0000] Preparing rootless build environment...      
INFO[0000] Prepared config...                           
INFO[0000] Prepared capabilities...                     
INFO[0000] Prepared buildStoreOptions...                
INFO[0000] Prepared build store...                      
&{/tmp/storage-run-10010/containers vfs [] 0xc00021d0e0 0xc00021d140 /var/lib/containers/storage [vfs.ignore_chown_errors=true]  map[] [] []  1024 65536 0xc0005bb000 [0xc0005bb000] [] 0xc00045dea0 /tmp/storage-run-10010/containers/vfs-locks false true {[]} 0xc000131340 <nil> [] <nil> <nil>}
{/tmp/storage-run-10010/containers /var/lib/containers/storage  /app/.local/share/containers/storage vfs [] [vfs.ignore_chown_errors=true] [] []  0 0 map[] false true}
INFO[0000] Starting image build...                      
BuildStore: &{/tmp/storage-run-10010/containers vfs [] 0xc00021d0e0 0xc00021d140 /var/lib/containers/storage [vfs.ignore_chown_errors=true]  map[] [] []  1024 65536 0xc0005bb000 [0xc0005bb000] [] 0xc00045dea0 /tmp/storage-run-10010/containers/vfs-locks false true {[]} 0xc000131340 <nil> [] <nil> <nil>}
panic: copying system image from manifest list: writing blob: adding layer with blob "sha256:ba83bbfca9443648a883d1404b33faa0f5e096a99a2b683e3bbaee8912bca845"/""/"sha256:886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7": creating read-only layer with ID "886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7": chown /var/lib/containers/storage/vfs/dir/886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7: operation not permitted

goroutine 1 [running]:
main.main()
        /go/src/main.go:79 +0x925

a-shirshov commented 2 days ago

Maybe it would be helpful - my project structure

rhatdan commented 2 days ago

On the outer podman try --security-opt unmask=/proc

a-shirshov commented 2 days ago

Okay. I tried

podman run --rm -it --security-opt unmask=/proc $(podman build  -q .)

and

podman run --rm -it --security-opt unmask=/proc/* $(podman build  -q .)

Still the same problem:

INFO[0000] Preparing rootless build environment...      
INFO[0000] Prepared config...                           
INFO[0000] Prepared capabilities...                     
INFO[0000] Prepared buildStoreOptions...                
INFO[0000] Prepared build store...                      
INFO[0000] Starting image build...                      
panic: copying system image from manifest list: writing blob: adding layer with blob "sha256:ba83bbfca9443648a883d1404b33faa0f5e096a99a2b683e3bbaee8912bca845"/""/"sha256:886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7": creating read-only layer with ID "886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7": chown /var/lib/containers/storage/vfs/dir/886960d374e8650394e753ad35fbc38498e4b58a204d8d2243006512033f0ba7: operation not permitted

nalind commented 2 days ago

The InitReexec() call, or an equivalent, is still required - it's used for doing work in subprocesses. If the application isn't running as UID 0 in the namespace in which it's started, it needs to create a new namespace in which it can be "UID 0" order to be able to use mount(), which is required for handling RUN instructions. The "ignore_chown_errors" option is used when pulling base images, and is only consulted when writing an item from a layer blob into storage. The error you're seeing is being hit on a directory that is part of how the storage library manages things, likely because the application isn't UID 0 in its user namespace.

a-shirshov commented 2 days ago

So as I understand - if the operating system has max_user_namespaces set to 0 - it is not possible to run rootless Buildah? When unshare and InitReexec were not commented - my program worked fine. But one day I got error like - user namespaces not enabled so now I am trying to run it without unshare. I am afraid about security - UID 0 sounds not nice, but I am not an expert. Are user namespaces safe?

rhatdan commented 2 days ago

Container tools need to mount, the only way for a rootless user to mount is to create a user namespace. So setting no user namespaces means you can not run rootless containers.

a-shirshov commented 2 days ago

Got it. Thanks for help. If there are any new questions/problems - I'll create a new issue or write another comment here. This one can be closed.

containers / buildah

Chown error on layers - writing go program to run rootless buildah #5744