containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
24.01k stars 2.43k forks source link

homeassistant doesn't start with `--userns auto` #16652

Closed M1cha closed 2 years ago

M1cha commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description It looks like there's something wrong with the mounts for this specific container. Specifically when the runtime tries to write files like .containerenv or resolv.conf. The error messages differ between runtimes. Also, this only happens with podman running as root.

crun + no network:

# podman run --runtime crun --net none --userns auto --rm -it ghcr.io/home-assistant/home-assistant:stable
Error: crun: open `/var/lib/containers/storage/overlay/f21294f531a0d2df0020992ed20e2b30e60a23063f0f1ce91f7a6fa2640ef134/merged/.containerenv`: No such file or directory: OCI runtime attempted to invoke a command that was not found

runc + no network:

# podman run --runtime runc --net none --userns auto --rm -it ghcr.io/home-assistant/home-assistant:stable
Error: runc: runc create failed: unable to start container process: error during container init: error mounting "/run/containers/storage/overlay-containers/c1141f52b106403557c1a0fa1bb78e2d05b232a48cad15e527c1926f6db17497/userdata/.containerenv" to rootfs at "/run/.containerenv": open /var/lib/containers/storage/overlay/c7db5a8b46494e51b7e8e72e695ae0a06a082ac31fffa1323fa0c262b15486bd/merged/run/.containerenv: permission denied: OCI permission denied

crun + default bridge:

# podman run --runtime crun --userns auto --rm -it ghcr.io/home-assistant/home-assistant:stable
Error: crun: open `/var/lib/containers/storage/overlay/01a079b549b229a9398bfcd6306c7fbbc8c795b6b70e25904481f43319b4da75/merged/.containerenv`: No such file or directory: OCI runtime attempted to invoke a command that was not found

runc + default bridge:

# podman run --runtime runc --userns auto --rm -it ghcr.io/home-assistant/home-assistant:stable
Error: runc: runc create failed: unable to start container process: error during container init: error mounting "/run/containers/storage/overlay-containers/6f09d411047f4942999c57eeb4c39962348cb1344686afa8e16f33bee52efab2/userdata/resolv.conf" to rootfs at "/etc/resolv.conf": open /var/lib/containers/storage/overlay/57744b1a8449467952bfa095b8c29b372353421caf61749369f01fa23d347a10/merged/etc/resolv.conf: permission denied: OCI permission denied

Output of podman version:

Client:       Podman Engine
Version:      4.3.0
API Version:  4.3.0
Go Version:   go1.18.7
Built:        Fri Oct 21 08:12:52 2022
OS/Arch:      linux/arm64

Output of podman info:

host:
  arch: arm64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-3.fc36.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: '
  cpuUtilization:
    idlePercent: 98.97
    systemPercent: 0.59
    userPercent: 0.44
  cpus: 6
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: homeserver
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.0.5-200.fc36.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 3275083776
  memTotal: 3994021888
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.6-2.fc36.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.6
      commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.aarch64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 0h 43m 36.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 2
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 474196275200
  graphRootUsed: 3013447680
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.0
  Built: 1666339972
  BuiltTime: Fri Oct 21 08:12:52 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.3.0

Package info (output of rpm -q podman:

podman-4.3.0-2.fc36.aarch64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): Fedora CoreOS aarch64 bare metal.

giuseppe commented 2 years ago

I can reproduce it with 4.3.0 but it is fixed upstream with 6fe64591d630f6f532f6233c4ea8c97f79809797

M1cha commented 2 years ago

@giuseppe When will the next release be? Technically this is a bug and should be fixed in 4.3 but given that we don't know what fixed it and that commit updated a lot of dependencies it'll probably only arrive in the next minor or major release which can be months apart.

mheon commented 2 years ago

Target is late January, with RCs beginning in early January.

M1cha commented 2 years ago

In that case I'll want to invest time into figuring out how to build podman for fedora coreOS (rpm-ostree).

a 30minute google search lets me believe that this is non-trivial - let alone cross-compiling for aarch64. There was a tutorial but it seems very outdated: https://podman.io/blogs/2018/11/19/build_libpod-container-images.html

M1cha commented 2 years ago

I know this is off-topic but since I've already mentioned it, here are the steps to cross-compile from x64 to aarch64 on fedora

This takes a long time to build since qemu-user isn't exactly known to be fast :shrug: Create a fedora 37 x86_64 VM, I tried toolbox but it only seems to work without cross-compiling.

$ sudo dnf install git qemu-user-static fedora-packager rpmdevtools gcc
$ rpmdev-setuptree
$ git clone https://src.fedoraproject.org/rpms/podman.git
$ # apply the patch I've attached
$ cd podman
$ spectool -agR podman.spec
$ rpmbuild -bs podman.spec
$ sudo mock -r /etc/mock/fedora-37-aarch64.cfg --rebuild ~/rpmbuild/SRPMS/podman-4.4.0-1.fc37.src.rpm

based on 86969acc658495a3b6f7de27c57dc4b6e60c82dd

diff --git a/podman.spec b/podman.spec
index 97d7fa4..adab288 100644
--- a/podman.spec
+++ b/podman.spec
@@ -29,7 +29,7 @@
 %global git_gvproxy https://%{import_path_gvproxy}
 %global commit_gvproxy fdc231ae7b8fe1aec4cf0b8777274fa21b70d789

-%global built_tag v4.3.1
+%global built_tag v4.4.0
 %global built_tag_strip %(b=%{built_tag}; echo ${b:1})
 %global gen_version %(b=%{built_tag_strip}; echo ${b/-/"~"})

@@ -42,7 +42,7 @@ ExclusiveArch: %{golang_arches}
 Summary: Manage Pods, Containers and Container Images
 URL: https://%{name}.io/
 # All SourceN files fetched from upstream
-Source0: %{git0}/archive/%{built_tag}.tar.gz
+Source0: %{git0}/archive/480c7fbf5361f3bd8c1ed81fe4b9910c5c73b186.tar.gz
 Source1: %{git_plugins}/archive/%{commit_plugins}/%{repo_plugins}-%{commit_plugins}.tar.gz
 Source2: %{git_gvproxy}/archive/%{commit_gvproxy}/%{repo_gvproxy}-%{commit_gvproxy}.tar.gz
 Provides: %{name}-manpages = %{epoch}:%{version}-%{release}
@@ -286,8 +286,9 @@ It is based on the network stack of gVisor. Compared to libslirp,
 gvisor-tap-vsock brings a configurable DNS server and
 dynamic port forwarding.

-%prep
-%autosetup -Sgit -n %{name}-%{built_tag_strip}
+%_topdir /tmp/rpmbuild
+%prep -n %{name}-480c7fbf5361f3bd8c1ed81fe4b9910c5c73b186
+%autosetup -Sgit -n %{name}-480c7fbf5361f3bd8c1ed81fe4b9910c5c73b186
 sed -i 's;@@PODMAN@@\;$(BINDIR);@@PODMAN@@\;%{_bindir};' Makefile

 # untar dnsname
@@ -329,6 +330,8 @@ export BUILDTAGS="seccomp exclude_graphdriver_devicemapper $(hack/btrfs_installe

 %gobuild -o bin/%{name} %{import_path}/cmd/%{name}

+%gobuild -o bin/quadlet %{import_path}/cmd/quadlet
+
 # build %%{name}-remote
 export BUILDTAGS="seccomp exclude_graphdriver_devicemapper exclude_graphdriver_btrfs btrfs_noversion $(hack/selinux_tag.sh) $(hack/systemd_tag.sh) $(hack/libsubid_tag.sh) remote"
 %gobuild -o bin/%{name}-remote %{import_path}/cmd/%{name}
@@ -408,6 +411,9 @@ rm -f %{buildroot}%{_datadir}/user-tmpfiles.d/%{name}-docker.conf
 %{_bindir}/%{name}
 %dir %{_libexecdir}/%{name}
 %{_libexecdir}/%{name}/rootlessport
+%{_libexecdir}/%{name}/quadlet
+%{_systemdgeneratordir}/podman-system-generator
+%{_systemdusergeneratordir}/podman-user-generator
 %{_datadir}/bash-completion/completions/%{name}
 # By "owning" the site-functions dir, we don't need to Require zsh
 %dir %{_datadir}/zsh/site-functions