Toolbox containers cannot be entered in fresh install of Fedora Silverblue 40

baldemoto commented 2 months ago

Issue Description

After creating a toolbox container (toolbox create env), attempting to enter it with toolbox enter env fails.

Steps to reproduce the issue

Install Fedora Linux 40.20240828.0 (Silverblue)
Run podman system reset
Create toolbox (toolbox create testenv)
Start toolbox with toolbox enter testenv

Describe the results you received

I expected toolbox to start without any further issues.

Describe the results you expected

After downloading the Fedora 40 image registry.fedoraproject.org/fedora-toolbox:40 and creating the container, I receive the following error after attempting to enter the environment: Error: failed to start container testenv

Further debug information is attached below

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 88.54
    systemPercent: 5.68
    userPercent: 5.78
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: silverblue
    version: "40"
  eventLogger: journald
  freeLocks: 2047
  hostname: bee-laptop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.10.6-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 723128320
  memTotal: 65098338304
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.1-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240821.g1d6142f-1.fc40.x86_64
    version: |
      pasta 0^20240821.g1d6142f-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8587571200
  swapTotal: 8589930496
  uptime: 1h 48m 54.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/$USER/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/$USER/.local/share/containers/storage
  graphRootAllocated: 1998678130688
  graphRootUsed: 397472518144
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/$USER/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Tue Aug 20 17:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

This is a fresh install of Fedora Silverblue. The actions I have performed include:

Create 2 users and add them to a user group to grant equal permission to their home folders (chmod 770)
Install rpm-ostree packages (nordvpn-3.18.4-1.x86_64, rpmfusion free and nonfree packages , mesa-va-drivers-freeworld-24.1.6-1.fc40.x86_64, Cosmic DE)

This is a very similar issue to #13807 , though that one was seemingly resolved on its own.

Additionally, running podman start --attach testenv outputs the following error:

Error: unable to start container d4ce9e64b3e957bfb822c6e12aebce65e908027c04129adbd6026f7fbefe0029: crun: make `/var/home/$USER/.local/share/containers/storage/overlay/d73761d0ac5b92d44f8c2b3a6922278ac4beda190ad4da625cb7e790b46c3abb/merged` private: Permission denied: OCI permission denied

The following is the output of toolbox enver -vvvv testenv:

DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Looking up sub-GID and sub-UID ranges for user $USER 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/$USER/.config/toolbox 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called version.PersistentPreRunE(podman --log-level debug version --format json) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/$USER/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/$USER/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/$USER/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Called version.PersistentPostRunE(podman --log-level debug version --format json) 
DEBU[0000] Shutting down engines                        
DEBU Current Podman version is 5.2.2              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 5.2.2                  
DEBU Migration not needed: Podman version 5.2.2 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /var/home/$USER/.config/containers/toolbox.conf not found 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-40'               
DEBU Image: 'fedora-toolbox:40'                   
DEBU Release: '40'                                
DEBU Resolving container and image names          
DEBU Container: 'testenv'                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'testenv'                         
DEBU Image: 'fedora-toolbox:40'                   
DEBU Release: '40'                                
DEBU Checking if container testenv exists         
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exists.PersistentPreRunE(podman --log-level debug container exists testenv) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/$USER/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/$USER/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/$USER/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Called exists.PersistentPostRunE(podman --log-level debug container exists testenv) 
DEBU[0000] Shutting down engines                        
INFO[0000] Received shutdown.Stop(), terminating!        PID=39216
DEBU Inspecting mounts of container testenv       
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container testenv) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/$USER/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/$USER/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/$USER/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Looking up image "9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" in local containers storage 
DEBU[0000] Trying "9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" ... 
DEBU[0000] parsed reference into "[overlay@/var/home/$USER/.local/share/containers/storage+/run/user/1000/containers]@9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" 
DEBU[0000] Found image "9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" as "9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" in local containers storage 
DEBU[0000] Found image "9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" as "9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4" in local containers storage ([overlay@/var/home/$USER/.local/share/containers/storage+/run/user/1000/containers]@9a593cd30adc9db4ff5c064f40ab7be81ec30ada138765f884b815a53948aab4) 
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container testenv) 
DEBU[0000] Shutting down engines                        
INFO[0000] Received shutdown.Stop(), terminating!        PID=39231
DEBU Starting container testenv                   
Error: failed to start container testenv

baldemoto commented 2 months ago

Update: After a few reinstalls of Silverblue, I have narrowed down the potential cause of the issue.

Toolbox works as expected after a fresh reinstall
After moving my old /home directories to the fresh install INCLUDING hidden directories, toolbox recapitulates the issue on both users
After moving the directories to the fresh install WITHOUT hidden directories, toolbox still works as expected
After installing/setting up flatpaks and rpm-ostree packages, as well as user group permissions for home folders, the issue returns on both users. Rolling back rpm-ostree does not fix the issue

My suspicion is there might be a conflict within some hidden directory in the /home folders or within home directory permissions themselves. I will investigate further later.

Luap99 commented 2 months ago

Looks like toolbox just eats up all stderr from the start command and never logs the actual podman error so you likely want to fix this first. https://github.com/containers/toolbox/blob/b684b190d1d931434d0d7d63c841973101dfb928/src/cmd/run.go#L976-L984

Podman stores all the images, containers,etc... under ~/.local/share/containers/ and yes moving this to another computer will not work if your UID is changed, /etc/sub{u,g}id were changed or in general some paths where changed that now trip over over our db validation code.

github-actions[bot] commented 1 month ago

A friendly reminder that this issue had no activity for 30 days.

containers / podman