Fails to use storage.conf settings

jkirkpatrick commented 9 months ago

Issue Description

I do not want to store gigs of data in my user home directory on a server; a separate volume (/data) has been created with proper permissions, fstab, etc. and works fine.

The default settings ignore any changes in storage.conf for the correct graphroot path.

Steps to reproduce the issue

Edit /home/ubuntu/.config/containers/storage.conf

graphroot = "/data/containers/storage"

(optional) Logout or Reboot

Describe the results you received

$ podman info

...
store:
  configFile: /home/ubuntu/.config/containers/storage.conf
...
  graphRoot: /home/ubuntu/.local/share/containers/storage
...

Describe the results you expected

$ podman info

store:
  configFile: /home/ubuntu/.config/containers/storage.conf
...
  graphRoot: /data/containers/storage


### podman info output

```yaml
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.8-0ubuntu22.04+obs16.13_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 99.26
    systemPercent: 0.41
    userPercent: 0.34
  cpus: 2
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2048
  hostname: ip-10-0-2-78
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.2.0-1012-aws
  linkmode: dynamic
  logDriver: journald
  memFree: 3472150528
  memTotal: 4094164992
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.3.0
  ociRuntime:
    name: crun
    package: crun_101:1.9-0ubuntu22.04+obs59.11_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.2-0ubuntu22.04+obs12.3_amd64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 19m 53.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/ubuntu/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /home/ubuntu/.local/share/containers/storage
  graphRootAllocated: 16455237632
  graphRootUsed: 7035518976
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ubuntu/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

AWS EC2 Ubuntu w/ latest updates (via apt) and latest upstream podman

AMI ID ami-053b0d53c279acc90

AMI name ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230516

uname -a Linux ip-#-#-#-# 6.2.0-1012-aws #12~22.04.1-Ubuntu SMP Thu Sep 7 14:01:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

jkirkpatrick commented 9 months ago

rhatdan commented 9 months ago

Set rootless_storage_path for rootless users /usr/share/containers/storage.conf and /etc/containers/storage.conf really configure your containers for rootful mode.

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

jkirkpatrick commented 9 months ago

I didn't include it in the bug report (as I should) but the rootless_storage_path parameter was also enabled set to the same value (/data/containers/storage)

I ended up creating a new server and reinstalling podman to test.

There are additional issues with the upgrade from the distro 3.x version to the latest 4.x. Once the storage issue is resolved I will try to figure out what happened. Both servers were just instantiated; the first had apt-get install podman and then had the upgrade according to instructions.

rhatdan commented 9 months ago

Did you actually get the rootless_storage_path to work? and you only had issues with upgrade?

github-actions[bot] commented 8 months ago

A friendly reminder that this issue had no activity for 30 days.

jkirkpatrick commented 8 months ago

I can confirm that the "nuke from orbit and start from nothing" approach works as expected.

On Sat, Oct 21, 2023 at 8:07 PM github-actions[bot] < @.***> wrote:

A friendly reminder that this issue had no activity for 30 days.

— Reply to this email directly, view it on GitHub https://github.com/containers/podman/issues/20068#issuecomment-1773950490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIYUB7KDZN6YRSHAVLHRZTYARPTLAVCNFSM6AAAAAA5AOEP6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTHE2TANBZGA . You are receiving this because you authored the thread.Message ID: @.***>

github-actions[bot] commented 7 months ago

A friendly reminder that this issue had no activity for 30 days.

MadMud commented 3 months ago

can confirm this issue arises on podman 5.0.1.

by experimentation I could fix it by deleting storage/db.sql YMMV

My best guess is, that the contents of storage.conf get read and duplicated to the database. From @rhatdan 's answer this was never meant to happen ("rootless_storage_path" replaces "graph_root")

The same can be observed with runroot (where there is no "rootless_" replacement)

From where I sit, this seems to be an unintended consequence overall. Perhaps we should avoid configuring storage.conf altogether?

rhatdan commented 3 months ago

@mheon PTAL

MadMud commented 3 months ago

one small adendum: https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md states:

In rootless Podman, certain fields in /etc/containers/storage.conf are ignored. These fields are: (...) graphroot & runroot (highlight is mine)

that part of the docs is either not correct, or OP an I are expirencing a bug.

mheon commented 3 months ago

This is our built-in path overriding. We do not allow changing critical paths on an existing installation as it is guaranteed to cause existing containers to fail. If you have no containers, podman system reset will remove the database and reset the paths to the ones you want to use.

As for the second bug, graphroot and runroot not being ignored - that's definitely acting as the docs say (just handled a bug in the opposite direction) so likely to be an environment-related bug?

jkirkpatrick commented 2 months ago

@mheon If a nuke and re-install fixes the problem, it can't be the enviro since the installer and day-to-day usage shouldn't modify the enviro. If the enviro is being modified by the installer ... there's no chance of podman ever being used for reproducible builds. And since podman is clearly able to create reproducible builds, I think the source of the bug is safely within podman config tooling, and not the enviro.

containers / podman