SQLite backend places the db file in a wrong directory.

[mskarbek]

Issue Description

When you specify separate static_dir (/var/lib/libpod in my case) boltdb backed uses it to store podman data. After upgrading to 4.5.1-4.el9 from CeontOS Stream and changing the db backend to SQLite, new db file is placed in /var/lib/containers/storage ignoring static_dir configuration.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman run -it --rm ubi9/ubi:9.2
2. dnf install https://kojihub.stream.centos.org/kojifiles/packages/podman/4.5.1/4.el9/x86_64/podman-4.5.1-4.el9.x86_64.rpm
3. cp /usr/share/containers/containers.conf /etc/containers/
4. sed -i 's/#static_dir.*/static_dir = "\/var\/lib\/libpod"/' /etc/containers/containers.conf
5. podman info|grep databaseBackend -> databaseBackend: boltdb
6. ls /var/lib/libpod/ -> bolt_state.db
7. rm -rf /var/lib/libpod/*
8. sed -i 's/\[engine\]/\[engine\]\ndatabase_backend = "sqlite"/' /etc/containers/containers.conf
9. podman info|grep databaseBackend -> databaseBackend: sqlite
10. ls /var/lib/libpod/ -> empty
11. ls /var/lib/containers/storage/ -> db.sql, WAL and other SQLite related files

Describe the results you received

SQLite backed places db file in a wrong directory ignoring static_dir configuration.

Describe the results you expected

SQLite backend correctly places Podman state in configured directory.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: fab2fef7227d2dc16478d29f1185953f81451702'
  cpuUtilization:
    idlePercent: 99.9
    systemPercent: 0.05
    userPercent: 0.05
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: '"almalinux"'
    version: "9.2"
  eventLogger: journald
  hostname: tachikoma13
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-284.11.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1242116096
  memTotal: 1813958656
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.el9_2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 10h 48m 16.00s (Approximately 0.42 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 40165670912
  graphRootUsed: 1570525184
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/lib/containers/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/volumes
version:
  APIVersion: 4.5.1
  Built: 1686840794
  BuiltTime: Thu Jun 15 14:53:14 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[rhatdan]

@mheon PTAL

[vrothberg]

That's intentional as the sqlite DB may be used to store images/containers in a very distant future. @mskarbek can you elaborate why you think it's a bug?

[mskarbek]

Bug in a sense: "it doesn't behave as configured and expected". On some installations, I have to deal with partitions size and layout enforced by policies, so I keep state, containers and volumes in separate locations to deal with occasional containers' storage lack of space, which almost always meant state corruption until I moved it to /var/lib/libpod on root partition. With separation, lack of space is recoverable without full state restart. I'm afraid that the SQLite backend won't be immune to corruption either in this case.

[vrothberg]

@mheon WDYT?

[mheon]

I think we can alter the DB loading logic to check for both the current directory and the more-standard Libpod static directory where we put BoltDB, and change DB create logic to create in the static dir by default. This should ensure no breakage, put the DB in a more predictable location, and if we do eventually have a c/storage DB we should still use it because the DB is initialized after c/storage, which presumably will have already made its own sqlite DB for us to use.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon is this something we need before we switch to sqlite?

[mheon]

Not essential, I think - the change is different from expectations but unlikely to cause serious issue.