Error: database storage graph driver "" does not match our storage graph driver "overlay": database configuration mismatch

[bduffany]

Issue Description

We see this error Error: database storage graph driver "" does not match our storage graph driver "overlay": database configuration mismatch when our customers upgrade to a newer version of podman (we distribute podman with our software, and when customers upgrade our software, they also are upgraded to a newer podman version).

The issue https://github.com/containers/podman/issues/2347 mentioned that there was a fix in-progress for this issue that would avoid the need to manually delete /var/lib/containers. We would be really interested in this fix, because the current approach of deleting /var/lib/containers winds up deleting cached images, and is also tricky for us to roll out for various reasons (mostly because there is some flexibility in how customers mount this directory).

I am filing this issue just to express that the "database configuration mismatch" issue is causing us some headaches, and that a more automatic solution would be very much appreciated!

Steps to reproduce the issue

N/A, see discussion in original linked issue.

Describe the results you received

N/A, see discussion in original linked issue.

Describe the results you expected

N/A, see discussion in original linked issue.

podman info output

N/A, see discussion in original linked issue.

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[rhatdan]

Can you create a storage.conf file with driver="overlay" to solve it?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[MaxKingPor]

Can you create a storage.conf file with driver="overlay" to solve it?

>$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

>$ uname -a
Linux PowerEdge-R530xd 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

>$ podman --version
podman version 4.9.3

[dcasier]

I have no idea what the consequences are, but I found this trick:

` cd /var/lib/containers/storage

sqlite db.sql

update DBConfig set GraphDriver = 'overlay' where GraphDriver = ''; `

[tacerus]

Hi,

the error persists with

$ more /etc/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

(deleting /var/lib/containers/* helps, with and without storage.conf)

[MaxKingPor]

@tacerus Remove runroot and graphroot tests

[tacerus]

@MaxKingPor, removing runroot and graphroot lines from storage.conf will cause an error that these must be set.

[MaxKingPor]

@tacerus Delete the configured directory. rm -ir <runroot> <graphroot>

[tacerus]

I already stated in my comment that deleting the whole data directory helped. Good to know if deleting only these two directories would have sufficed, but too late now. ;-)