Error: database storage graph driver "" does not match our storage graph driver "overlay": database configuration mismatch

bduffany commented 9 months ago

Issue Description

We see this error Error: database storage graph driver "" does not match our storage graph driver "overlay": database configuration mismatch when our customers upgrade to a newer version of podman (we distribute podman with our software, and when customers upgrade our software, they also are upgraded to a newer podman version).

The issue https://github.com/containers/podman/issues/2347 mentioned that there was a fix in-progress for this issue that would avoid the need to manually delete /var/lib/containers. We would be really interested in this fix, because the current approach of deleting /var/lib/containers winds up deleting cached images, and is also tricky for us to roll out for various reasons (mostly because there is some flexibility in how customers mount this directory).

I am filing this issue just to express that the "database configuration mismatch" issue is causing us some headaches, and that a more automatic solution would be very much appreciated!

Steps to reproduce the issue

N/A, see discussion in original linked issue.

Describe the results you received

N/A, see discussion in original linked issue.

Describe the results you expected

N/A, see discussion in original linked issue.

podman info output

N/A, see discussion in original linked issue.

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

rhatdan commented 9 months ago

Can you create a storage.conf file with driver="overlay" to solve it?

github-actions[bot] commented 8 months ago

A friendly reminder that this issue had no activity for 30 days.

MaxKingPor commented 6 months ago

Can you create a storage.conf file with driver="overlay" to solve it?

>$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

>$ uname -a
Linux PowerEdge-R530xd 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

>$ podman --version
podman version 4.9.3

dcasier commented 6 months ago

I have no idea what the consequences are, but I found this trick:

` cd /var/lib/containers/storage

sqlite db.sql

update DBConfig set GraphDriver = 'overlay' where GraphDriver = ''; `

tacerus commented 5 months ago

Hi,

the error persists with

$ more /etc/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

(deleting /var/lib/containers/* helps, with and without storage.conf)

MaxKingPor commented 5 months ago

@tacerus Remove runroot and graphroot tests

tacerus commented 5 months ago

@MaxKingPor, removing runroot and graphroot lines from storage.conf will cause an error that these must be set.

MaxKingPor commented 5 months ago

@tacerus Delete the configured directory. rm -ir <runroot> <graphroot>

tacerus commented 5 months ago

I already stated in my comment that deleting the whole data directory helped. Good to know if deleting only these two directories would have sufficed, but too late now. ;-)

Poseidon6473 commented 4 months ago

I have no idea what the consequences are, but I found this trick:

` cd /var/lib/containers/storage

sqlite db.sql

update DBConfig set GraphDriver = 'overlay' where GraphDriver = ''; `

I also have no idea what the consequences are, but this worked for me. Although the process for running the command I used was a bit different:

cd /var/lib/containers/storage

# (sqlite as a command on it's own yelled at me)
sqlite3 db.sql

# (now in the sqlite3 shell)
update DBConfig set GraphDriver = 'overlay' where GraphDriver = "";

The sqlite3 shell should return to an empty line expecting a new command input, this means it went successfully. Was able to recover fully after this.

Thank you sir

containers / podman