RFE: add an option to park the rpmdb

[cgwalters]

This is a followup to https://github.com/rpm-software-management/rpm/commit/71456f2fc09900a027a33dc3d6d75c69a9b39488 which is about generating bit-for-bit reproducible images (container/disk) that include an RPM database.

At the time, the person working on that PR was looking at RHEL8 (BDB format rpm database). We've since moved on to sqlite, which is nicer.

Except in my testing, this rpmdb.sqlite-shm is the very last thing blocking reproducible rpm-ostree container images. I hacked up https://github.com/coreos/rpm-ostree/pull/4074 but this issue is to track fixing this more properly inside librpm.

I think what would be the best is if this file could be deleted entirely at the end of a build. It seems like rpm (or sqlite) will auto-recreate it in that case, but what blocks us from doing that is that things fail if the mount is read-only.

[pmatilai]

The existence of .sqlite-shm is required for read-only WAL mode to work at all (a very important use-case being queries by regular users), see https://www.sqlite.org/wal.html#read_only_database

[pmatilai]

Hmm. Actually doing the WAL docs suggested PRAGMA journal_mode=DELETE on the database results in a database that can still be opened by rpm, regular users and all. Rpm will try to re-establish WAL if opened in read-write mode, so nothing too terrible should happen if you do the journal delete as a part of the image building. Except voiding the warranty :wink:

It'd be nicer of course if rpm had a supported procedure to "park" databases for this kind of thing. --rebuilddb with some special flag maybe.

[champtar]

rpmdb reproducibility is also an issue when building (EL9) containers, it would be nice to have a park option indeed.

[DemiMarie]

It'd be nicer of course if rpm had a supported procedure to "park" databases for this kind of thing. --rebuilddb with some special flag maybe.

--rebuilddb is much heavier than just a single SQL command. Perhaps --parkdb, along with a corresponding C API function?

[cgwalters]

The existence of .sqlite-shm is required for read-only WAL mode to work at all (a very important use-case being queries by regular users), see https://www.sqlite.org/wal.html#read_only_database

I find this weird - because unprivileged code can't write directly to the database, what practical use is the shared memory?

Hmm, I just found https://www.sqlite.org/uri.html#uriimmutable - seems like rpm may be able to use that when it detects it can't write?

[pmatilai]

As per https://www.sqlite.org/wal.html#read_only_databases:

Even though it is possible to open a read-only WAL-mode database, it is good practice to converted to PRAGMA journal_mode=DELETE prior to burning an SQLite database image onto read-only media.

We want to follow good practises so...

Any operation should do a database rebuild to free up any fluff in the files, so I think this should be either a default thing in rebuilddb or an extra flag to it if default is not feasible for technical reasons.

[cgwalters]

@ffesti the issue I mentioned at Devconf.cz

That said...now that I'm digging in again to reproducible builds, https://github.com/containers/buildah/issues/5592 is higher on my radar, but this is probably in the top 5.

[cgwalters]

so I think this should be either a default thing in rebuilddb

On this topic, it would be a nontrivial hit to the ergonomics of default container builds if we really start telling people to do:

RUN dnf -y install foo && dnf clean all && rpm --fix-database

So probably what should happen is dnf should detect it's in a container and do this by default as part of dnf clean all most likely.

[cgwalters]

I came across https://github.com/microsoft/rpmoci/blob/47443c4312505222460dfcd185308db7153897bb/src/lockfile/build.rs#L173-L179