[BUG] MAJOR ISSUE - New 4.4.0 release corrupts existing SQLITE3 databases

InspectorCaracal commented 2 hours ago

Describe the bug

After upgrading to 4.4.0 in an existing game, the game database becomes corrupted.

Any attempts to write to the database - even just with updating session info - will result in a data I/O error or malformed disk image error, which then persists and prevents all further data reads as well.

Downgrading back to 4.3.0 does NOT resolve the issue; the corruption appears to be permanent.

The issue does not become obvious when actively playing in-game, due to Evennia's internal caching, but a stop/start of the server makes it obvious very quickly that none of the changes that result due to gameplay are persisted - the database does not actually get written to.

To Reproduce

Steps to reproduce the behavior:

Create a game and its data in v4.3.0
Update to v4.4.0 and reboot evennia
See error

Environment, Evennia version, OS etc

This has so far been reproduced in Debian 12, Ubuntu 22 and macOS 12.7.6

Additional context

This is almost certainly due to the recent PRAGMA changes, which I'm guessing were only tested on a fresh database.

owllex commented 2 hours ago

To add, this is serious. My prod server has suffered data loss as a result. My data is backed up and can be restored, but others may not be so lucky if they upgrade.

InspectorCaracal commented 1 hour ago

This issue appears to be avoidable if you configure the new SQLITE3_PRAGMAS setting to match the old options BEFORE upgrading and restarting evennia:

SQLITE3_PRAGMAS = (
    "PRAGMA cache_size=10000",
    "PRAGMA synchronous=OFF",
    "PRAGMA count_changes=OFF",
    "PRAGMA temp_store=2",
)

Changing the configuration after the update has already been applied does not fix the issue; it only prevents it from occurring. I have not found any way to fix an affected database so far.

evennia / evennia