SqliteException(11): database disk image is malformed

[stx]

First, thank you for this lovely software.

Rarely, we'll see this issue on production via Crashlytics:

SqliteException(11): database disk image is malformed Causing statement: DROP TABLE IF EXISTS <table name>;. Error thrown null.

Non-fatal Exception: FlutterError
0  ???                            0x0 DatabaseImpl.execute + 129 (database.dart:129)
1  ???                            0x0 _VmDelegate._runWithArgs + 204 (database.dart:204)
2  ???                            0x0 _VmDelegate.runCustom + 214 (database.dart:214)
3  ???                            0x0 _BaseExecutor.runCustom.<fn> + 109 (engines.dart:109)
4  ???                            0x0 _BaseExecutor._synchronized + 55 (engines.dart:55)
5  ???                            0x0 _BaseExecutor.runCustom + 105 (engines.dart:105)
6  ???                            0x0 ServerImplementation._runQuery + 130 (server_impl.dart:130)

We're not issuing any DROP TABLE statements except in our migration:

  @override
  MigrationStrategy get migration => MigrationStrategy(
        onUpgrade: (Migrator m, int from, int to) async {
          for (TableInfo<Table, dynamic> table in allTables) {
            await m.deleteTable(table.actualTableName);
            await m.createTable(table);
          }
        },
      );

Is there a better way to do this, or a way to fix or prevent this error?

[simolus3]

Thanks for the report. Are all of those "database disk image is malformed" errors coming from the migration? Or are there any others that with a similar error message but a different trace? For further analysis (since unfortunately this sounds like a bug that will be hard to trace down), it could be helpful try and catch this SqliteException and add the results of running pragma integrity_check to the Crashlytics error.

There's a list of things commonly causing database corruption, but most of these shouldn't be relevant with drift. Are you opening the same database file on multiple isolates?

Is there a better way to do this, or a way to fix or prevent this error?

Is it your intention to clear the entire database on each schema upgrade? Because if so, it might be easier to just use the sqlite3 package to open the raw database, check and compare the schema version and delete the database file when needed.

[stx]

Dug into the Crashlytics, and it's iOS only and only during migrations. Traces are always the same.

The database is opened once, on one isolate.

Yes, the intention is to clear out the data and recreate the schemas on each upgrade. I like how Drift handles the migration process automatically and it works extremely well except in these incredibly rare cases. How would you propose doing it better/easier?

[simolus3]

You could use something like this to delete the entire database file if it has a lower schema version:

import 'package:sqlite3/sqlite3.dart';

// This can be passed to the database's constructor
LazyDatabase open(int targetVersion) {
  return LazyDatabase(() async {
    // proper logic to determine which file to open...
    final file = File('/tmp/database.db');

    if (await file.exists()) {
      try {
        final db = sqlite3.open(file.path);
        if (db.userVersion < targetVersion) {
          db.dispose();
          await file.delete();
        }
      } on SqliteException {
        // This exception really shouldn't be thrown, but if the database file
        // is corrupted already, deleting it may be the best action.
        await file.delete();
      }
    }

    return NativeDatabase(file);
  });
}

Unfortunately I don't have an idea on where that error might be coming from or what's corrupting the database yet.

[abdelaziz-mahdy]

I have the same error on the web it only happens when I store a very large text

When I try to read the same table after that insert it shows up the database is malformed error

[simolus3]

@zezo357 Are you using a WebDatabase or a WasmDatabase? In either way, if you have a way to reproduce this (just storing a large string? if so, how large?), can you share that?

[abdelaziz-mahdy]

@zezo357 Are you using a WebDatabase or a WasmDatabase? In either way, if you have a way to reproduce this (just storing a large string? if so, how large?), can you share that?

WasmDatabase

How large it's a JSON file containing 10000 objects

Sadly I can't share it , but I will try to make a code to reproduce it

[abdelaziz-mahdy]

@simolus3 I was storing all of objects in a single text cell and just found out that their is a limit in text cell

The maximum number of bytes in the text of an SQL statement is limited to SQLITE_MAX_SQL_LENGTH which defaults to 1,000,000,000.

from my understanding I broke the limit :)

looks like i will need to separate it to rows of objects in the table :(

[stx]

@simolus3

Using WasmSqlite3.loadFromUrl() is causing this to happen more frequently. Some users are experiencing this, and both my debug and production builds just became malformed, so we're rolling back to the old method.

[lomirus]

I have the same error when storing a large amount of data, too. To reproduce, you can:

1. Download the lexicon.json.
2. Go to the https://yuorb.github.io/enthrirhs/
3. Click the "Import lexicon from local files" and select "lexicon.json" (It may get stuck for a few seconds)
4. Refresh the page.

It can be reproduced stably in my environment.

Related code: https://github.com/yuorb/enthrirhs/blob/main/lib/database/shared.dart

[simolus3]

Thanks for the report - I'll try to check it out soon. In the meantime, do you see on which VFS implementation this happens (opfs shared or opfs locked, shared indexeddb?)

[lomirus]

do you see on which VFS implementation this happens (opfs shared or opfs locked, shared indexeddb?)

lib/database/web.dart:

final db = LazyDatabase(() async {
  // Create virtual filesystem for sqlite3 implemented over blobs stored in an
  // IndexedDB database (named `my_app` here).
  final fs = await IndexedDbFileSystem.open(dbName: 'my_app');

  final sqlite3 = await WasmSqlite3.loadFromUrl(
    Uri.parse('sqlite3.wasm'),
    environment: SqliteEnvironment(fileSystem: fs),
  );

  // Then, open a database:
  return WasmDatabase(sqlite3: sqlite3, path: '/app.db');
});

[lomirus]

🤔 Is there any progress now about this problem?

[simolus3]

Sorry for the slow response - unfortunately I couldn't reproduce this after upgrading to the latest version of the sqlite3 package and using the latest sqlite3.wasm file. The only things I've changed are the sqlite3/drift versions in the pubspec and

diff --git a/lib/database/web.dart b/lib/database/web.dart
index 9f3187e..9bd90e3 100644
--- a/lib/database/web.dart
+++ b/lib/database/web.dart
@@ -9,10 +9,8 @@ Database constructDb() {
     // IndexedDB database (named `my_app` here).
     final fs = await IndexedDbFileSystem.open(dbName: 'my_app');

-    final sqlite3 = await WasmSqlite3.loadFromUrl(
-      Uri.parse('sqlite3.wasm'),
-      environment: SqliteEnvironment(fileSystem: fs),
-    );
+    final sqlite3 = await WasmSqlite3.loadFromUrl(Uri.parse('sqlite3.wasm'));
+    sqlite3.registerVirtualFileSystem(fs, makeDefault: true);

     // Then, open a database:
     return WasmDatabase(sqlite3: sqlite3, path: '/app.db');

[lomirus]

You are right. I upgraded the drift and packages as you said, and found that the problem has already been fixed in the previous versions. Thanks for you work👍

[jakobhec]

I encountered the same issue: SqliteException(11): database disk image is malformed when installing a new app version requiring an updated database.

I manually deleted the existing SQLite database, then copied a new version in its place and opened it in WAL mode.

Deleting the -wal and -shm files seems to fix the issue. Here's how I now remove the database before copying over a new one:

final file = await getFileLocationDrift();
if (await file.exists()) {
  try {
    // https://github.com/simolus3/drift/issues/1773#issuecomment-1079767310
    final db = sqlite3.open(file.path);
    db.dispose();

    await file.delete();
  } on SqliteException {
    // This exception really shouldn't be thrown, but if the database file
    // is corrupted already, deleting it may be the best action.
    await file.delete();
  }

  final walFile = File('${file.path}-wal');
  if (await walFile.exists()) {
    await walFile.delete();
  }
  final shmFile = File('${file.path}-shm');
  if (await shmFile.exists()) {
    await shmFile.delete();
  }
}

Hope it is helpful for some. Alway happy to hear improvement ideas!

[jakobhec]

@simolus3 If the above behavior is expected, it might make sense to add it to the documentation.

[simolus3]

I agree, is there a particular place where you'd expect this information? Perhaps there should be a dedicated page about backups.

[jakobhec]

• https://drift.simonbinder.eu/docs/platforms/vm/#using-moor-ffi-with-an-existing-database
• https://drift.simonbinder.eu/docs/examples/existing_databases/#extracting-the-database
• and potentially under "Migrations" https://drift.simonbinder.eu/docs/migrations/

Or with a dedicated page as you've said and then you could link to it "See more about migrations/backups/existing-dbs" in the above places.

[simolus3]

@jakobhec To confirm, you're running into this with existing database files that have originally been created through an asset or another package and that you now want to use with drift?

If these databases are supposed to use the WAL journaling mode, I think deleting the -wal file might delete some recent modifications to the database.

[jakobhec]

I create the .sqlite database files on a server (using sqlalchemy / pandas / python).

Approximately every 2 months I want to completely replace the drift database on the mobile device with an updated version from the server. So I simply put a newly generated sqlite file from the server in the assets folder of my flutter application and replace the existing db on first app open after an udpate.

Deleting recent modifications is not an issue in this case.

[bendangelo]

I encountered the same issue: SqliteException(11): database disk image is malformed when installing a new app version requiring an updated database.

I manually deleted the existing SQLite database, then copied a new version in its place and opened it in WAL mode.

Deleting the -wal and -shm files seems to fix the issue. Here's how I now remove the database before copying over a new one:

final file = await getFileLocationDrift();
if (await file.exists()) {
  try {
    // https://github.com/simolus3/drift/issues/1773#issuecomment-1079767310
    final db = sqlite3.open(file.path);
    db.dispose();

    await file.delete();
  } on SqliteException {
    // This exception really shouldn't be thrown, but if the database file
    // is corrupted already, deleting it may be the best action.
    await file.delete();
  }

  final walFile = File('${file.path}-wal');
  if (await walFile.exists()) {
    await walFile.delete();
  }
  final shmFile = File('${file.path}-shm');
  if (await shmFile.exists()) {
    await shmFile.delete();
  }
}

Hope it is helpful for some. Alway happy to hear improvement ideas!

This should be added to the docs. I had issues deleting the old db and replacing it because of these files hanging around. Every time I loaded the replaced db I would get malformed error or other weird issues.