search

fabianonline / telegram_backup

Java app to download all your telegram data.
GNU General Public License v3.0
447 stars 91 forks source link

Fatal database error after updating to latest jar #103

Closed leijurv closed 6 years ago

leijurv commented 6 years ago 
.08:20:33.606 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 115000           08:20:33.740 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:33.740 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 120000
08:20:33.875 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:33.875 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 125000
08:20:34.010 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.010 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 130000
08:20:34.148 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.149 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 135000
08:20:34.283 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.283 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 140000           08:20:34.418 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.418 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 145000
08:20:34.552 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.552 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 150000
08:20:34.687 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.687 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 155000           08:20:34.821 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()        .08:20:34.821 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 160000           08:20:34.955 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:34.956 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 165000
08:20:35.089 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:35.089 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 170000
08:20:35.222 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:35.222 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 175000
08:20:35.357 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:35.357 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 180000
08:20:35.494 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:35.494 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 185000
08:20:35.628 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.08:20:35.629 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 190000
08:20:35.762 DEBUG DatabaseUpdate.execute             : Executing: DELETE FROM messages WHERE id IN ()
.
08:20:35.763 INFO  DB_Update_9._doUpdate              : Converted 91171 of 192686 messages.
    Cleaning up the database (this might also take some time, sorry)...
08:20:35.763 DEBUG DatabaseUpdate.execute             : Executing: VACUUM
An error occured!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (cannot V
ACUUM - SQL statements in progress)
        at de.fabianonline.telegram_backup.DatabaseUpdates.doUpdates(DatabaseUpdates.kt:107)
        at de.fabianonline.telegram_backup.Database.<init>(Database.kt:234)
        at de.fabianonline.telegram_backup.Database.<init>(Database.kt:50)
        at de.fabianonline.telegram_backup.Database$Companion.init(Database.kt:766)
        at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.kt:102)
        at de.fabianonline.telegram_backup.CommandLineRunnerKt.main(CommandLineRunner.kt:41)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (cannot VACUUM - SQL statements in progress)
        at org.sqlite.core.DB.newSQLException(DB.java:909)
        at org.sqlite.core.DB.newSQLException(DB.java:921)
        at org.sqlite.core.DB.throwex(DB.java:886)
        at org.sqlite.core.NativeDB._exec_utf8(Native Method)
        at org.sqlite.core.NativeDB._exec(NativeDB.java:87)
        at org.sqlite.jdbc3.JDBC3Statement.executeUpdate(JDBC3Statement.java:116)
        at de.fabianonline.telegram_backup.DatabaseUpdate.execute(DatabaseUpdates.kt:168)
        at de.fabianonline.telegram_backup.DB_Update_9._doUpdate(DatabaseUpdates.kt:445)
        at de.fabianonline.telegram_backup.DatabaseUpdate.doUpdate(DatabaseUpdates.kt:155)
        at de.fabianonline.telegram_backup.DatabaseUpdates.doUpdates(DatabaseUpdates.kt:104)
        ... 5 more
08:20:35.768 ERROR CommandLineController.<init>       : Exception caught!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (cannot VACUUM - SQL statements in progress)
        at de.fabianonline.telegram_backup.DatabaseUpdates.doUpdates(DatabaseUpdates.kt:107)
        at de.fabianonline.telegram_backup.Database.<init>(Database.kt:234)
        at de.fabianonline.telegram_backup.Database.<init>(Database.kt:50)
        at de.fabianonline.telegram_backup.Database$Companion.init(Database.kt:766)
        at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.kt:102)
        at de.fabianonline.telegram_backup.CommandLineRunnerKt.main(CommandLineRunner.kt:41)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (cannot VACUUM - SQL statements in progress)
        at org.sqlite.core.DB.newSQLException(DB.java:909)
        at org.sqlite.core.DB.newSQLException(DB.java:921)
        at org.sqlite.core.DB.throwex(DB.java:886)
        at org.sqlite.core.NativeDB._exec_utf8(Native Method)
        at org.sqlite.core.NativeDB._exec(NativeDB.java:87)
        at org.sqlite.jdbc3.JDBC3Statement.executeUpdate(JDBC3Statement.java:116)
        at de.fabianonline.telegram_backup.DatabaseUpdate.execute(DatabaseUpdates.kt:168)
        at de.fabianonline.telegram_backup.DB_Update_9._doUpdate(DatabaseUpdates.kt:445)
        at de.fabianonline.telegram_backup.DatabaseUpdate.doUpdate(DatabaseUpdates.kt:155)
        at de.fabianonline.telegram_backup.DatabaseUpdates.doUpdates(DatabaseUpdates.kt:104)
        ... 5 common frames omitted

----- EXIT -----
leijurv commented 6 years ago

It seems to have run some extremely long sql commands, lists of thousands and thousands of message ids

leijurv commented 6 years ago

This is what the beginning of the output looks like:

leijurv@swamp:/mnt/data/Dropbox$ java -jar telegram_backup.jar --with-supergroups --with-channels --account +16503876684 --debug --target /mnt/tgdata/
08:30:40.351 DEBUG Utils.getNewestVersion             : Requesting current release info from https://api.github.com/repos/fabianonline/telegram_backup/releases
08:30:41.581 DEBUG Utils.getNewestVersion             : Found current release version 1.1.3
08:30:41.581 DEBUG Utils.compareVersions              : Comparing versions 1.1.3-3-gad6f57d and 1.1.3.
08:30:41.600 DEBUG Utils.compareVersions              : v1 has a suffix, v2 not.
08:30:41.601 INFO  CommandLineController.<init>       : CommandLineController started. App version 1.1.3-3-gad6f57d
Telegram_Backup version 1.1.3-3-gad6f57d, Copyright (C) 2016, 2017 Fabian Schlenz

Telegram_Backup comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions; run it with '--license' for details.

08:30:41.602 DEBUG CommandLineController.setupFileBase: Target dir at startup: /home/leijurv/.telegram_backup
08:30:41.602 DEBUG CommandLineController.setupFileBase: Target dir after options: /mnt/tgdata/
Base directory for files: /mnt/tgdata/
08:30:41.602 DEBUG CommandLineController.<init>       : Initializing TelegramApp
08:30:41.602 DEBUG CommandLineController.selectAccount: Account requested: +1REDACTEDXX
08:30:41.602 DEBUG CommandLineController.selectAccount: accounts.size: 1
08:30:41.602 DEBUG CommandLineController.selectAccount: account: +1REDACTEDXX
08:30:41.602 DEBUG CommandLineController.<init>       : CommandLineOptions.cmd_login: false
08:30:41.602 INFO  CommandLineController.<init>       : Initializing ApiStorage
08:30:41.603 INFO  CommandLineController.<init>       : Initializing TelegramUpdateHandler
08:30:41.603 INFO  CommandLineController.<init>       : Creating Client
08:30:42.194 INFO  CommandLineController.<init>       : Initializing UserManager
08:30:42.195 DEBUG UserManager.<init>                 : Calling getFullUser
08:30:42.323 DEBUG CommandLineController.<init>       : CommandLineOptions.cmd_login: false
Opening database...
08:30:42.376 DEBUG DatabaseUpdates.<init>             : Registering Database Updates...
08:30:42.376 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_1 as update to version 1
08:30:42.377 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_2 as update to version 2
08:30:42.377 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_3 as update to version 3
08:30:42.377 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_4 as update to version 4
08:30:42.377 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_5 as update to version 5
08:30:42.377 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_6 as update to version 6
08:30:42.378 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_7 as update to version 7
08:30:42.378 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_8 as update to version 8
08:30:42.378 DEBUG DatabaseUpdates.register           : Registering class de.fabianonline.telegram_backup.DB_Update_9 as update to version 9
08:30:42.378 DEBUG DatabaseUpdates.doUpdates          : DatabaseUpdate.doUpdates running
08:30:42.378 DEBUG DatabaseUpdates.doUpdates          : Getting current database version
08:30:42.378 DEBUG DatabaseUpdates.doUpdates          : Checking if table database_versions exists
08:30:42.379 DEBUG DatabaseUpdates.doUpdates          : Table exists. Checking max version
08:30:42.379 DEBUG DatabaseUpdates.doUpdates          : version: 8
Database version: 8
08:30:42.379 DEBUG DatabaseUpdates.doUpdates          : Max available database version is 9
08:30:42.379 DEBUG DatabaseUpdates.doUpdates          : Update is necessary. 8 => 9.
08:30:42.379 DEBUG DatabaseUpdates.doUpdates          : Update to version 9 needs a backup
08:30:42.379 DEBUG DatabaseUpdates.doUpdates          : Performing backup
  Creating a backup of your database as database.version_8.backup.sqlite
08:30:42.379 DEBUG Database.backupDatabase            : Copying /mnt/tgdata//+1REDACTEDXX/database.sqlite to /mnt/tgdata//+1REDACTEDXX/database.version_8.backup.sqlite
08:30:42.490 DEBUG DatabaseUpdates.doUpdates          : Applying updates
08:30:42.491 DEBUG DatabaseUpdate.doUpdate            : Applying update to version 9
  Updating to version 9...
    Updating supergroup channel message data (this might take some time)...
    08:30:42.562 DEBUG DB_Update_9._doUpdate              : Found 192686 candidates for conversion
08:30:42.562 DEBUG DB_Update_9._doUpdate              : Querying with limit 5000 and offset 0
leijurv commented 6 years ago

This could be because my database is very big and I have a lot of messages and media:

4.0K    auth.dat
335M    database.sqlite
48K database.sqlite-journal
335M    database.version_8.backup.sqlite
4.0K    dc.dat
75G files
itskenny0 commented 6 years ago

I'm also affected by this.

image

My database size is very close to @leijurv 's, so that might be why.

image

itskenny0 commented 6 years ago

@leijurv : see #94 for possible workaround

leijurv commented 6 years ago

Thank you, that fixes it

fabianonline commented 6 years ago

A real bugfix is in the works. Sorry for taking so long. :-(

fabianonline commented 6 years ago

1.1.4 is out.