search

fabianonline / telegram_backup

Java app to download all your telegram data.
GNU General Public License v3.0
448 stars 91 forks source link

Number of messages bigger locally than server-side #38

Closed aleschmid closed 6 years ago

aleschmid commented 7 years ago

Hello there! 😃

First of all, thanks so much for such cool repo what you got here, I already mentioned you on a tweet, but it seems like you haven't been very active lately on Twitter.

I'm letting you know with this message that there's a tiny problem with the max number of messages. Since you can make a backup, remove messages from the server and run the program again, you're causing the number of messages on the local DB exceed the ones on the server. I don't know if this is really a big deal because I haven't take the time to read the code, but I hope the the program is using the messages' ID to download in order to not crush (overwrite) local ones that don't exist on the server.

Here's the exception thrown:

java.lang.RuntimeException: max_database_id is bigger then max_message_id. This shouldn't happen. But the telegram api nonetheless does that sometimes. Just ignore this error, wait a few seconds and then try again.
    at de.fabianonline.telegram_backup.DownloadManager._downloadMessages(DownloadManager.java:130)
    at de.fabianonline.telegram_backup.DownloadManager.downloadMessages(DownloadManager.java:75)
    at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.java:168)
    at de.fabianonline.telegram_backup.CommandLineRunner.main(CommandLineRunner.java:77)

Viele grüße!

fabianonline commented 7 years ago

That's not a problem. Telegram uses a strange way for message ids: Every message you receive in a chat or normal group will be copied to your account and given a sequentially rising ID. (Supergroups and Channels are exempt from this.) So you don't have something like global ids but only your "personal" ids. When a message is deleted, Telegram replaces it in it's database by an empty message. So the ID stays in use, but the message is still deleted.

I expect this behavior to happen: An empty message will never overwrite an existing non-empty message in the backup. Normally this shouldn't even happen, because if you have messages with IDs 1 to 100, the next downloaded message will be message 101.

For the error you're seeing: Is that still happening? If yes, can you please run the tool with --trace and send me the output from that?

aleschmid commented 7 years ago

Hi again! 😃

I've done some testing in order to make sure there's no lost data, and like you said, there seems to be no risk for any message to be overwritten thanks to their unique ID. So… cool!

Answering your question: No, the exception doesn't show up anymore; however I can easily make it happen. How? That's simple, the only thing I need to do is run the app, make the backup and once I've got all the messages copied I just need to delete the very last messages of a conversation (not some in between, only the newest ones, otherwise you won't get the exception).

The reason I came up with for this behavior, is that whenever you delete the last messages, Telegram does indeed remove them (or at least doesn't count them), but because of the unique IDs, once the server receives new messages Telegram will fill those blank spaces increasing the length up to the last ID (like you said).

For example: If I had 2325 messages (already backed up) and I deleted the last one, it's gonna decrease the server side count to 2324, causing the exception to appear. However if a friend sends me a picture after that, then instead of just adding 1, Telegram will add 2 messages, leaving a total number of 2326 (same as the ones locally).

Hope it makes sense. Viele grüße!

Rapti commented 7 years ago

I have the same problem.

Telegram_Backup version 1.0.6, Copyright (C) 2016 Fabian Schlenz

Telegram_Backup comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions; run it with '--license' for details.

Base directory for files: /home/leon/.telegram_backup
Using only available account: +49XXXXXXXXXXXX
You are logged in as Leon (@Rapti)
Opening database...
Database version: 7
Database is ready.
Downloading most recent dialogs... 
Top message ID is 456966
Top message ID in database is 470798
java.lang.RuntimeException: max_database_id is bigger then max_message_id. This shouldn't happen. But the telegram api nonetheless does that sometimes. Just ignore this error, wait a few seconds and then try again.
    at de.fabianonline.telegram_backup.DownloadManager._downloadMessages(DownloadManager.java:130)
    at de.fabianonline.telegram_backup.DownloadManager.downloadMessages(DownloadManager.java:75)
    at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.java:168)
    at de.fabianonline.telegram_backup.CommandLineRunner.main(CommandLineRunner.java:77)

----- EXIT -----

I've run the command multiple times now, but the error keeps occuring. The server side message count does increase as I send and receive new messages, but I don't want to wait ~14000 more messages before being able to backup again. I also think this might be caused by a huge supergroup that was deleted recently and might have had around that many messages.

Rapti commented 6 years ago

The error still occurs. Here's my output with --trace as requested above.

01:59:09 DEBUG Utils.getNewestVersion             : Requesting current release info from https://api.github.com/repos/fabianonline/telegram_backup/releases
01:59:12 DEBUG Utils.getNewestVersion             : Found current release version 1.0.6
01:59:12 DEBUG Utils.compareVersions              : Comparing versions 1.0.6 and 1.0.6.
01:59:12 INFO  CommandLineController.<init>       : CommandLineController started. App version 1.0.6
Telegram_Backup version 1.0.6, Copyright (C) 2016 Fabian Schlenz

Telegram_Backup comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions; run it with '--license' for details.

01:59:12 DEBUG CommandLineController.<init>       : Target dir at startup: /home/leon/.telegram_backup
01:59:12 DEBUG CommandLineController.<init>       : Target dir after options: /home/leon/.telegram_backup
Base directory for files: /home/leon/.telegram_backup
01:59:12 DEBUG CommandLineController.<init>       : Initializing TelegramApp
01:59:12 TRACE CommandLineController.<init>       : Checking accounts
Using only available account: +49XXXXXXXXXXX
01:59:12 DEBUG CommandLineController.<init>       : accounts.size(): 1
01:59:12 DEBUG CommandLineController.<init>       : account: +49XXXXXXXXXXX
01:59:12 DEBUG CommandLineController.<init>       : CommandLineOptions.cmd_login: false
01:59:12 INFO  CommandLineController.<init>       : Initializing ApiStorage
01:59:12 INFO  CommandLineController.<init>       : Initializing TelegramUpdateHandler
01:59:12 INFO  CommandLineController.<init>       : Creating Client
01:59:13 INFO  CommandLineController.<init>       : Creating UserManager
01:59:13 DEBUG UserManager.<init>                 : Calling getFullUser
01:59:13 DEBUG CommandLineController.<init>       : CommandLineOptions.val_export: null
01:59:13 DEBUG CommandLineController.<init>       : CommandLineOptions.cmd_login: false
You are logged in as Leon (@Rapti)
01:59:13 INFO  CommandLineController.<init>       : Initializing Download Manager
Opening database...
01:59:13 DEBUG DatabaseUpdates.<init>             : Registering Database Updates...
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_1 as update to version 1
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_2 as update to version 2
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_3 as update to version 3
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_4 as update to version 4
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_5 as update to version 5
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_6 as update to version 6
01:59:13 DEBUG DatabaseUpdates.register           : Registering de.fabianonline.telegram_backup.DB_Update_7 as update to version 7
01:59:13 DEBUG DatabaseUpdates.doUpdates          : DatabaseUpdate.doUpdates running
01:59:13 DEBUG DatabaseUpdates.doUpdates          : Getting current database version
01:59:13 DEBUG DatabaseUpdates.doUpdates          : Checking if table database_versions exists
01:59:13 DEBUG DatabaseUpdates.doUpdates          : Table exists. Checking max version
01:59:13 DEBUG DatabaseUpdates.doUpdates          : version: 7
Database version: 7
01:59:13 DEBUG DatabaseUpdates.doUpdates          : Max available database version is 7
01:59:13 DEBUG DatabaseUpdates.doUpdates          : No update necessary.
Database is ready.
01:59:13 DEBUG CommandLineController.<init>       : Calling DownloadManager.downloadMessages with limit null
01:59:13 INFO  DownloadManager._downloadMessages  : This is _downloadMessages with limit null
01:59:13 INFO  DownloadManager._downloadMessages  : Downloading the last 100 dialogs
Downloading most recent dialogs... 
01:59:14 DEBUG DownloadManager._downloadMessages  : Got 100 dialogs
01:59:14 TRACE DownloadManager._downloadMessages  : Updating top message id: 0 => 466515. Dialog type: com.github.badoualy.telegram.tl.api.TLPeerUser
Top message ID is 466515
Top message ID in database is 470798
java.lang.RuntimeException: max_database_id is bigger then max_message_id. This shouldn't happen. But the telegram api nonetheless does that sometimes. Just ignore this error, wait a few seconds and then try again.
    at de.fabianonline.telegram_backup.DownloadManager._downloadMessages(DownloadManager.java:130)
    at de.fabianonline.telegram_backup.DownloadManager.downloadMessages(DownloadManager.java:75)
    at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.java:168)
    at de.fabianonline.telegram_backup.CommandLineRunner.main(CommandLineRunner.java:77)

----- EXIT -----
fabianonline commented 6 years ago

That was caused by a bug that lead to using a supergroup's max_message_id. That bug is fixed now, but if you were hit by it, you would probably have to restart the while backup to 'fix' it. Sorry for that. :-(