search

fabianonline / telegram_backup

Java app to download all your telegram data.
GNU General Public License v3.0
447 stars 91 forks source link

(not ready for merging yet) update dependency and telegram api layer from 53 to 66 #88

Open leijurv opened 6 years ago

leijurv commented 6 years ago

Initial attempt to update telegram api layer and dependency.

Version 1.0.0-RC2 seems to work with minimal changes, and seems the most promising for fixing my error. Going beyond that (1.0.0-RC3 and 1.0.0-RC4) cause many dependency errors.

Goal is to fix this error:

........   ......... .. . . ........ .  . .  .. .. .    ....  .. ..... ......  ...  ............ .  - 90800/121491
   .   .... ...  ........... .      ...  .  ...  ...  .    .. .       .  . .. ........ ...   ....... - 90900/121491
......................................................................... ...................... ... - 91000/121491
....... ......... ..  .     ..11:05:27.480 DEBUG DownloadManager.downloadFileFromDc : Downloading file /mnt/tgdata//+REDACTEDXXX/files/channel_1088057488_145.pdf
400 error code
400: DC_ID_INVALID
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQueries(DefaultTelegramClient.kt:209)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQueries$default(DefaultTelegramClient.kt:181)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQueries(DefaultTelegramClient.kt:160)
        at com.github.badoualy.telegram.api.TelegramClient$DefaultImpls.executeRpcQuery(TelegramClient.kt:53)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQuery(DefaultTelegramClient.kt:157)
        at com.github.badoualy.telegram.tl.api.TelegramApiWrapper.authExportAuthorization(TelegramApiWrapper.java:213)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.getExportedMTProtoHandler(DefaultTelegramClient.kt:293)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQueries(DefaultTelegramClient.kt:200)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQueries$default(DefaultTelegramClient.kt:181)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQueries(DefaultTelegramClient.kt:171)
        at com.github.badoualy.telegram.api.TelegramClient$DefaultImpls.executeRpcQuery(TelegramClient.kt:56)
        at com.github.badoualy.telegram.api.DefaultTelegramClient.executeRpcQuery(DefaultTelegramClient.kt:25)
        at de.fabianonline.telegram_backup.DownloadManager$Companion.downloadFileFromDc(DownloadManager.kt:429)
        at de.fabianonline.telegram_backup.DownloadManager$Companion.downloadFile(DownloadManager.kt:399)
        at de.fabianonline.telegram_backup.mediafilemanager.DocumentFileManager.download(DocumentFileManager.kt:102)
        at de.fabianonline.telegram_backup.DownloadManager._downloadMedia(DownloadManager.kt:367)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMedia(DownloadManager.kt:317)
        at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.kt:138)
        at de.fabianonline.telegram_backup.CommandLineRunnerKt.main(CommandLineRunner.kt:40)
400 error code
D        .....  .. .... ..        .... ............ .............. . . - 91100/121491
. ...     . ...     ...  .. ...  .    .. . ..... . .... .. .. . .................. .....     .. ...e - 91200/121491-

Note: running this version requires a redownload of every media message due to the API layer being too old. This updates the api_layer for every entry in the messages table (where has_media=1) from 53 to 66.

I'm not sure if some data is being lost in this transition...

@fabianonline what needs to change when updating the telegram api layer? Maybe new columns need to be added to the messages database. It's redownloading but nothing's changed, I think it's missing something.

leijurv commented 6 years ago

Note: this PR is built on top of #87 and includes the changes there

leijurv commented 6 years ago

Right now I am running this new version and waiting for it to finish redownloading ~120,000 messages. I checked, and the number of messages it's redownloading is exactly equal to the number of messages where has_media=1 and api_layer=53.

Once that completes, I'll be able to see if updating this dependency fixes the DC_ID_INVALID error from before. At least until then, this should definitely not be merged because I have no idea what the other effects of updating this dependency might be.

leijurv commented 6 years ago

Never mind on this, layer 66 has introduced many new things that appear to not work.

java.lang.RuntimeException: Unexpected com.github.badoualy.telegram.tl.api.TLMessageMediaGame
        at de.fabianonline.telegram_backup.mediafilemanager.AbstractMediaFileManager$Companion.throwUnexpectedObjectError(AbstractMediaFileManager.kt:97)
        at de.fabianonline.telegram_backup.mediafilemanager.FileManagerFactory.getFileManager(FileManagerFactory.kt:69)
        at de.fabianonline.telegram_backup.Database.saveMessages(Database.kt:352)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMessages(DownloadManager.kt:296)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMessages$default(DownloadManager.kt:242)
        at de.fabianonline.telegram_backup.DownloadManager._downloadMedia(DownloadManager.kt:347)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMedia(DownloadManager.kt:318)
        at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.kt:138)
        at de.fabianonline.telegram_backup.CommandLineRunnerKt.main(CommandLineRunner.kt:40)
java.lang.RuntimeException: Exception shown above happened.
        at de.fabianonline.telegram_backup.Database.saveMessages(Database.kt:436)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMessages(DownloadManager.kt:296)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMessages$default(DownloadManager.kt:242)
        at de.fabianonline.telegram_backup.DownloadManager._downloadMedia(DownloadManager.kt:347)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMedia(DownloadManager.kt:318)
        at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.kt:138)
        at de.fabianonline.telegram_backup.CommandLineRunnerKt.main(CommandLineRunner.kt:40)
13:06:58.205 ERROR CommandLineController.<init>       : Exception caught!
java.lang.RuntimeException: Exception shown above happened.
        at de.fabianonline.telegram_backup.Database.saveMessages(Database.kt:436)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMessages(DownloadManager.kt:296)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMessages$default(DownloadManager.kt:242)
        at de.fabianonline.telegram_backup.DownloadManager._downloadMedia(DownloadManager.kt:347)
        at de.fabianonline.telegram_backup.DownloadManager.downloadMedia(DownloadManager.kt:318)
        at de.fabianonline.telegram_backup.CommandLineController.<init>(CommandLineController.kt:138)
        at de.fabianonline.telegram_backup.CommandLineRunnerKt.main(CommandLineRunner.kt:40)

----- EXIT -----
fabianonline commented 6 years ago

Problem also is: As soon as I update the Kotlogram version to another one with newer api_layer, I can't ever use the old message objects in the database again. (I don't mean "read the values from the table", but "use the binary data to reconstruct the original message object".) I want to keep this option to be able to later maybe get data from messages that right now doesn't seem to be important. I think the right way to do this is by converting the message objects to JSON and storing that in the database instead of the raw message objects...

leijurv commented 6 years ago

@fabianonline i'm at message ID 997,004 so time is running out =)