Error: interface conversion: tg.MessageClass is *tg.MessageEmpty, not *tg.Message when trying to download deleted message

[steffen-poulsen]

I get this error when I try to download files that were uploaded in 2023.

Initial inspection suggests that files uploaded before 9-Dec-2023 fail, files uploaded after 12-Dec-2023 are ok.

This is a docker install off the main branch. Just updated recently, hasn't been updated for a while.

teldrive  | 31/05/2024 12:39 PM DEBUG   requesting file {"name": "<redacted>", "bot": "<redacted>", "botNo": 1, "start": 0, "end": 1890758655, "fileSize": 1890758656}
teldrive  | 2024/05/31 12:39:54 http: panic serving 172.18.0.1:49150: interface conversion: tg.MessageClass is *tg.MessageEmpty, not *tg.Message
teldrive  | goroutine 1573 [running]:
teldrive  | net/http.(*conn).serve.func1()
teldrive  |     net/http/server.go:1898 +0xbe
teldrive  | panic({0x1834860?, 0xc0001208d0?})
teldrive  |     runtime/panic.go:770 +0x132
teldrive  | github.com/divyam234/teldrive/pkg/services.getParts({0x2214648, 0xc000264c00}, 0xc00053bc08, 0xc0012154a0, {0xc0010c75c0, 0xa})
teldrive  |     github.com/divyam234/teldrive/pkg/services/common.go:229 +0x51e
teldrive  | github.com/divyam234/teldrive/pkg/services.(*FileService).GetFileStream(0xc000010960, 0xc000264c00)
teldrive  |     github.com/divyam234/teldrive/pkg/services/file.go:600 +0x143e
teldrive  | github.com/divyam234/teldrive/pkg/controller.(*Controller).GetFileStream(...)
teldrive  |     github.com/divyam234/teldrive/pkg/controller/file.go:196
teldrive  | github.com/gin-gonic/gin.(*Context).Next(...)
teldrive  |     github.com/gin-gonic/gin@v1.10.0/context.go:185
teldrive  | github.com/divyam234/teldrive/cmd.initApp.func1(0xc000264c00)
teldrive  |     github.com/divyam234/teldrive/cmd/run.go:227 +0x1a3
teldrive  | github.com/gin-gonic/gin.(*Context).Next(0xc000264c00)
teldrive  |     github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
teldrive  | github.com/divyam234/teldrive/cmd.initApp.GinzapWithConfig.func4(0xc000264c00)
teldrive  |     github.com/gin-contrib/zap@v1.1.3/zap.go:68 +0xb1
teldrive  | github.com/gin-gonic/gin.(*Context).Next(...)
teldrive  |     github.com/gin-gonic/gin@v1.10.0/context.go:185
teldrive  | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0000d8d00, 0xc000264c00)
teldrive  |     github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x892
teldrive  | github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0000d8d00, {0x2211720, 0xc0000f6620}, 0xc0000feea0)
teldrive  |     github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1b2
teldrive  | net/http.serverHandler.ServeHTTP({0xc0012152f0?}, {0x2211720?, 0xc0000f6620?}, 0x6?)
teldrive  |     net/http/server.go:3137 +0x8e
teldrive  | net/http.(*conn).serve(0xc00100d0e0, {0x2214418, 0xc0001c1650})
teldrive  |     net/http/server.go:2039 +0x5e8
teldrive  | created by net/http.(*Server).Serve in goroutine 40
teldrive  |     net/http/server.go:3285 +0x4b4

[steffen-poulsen]

Ok, it seems I have hit a 1.000.000 file limit for telegram channels here.

My last entries in the "parts" column of the "files" table has crossed 1.000.000 by almost 190.000:

"[{""id"": 1189026, ""salt"": <redacted>}]"

And when I try to retrieve a file part using manual links in the browser, all parts before 190.000 are simply gone:

• https://t.me/c//180000 - "message does not exist"
• https://t.me/c//190000 - found ok

Also, my "Channel info" in the Telegram app shows that there are only 999.860 files in the channel - not 1.189.026 files as expected.

The page describing the limits for telegram does however not seem to mention a 1.000.000 file limit: https://limits.tginfo.me/en. Some sources say its a known bug for supergroups and channels.

Anyway, to sum up: File uploads beyond 1.000.000 parts simply pushes out/deletes the oldest parts quietly, FIFO-style.

Suggestions:

• Let teldrive return with a "file not found" message, when a part is not found (not the current interface conversion exception). At the same time, the file entry should be deleted from the files table, and other mentioned parts for the file should be deleted from the telegram backend.
• Let teldrive return a "drive is full" message when trying to upload a file, when the number of parts approaches 1.000.000.
• Let teldrive offer a "Split File Size" of 4GB (to support the file size limit for Premium users). Let the default be 2GB.
• Let teldrive show the number of files in each folder when browsing (similar to how the storage used is shown).

As my teldrive database and the telegram backend are now out of sync, I'm wondering how to bring them back in sync again. A magic "reestablish teldrive database integrity against telegram backend"-button would have been be absolutely marvelous here :D

The magic button would:

1) Walk through all files in the teldrive database, making sure each file is still available in the telegram backend.

• If a part is not available, the file is deleted from the database and all parts of the file deleted at the telegram backend
• If all parts of the file are available, the timestamp for the last successful availability check of the file is updated 2) Walk through the range of all part ids that could have been created in the Telegram backend at one point (1 -> max part number).
• If the part id belongs to a file that was succesfully checked in 1), we're good
• If the part id is not part of a file in the teldrive database, delete the file from the Telegram backend

This could also be a scheduled "scrubbing" job, running periodically in the background.