search

hinsley / tglfs

Telegram Large File Storage
https://t.me/tglfs
MIT License
1 stars 0 forks source link

Not able to use large chunk sizes #9

Open hinsley opened 6 months ago

hinsley commented 6 months ago

When I make the chunk size anything much larger than 0.3GB, uploads will work for a while on uploading the first chunk before finally giving up and saying Killed, force closing the application. I am not yet sure what is doing this, but the simple fact that the application itself closes out is indicative of a memory limitation. I need to investigate to be sure, as Telegram should not be imposing hidden limits like this.

hinsley commented 6 months ago

I have now had success sending a file in chunks which were 1.0GB pre-encryption (1365.3MB post-encryption). Not sure what the limit is, and still don't know why it says Killed for larger files. I didn't have success sending 1.5GB pre-encryption chunks.

hinsley commented 6 months ago

It seems like there might be a hard limit of 1.5GB per file allowed by user bots. It would be good if I could encrypt large files in a streaming manner instead of encrypting each chunk separately; then I could partition into chunks exactly according to the file size limit. This is worth investigating further; I may make another issue for it. I am considering this issue resolved.

hinsley commented 6 months ago

No! This seems like a local error. I think I'm running out of memory on my machine. From the Telethon docs:

client.upload_file() now works better when streaming files (anything that has a .read()), instead of reading it all into memory when possible.

I should try using this.

hinsley commented 6 months ago

Just found out this is due to the encryption. Parallax chunking should fix.