search

luiscib3r / Telezon-S3

Telezon S3 is a storage service that implements Amazon S3-compatible API using Telegram as the storage backend. It allows you to store and retrieve files using standard S3 clients while leveraging Telegram's infrastructure for actual file storage.
MIT License
13 stars 2 forks source link

Serving Files #5

Open gr8rehanna opened 5 hours ago

gr8rehanna commented 5 hours ago

Better approach to serve files Well thanks for the update..,.

Its now better and works fine but i noticed 1 major flaw.

Problem Now as i used this as an S3 alternative on mastodon. Its a small local development server. When we upload files they are now getting uploaded now even we can upload very larger files. When a user opens the timeline and suppose timeline will show 100 images. So then S3 has to fetch 100 images in no times. Our bot does not handle file as streaming or serving directly from telegram. Instead it uses tempfile to download the files on server then send to user/mastodon.

Actual Problem. Here is the thing, if we got 10K users and at same time 5K or even 1K users will open timeline. So each users timeline showing 100 images = 10K or 50K images to served to users. So our s3 server has to download 50K files from telegram and store on server in tempfile and serve.

Worse thing. Telegram rate limit is imposed on bots for too many concurrent request

Solution If possible can we directly stream files from telegram instead of downloading files on server eveerytime.

Additional context Well its my personal experience maybe i am wrong but i But as per below code when we call get_file it downloads file and sends to users. image

As i guess bots can not stream files directly from telegram. How about users? Session string of premium account can also improve uploading speed. And most important feature is serving files to millions without download in server and also maintaining speed and performance. We can use multi session strings/bots to handle uploads . So which ever bot is available will handle the uploading (I have implemented this)

At last serving files remains an issue. Sorry for creating new issues instead of opening existing one. Also got a job so not often on local projects. And its a hobby project if you are busy or not interested just ignore.

Thanks

luiscib3r commented 4 hours ago

Thank you for your valuable feedback! While I'm not actively developing it, I'm really glad to see someone else using this project. I recently made an update that should address the file serving concerns.

A pro tip: you can use a personal Telegram account instead of a bot, which doesn't have the same rate limitations. I personally use my Telegram account when working with this project. To set this up:

Run make setup_account_storage Enter your Telegram phone number (instead of a bot token) Input the authentication code you'll receive Use the generated session string to run the service

This will create a persistent session, so you won't need to authenticate again in future runs. Let me know if you have any questions.

luiscib3r commented 4 hours ago

Just want to add an important note about scalability: For high-traffic projects, Telegram's limitations might eventually become a bottleneck that can't be circumvented. In such cases, I'd recommend using actual S3 or another storage solution specifically designed for high-concurrency file serving. This project works great for personal/small deployments, but it's important to acknowledge its limitations for production environments with high demand.