search

rclone / rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
https://rclone.org
MIT License
44.95k stars 4.03k forks source link

[backend] add Telegram #5829

Open Kcchouette opened 2 years ago

Kcchouette commented 2 years ago

What is your current rclone version (output from rclone version)?

rclone v1.53.3-DEV

What problem are you are trying to solve?

Adding a new backend (telegram): https://core.telegram.org/method/upload.saveBigFilePart and https://core.telegram.org/method/upload.getFile

akhil-rana commented 2 years ago

It would really be great if we can just add a aingle chat/channel/group or something as a remote where all the files/media in that chat can be treated as files in the remote.

One more approach can be to have Chat Names as folder names and just let those folders have respective files inside them.

akhil-rana commented 2 years ago

Also "chunker" can be really great with telegram since there is a 2gb file limit. So using a private channel or something as a storage solution for all the files might be a nice idea

DevilStyle commented 2 years ago

hi guys, I'm also interested in the implementation of telegram, is there any news? if you need some more information I can help you. Thanks a lot and happy holidays

uGeek commented 2 years ago

I also find it very interesting to use Telegram with rclone. Being able to have the encrypted files in a private Telegram channel and with rclone make backups, mount a WebDAV or SFTP server.

uGeek commented 2 years ago

Being able to mount the content of a channel is amazing. I have tried some old project, but with rclone it would be easier. In addition, it would allow to clone Telegram channels or groups.Also clone the content of a channel to Drive, Dropbox, ..

DevilStyle commented 2 years ago

Essere in grado di montare il contenuto di un canale è sorprendente. Ho provato qualche vecchio progetto, ma con rclone sarebbe stato più facile. Inoltre, consentirebbe di clonare canali o gruppi di Telegram. Clonare anche il contenuto di un canale su Drive, Dropbox, ..

Hello, yes exactly agree, plus telegram offers unlimited space. I hope the dev can do something. A gift for Christmas :)

nikitabuida commented 2 years ago

Would be great to have Telegram storage in RClone. https://github.com/SlavikMIPT/tgcloud - looks deserted but it allowed to mount tg storage directly into filesystem.

ElDavoo commented 2 years ago

So people will start storing PB of data on TG servers and they will then be forced to introduce individual quota. Great!

birdup000 commented 2 years ago

Hello I'm a dev from Teledrive ready to start to integrate Rclone with the project. We're currently working on making this possible. -Thank you everyone for your ideas and we'll be looking on integration

ElDavoo commented 2 years ago

All of the rclone supported remotes are actual data storage solutions. Telegram is not a storage service but a messaging platform. I don't think ncw will approve it. You are of course allowed to develop code in your own fork but I don't think a pull request would be allowed.

ncw commented 2 years ago

@greengeckowizard Tell me more about your proposed integration? Would this call the telegram APIs directly?

birdup000 commented 2 years ago

This would go to through existing Teledrive api for uploading and downloading

birdup000 commented 2 years ago

Only issue is how would creating backend be possible?

buengese commented 2 years ago

Sorry to interrupt here. I think such a backend is a terrible idea. Telegram is a messenger and NOT a cloud storage provider. Abusing it as such is unacceptable and probably against their terms of service. I believe rclone should have no part in that.

I would very much prefer to close this and similar requests. @ncw has the final say though.

birdup000 commented 2 years ago

Sorry to interrupt here. I think such a backend is a terrible idea. Telegram is a messenger and NOT a cloud storage provider. Abusing it as such is unacceptable and probably against their terms of service. I believe rclone should have no part in that.

I would very much prefer to close this and similar requests. @ncw has the final say though.

Look at the posts on their Twitter feed they want people to use it for unlimited storage. What abuse like spamming? Also not you're opinion on this I'll do it if it means I have to strip the backend on this to get it to work

akhil-rana commented 2 years ago

Sorry to interrupt here. I think such a backend is a terrible idea. Telegram is a messenger and NOT a cloud storage provider. Abusing it as such is unacceptable and probably against their terms of service. I believe rclone should have no part in that.

I would very much prefer to close this and similar requests. @ncw has the final say though.

I somewhat agree with you. But yeah telegram really advertises their unlimited storage like nothing else. Moreover now they are going to release a premium plan with 4gb file size limit and increased upload/download speeds and this backend might directly compete with that. Or on the other hand this backend might be more useful for paying members.

In any case only "read" access backend is also extremely useful for people so atleast that should be implemented.

ncw commented 2 years ago

If someone wants to make a backend for this, I don't have a problem provided

  1. It uses only documented public APIs
  2. It doesn't allow or encourage breaking the terms of service of the provider

I don't know whether an rclone backend would breach 2. above or not, some evidence either way would be nice.

ElDavoo commented 2 years ago

some evidence either way would be nice. The ToS of Telegram are very short and do not give clues about abusing their storage option. My guess it that they will add something once people will start spamming stuff.

akhil-rana commented 1 year ago

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together.

My plan is to use official telegram APIs directly.

ncw commented 1 year ago

@akhil-rana yes, that's fine - go ahead :-)

birdup000 commented 1 year ago

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together.

My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

shirooo39 commented 1 year ago

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together. My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

uhm, so... just wanna ask about how's the progress so far?

akhil-rana commented 1 year ago

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together. My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

uhm, so... just wanna ask about how's the progress so far?

I'm still stuck on properly implementing the auth. The telegram APIs are really bad in itself and that too with go makes it really difficult. Support for it is also really less.

I'm rethinking my approach right now

shirooo39 commented 1 year ago

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together. My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

uhm, so... just wanna ask about how's the progress so far?

I'm still stuck on properly implementing the auth. The telegram APIs are really bad in itself and that too with go makes it really difficult. Support for it is also really less.

I'm rethinking my approach right now

uhm... maybe @ncw can give you some enlightenment

ElDavoo commented 1 year ago

uhm... maybe @ncw can give you some enlightenment

He's a wonderful developer but I don't think he's a telegram expert. :)

shirooo39 commented 1 year ago

uhm... maybe @ncw can give you some enlightenment

He's a wonderful developer but I don't think he's a telegram expert. :)

I have faith in the rclone god @ncw

kouhe3 commented 1 year ago

It would be great if I could download files using rclone.

johiny commented 1 year ago

I will keep my eye on this looks so promising

jacobmichels commented 1 year ago

@akhil-rana Any updates? If you've stopped working on this, I might pick it up

akhil-rana commented 1 year ago

@akhil-rana Any updates? If you've stopped working on this, I might pick it up

Go ahead. I tried for a while but there were too many hurdles. And now I have no free time from my college.

jacobmichels commented 1 year ago

Ok I'm gonna take a stab at it then.

akhil-rana commented 1 year ago

Ok I'm gonna take a stab at it then.

This project might help you. They have done some good work here.

jacobmichels commented 1 year ago

So I've been playing around with the telegram API, and I've found a problem that I'd like to discuss.

For this backend, I think it'd be nice to represent paths in the form chat_name/file_name, for example if you had a chat called "siblings" and a file in that chat called wishlist.txt, you could reference this file in rclone with siblings/wishlist.txt. However, in telegram chat titles and filenames are not unique, you can have multiple chats with the same name and multiple files in the same chat with the same name. Both chats and files have unique integer IDs, but replacing titles and names with IDs makes it impossible to know what you're interacting with. For example, sibling/wishlist.txt could look like 008752/2836483 when using IDs instead of names.

I'm curious to hear if anyone has any ideas on working around this.

ElDavoo commented 1 year ago

I'd just copy the discord implementation

jacobmichels commented 1 year ago

@ElDavoo can you point me at that? Not quite sure what you're referencing

Dark-kn1ght commented 1 year ago

@jacobmichels probably this https://github.com/rclone/rclone/pull/6250

jacobmichels commented 1 year ago

Just tried out the discord backend, it's pretty interesting. I could attempt something similar with the telegram backend, but I think I'd prefer to treat chats as buckets if possible, rather than restricting file storage to one chat. I'm gonna keep experimenting but that is a good fallback

Dark-kn1ght commented 1 year ago

@jacobmichels if I may ask, are you considering having folders support? Or is it too hard to implement because it's Telegram?

jacobmichels commented 1 year ago

I'd like to integrate with telegram in an intuitive way. In my head that means making this backend closely resemble the official telegram clients. That idea translates to the ability to access the files within chats, but no folders within a chat.

I'm open to suggestions if anyone disagrees. This idea may not be realistic anyway given how the telegram API works.

Dark-kn1ght commented 1 year ago

I see, I mean, I'm already thankful that someone is trying to add a Telegram backend, as much as I'd want to have folders support, it's better than nothing, if you feel it's not worth or is just too hard to implement, that's completely fine.

ncw commented 1 year ago

I'm curious to hear if anyone has any ideas on working around this.

In the Google photos backend we have this problem too. If there are duplicate names in a directory then rclone writes them out as name {ID} and when parsing names if there is a {ID} it uses that and ignores the name

jacobmichels commented 1 year ago

In the Google photos backend we have this problem too. If there are duplicate names in a directory then rclone writes them out as name {ID} and when parsing names if there is a {ID} it uses that and ignores the name

I think I'll use this approach, thanks for showing this to me.

Dark-kn1ght commented 1 year ago

This probably won't work or will be too hard to implement but what about using the "message" part in the file as the folder path and filename? considering Telegram has a long limit on it, it might work? that might fix the issue without having to use IDs in the filename.

D4rk4 commented 1 year ago

@greengeckowizard @ncw I give 250 USDT bounty to the person who implements the telegram backend for rclone when related MR is accepted.

ElDavoo commented 1 year ago

wow that's a plot twist

D4rk4 commented 1 year ago

Any news?

birdup000 commented 1 year ago

I've started a repository called RcloneTelegram I'm working on Rclone Support for Telegram

birdup000 commented 1 year ago

If someone wants to make a backend for this, I don't have a problem provided

  1. It uses only documented public APIs
  2. It doesn't allow or encourage breaking the terms of service of the provider

I don't know whether an rclone backend would breach 2. above or not, some evidence either way would be nice.

Hi in my Rclone to Telegram which is the backend. I include these listed. https://github.com/greengeckowizard/RcloneTelegram

musahi0128 commented 9 months ago

@ncw Please have a look at this and this

I am sure this telegram backend is better suited to run against a self-hosted telegram-bot-api. It's relatively easy to get one running using docker, for example:

docker -p 8081:8081 ghcr.io/bots-house/docker-telegram-bot-api:latest --local --api-id="2496" --api-hash="8da85b0d5bfe62527e5b244c209159c3" --max-webhook-connections=100000

I am no programmer but here is some of the public API I think relevant for this backend.

Uploading files: https://core.telegram.org/bots/api#senddocument

Downloading files: https://core.telegram.org/bots/api#getfile

We can use the caption to store the path information. Since there is 2gb file size limit (4gb for premium account), rclone should split file larger than that (or anything specified in the config) when uploading and append some information to the caption, so it is possible to differentiate between a whole file and file parts in a message and treat the files accordingly. Each API call require a bot, and one account is entitled to 20 bots max, that can be used in multi-threading.

I imagine the config entry will look like this

[telegram]
host = localhost
port = 8081
chat_id = XXXXXXXXXXXXXX
max_file_size = 2Gi
bot_01 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bot_02 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bot_03 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
...
bot_20 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

There is this project that also utilize telegram unlimited storage and the dev also made a rclone version that work against its own API. The project is great, and I have been using it for a while. But I think using a direct API to Telegram will be much better and easier to manage for rclone, and this should require no database at all.

Please also note that I am willing to sponsor this backend.

Thanks!