[backend] add Telegram

[Kcchouette]

What is your current rclone version (output from rclone version)?

rclone v1.53.3-DEV

What problem are you are trying to solve?

Adding a new backend (telegram): https://core.telegram.org/method/upload.saveBigFilePart and https://core.telegram.org/method/upload.getFile

[akhil-rana]

It would really be great if we can just add a aingle chat/channel/group or something as a remote where all the files/media in that chat can be treated as files in the remote.

One more approach can be to have Chat Names as folder names and just let those folders have respective files inside them.

[akhil-rana]

Also "chunker" can be really great with telegram since there is a 2gb file limit. So using a private channel or something as a storage solution for all the files might be a nice idea

[DevilStyle]

hi guys, I'm also interested in the implementation of telegram, is there any news? if you need some more information I can help you. Thanks a lot and happy holidays

[uGeek]

I also find it very interesting to use Telegram with rclone. Being able to have the encrypted files in a private Telegram channel and with rclone make backups, mount a WebDAV or SFTP server.

[uGeek]

Being able to mount the content of a channel is amazing. I have tried some old project, but with rclone it would be easier. In addition, it would allow to clone Telegram channels or groups.Also clone the content of a channel to Drive, Dropbox, ..

[DevilStyle]

Essere in grado di montare il contenuto di un canale è sorprendente. Ho provato qualche vecchio progetto, ma con rclone sarebbe stato più facile. Inoltre, consentirebbe di clonare canali o gruppi di Telegram. Clonare anche il contenuto di un canale su Drive, Dropbox, ..

Hello, yes exactly agree, plus telegram offers unlimited space. I hope the dev can do something. A gift for Christmas :)

[nikitabuida]

Would be great to have Telegram storage in RClone. https://github.com/SlavikMIPT/tgcloud - looks deserted but it allowed to mount tg storage directly into filesystem.

[ElDavoo]

So people will start storing PB of data on TG servers and they will then be forced to introduce individual quota. Great!

[birdup000]

Hello I'm a dev from Teledrive ready to start to integrate Rclone with the project. We're currently working on making this possible. -Thank you everyone for your ideas and we'll be looking on integration

[ElDavoo]

All of the rclone supported remotes are actual data storage solutions. Telegram is not a storage service but a messaging platform. I don't think ncw will approve it. You are of course allowed to develop code in your own fork but I don't think a pull request would be allowed.

[ncw]

@greengeckowizard Tell me more about your proposed integration? Would this call the telegram APIs directly?

[birdup000]

This would go to through existing Teledrive api for uploading and downloading

[birdup000]

Only issue is how would creating backend be possible?

[buengese]

Sorry to interrupt here. I think such a backend is a terrible idea. Telegram is a messenger and NOT a cloud storage provider. Abusing it as such is unacceptable and probably against their terms of service. I believe rclone should have no part in that.

I would very much prefer to close this and similar requests. @ncw has the final say though.

[birdup000]

Sorry to interrupt here. I think such a backend is a terrible idea. Telegram is a messenger and NOT a cloud storage provider. Abusing it as such is unacceptable and probably against their terms of service. I believe rclone should have no part in that.

I would very much prefer to close this and similar requests. @ncw has the final say though.

Look at the posts on their Twitter feed they want people to use it for unlimited storage. What abuse like spamming? Also not you're opinion on this I'll do it if it means I have to strip the backend on this to get it to work

[akhil-rana]

Sorry to interrupt here. I think such a backend is a terrible idea. Telegram is a messenger and NOT a cloud storage provider. Abusing it as such is unacceptable and probably against their terms of service. I believe rclone should have no part in that.

I would very much prefer to close this and similar requests. @ncw has the final say though.

I somewhat agree with you. But yeah telegram really advertises their unlimited storage like nothing else. Moreover now they are going to release a premium plan with 4gb file size limit and increased upload/download speeds and this backend might directly compete with that. Or on the other hand this backend might be more useful for paying members.

In any case only "read" access backend is also extremely useful for people so atleast that should be implemented.

[ncw]

If someone wants to make a backend for this, I don't have a problem provided

1. It uses only documented public APIs
2. It doesn't allow or encourage breaking the terms of service of the provider

I don't know whether an rclone backend would breach 2. above or not, some evidence either way would be nice.

[ElDavoo]

some evidence either way would be nice. The ToS of Telegram are very short and do not give clues about abusing their storage option. My guess it that they will add something once people will start spamming stuff.

[akhil-rana]

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together.

My plan is to use official telegram APIs directly.

[ncw]

@akhil-rana yes, that's fine - go ahead :-)

[birdup000]

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together.

My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

[shirooo39]

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together. My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

uhm, so... just wanna ask about how's the progress so far?

[akhil-rana]

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together. My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

uhm, so... just wanna ask about how's the progress so far?

I'm still stuck on properly implementing the auth. The telegram APIs are really bad in itself and that too with go makes it really difficult. Support for it is also really less.

I'm rethinking my approach right now

[shirooo39]

@akhil-rana

Is it okay if I start working on this? If anyone else is already working please communicate so that we can work together. My plan is to use official telegram APIs directly.

I was gonna start working on this but I'd be happy to be with you making this I'm a Teledrive Dev

uhm, so... just wanna ask about how's the progress so far?

I'm still stuck on properly implementing the auth. The telegram APIs are really bad in itself and that too with go makes it really difficult. Support for it is also really less.

I'm rethinking my approach right now

uhm... maybe @ncw can give you some enlightenment

[ElDavoo]

uhm... maybe @ncw can give you some enlightenment

He's a wonderful developer but I don't think he's a telegram expert. :)

[shirooo39]

uhm... maybe @ncw can give you some enlightenment

He's a wonderful developer but I don't think he's a telegram expert. :)

I have faith in the rclone god @ncw

[kouhe3]

It would be great if I could download files using rclone.

[johiny]

I will keep my eye on this looks so promising

[jacobmichels]

@akhil-rana Any updates? If you've stopped working on this, I might pick it up

[akhil-rana]

@akhil-rana Any updates? If you've stopped working on this, I might pick it up

Go ahead. I tried for a while but there were too many hurdles. And now I have no free time from my college.

[jacobmichels]

Ok I'm gonna take a stab at it then.

[akhil-rana]

Ok I'm gonna take a stab at it then.

This project might help you. They have done some good work here.

[jacobmichels]

So I've been playing around with the telegram API, and I've found a problem that I'd like to discuss.

For this backend, I think it'd be nice to represent paths in the form chat_name/file_name, for example if you had a chat called "siblings" and a file in that chat called wishlist.txt, you could reference this file in rclone with siblings/wishlist.txt. However, in telegram chat titles and filenames are not unique, you can have multiple chats with the same name and multiple files in the same chat with the same name. Both chats and files have unique integer IDs, but replacing titles and names with IDs makes it impossible to know what you're interacting with. For example, sibling/wishlist.txt could look like 008752/2836483 when using IDs instead of names.

I'm curious to hear if anyone has any ideas on working around this.

[ElDavoo]

I'd just copy the discord implementation

[jacobmichels]

@ElDavoo can you point me at that? Not quite sure what you're referencing

[Dark-kn1ght]

@jacobmichels probably this https://github.com/rclone/rclone/pull/6250

[jacobmichels]

Just tried out the discord backend, it's pretty interesting. I could attempt something similar with the telegram backend, but I think I'd prefer to treat chats as buckets if possible, rather than restricting file storage to one chat. I'm gonna keep experimenting but that is a good fallback

[Dark-kn1ght]

@jacobmichels if I may ask, are you considering having folders support? Or is it too hard to implement because it's Telegram?

[jacobmichels]

I'd like to integrate with telegram in an intuitive way. In my head that means making this backend closely resemble the official telegram clients. That idea translates to the ability to access the files within chats, but no folders within a chat.

I'm open to suggestions if anyone disagrees. This idea may not be realistic anyway given how the telegram API works.

[Dark-kn1ght]

I see, I mean, I'm already thankful that someone is trying to add a Telegram backend, as much as I'd want to have folders support, it's better than nothing, if you feel it's not worth or is just too hard to implement, that's completely fine.

[ncw]

I'm curious to hear if anyone has any ideas on working around this.

In the Google photos backend we have this problem too. If there are duplicate names in a directory then rclone writes them out as name {ID} and when parsing names if there is a {ID} it uses that and ignores the name

[jacobmichels]

In the Google photos backend we have this problem too. If there are duplicate names in a directory then rclone writes them out as name {ID} and when parsing names if there is a {ID} it uses that and ignores the name

I think I'll use this approach, thanks for showing this to me.

[Dark-kn1ght]

This probably won't work or will be too hard to implement but what about using the "message" part in the file as the folder path and filename? considering Telegram has a long limit on it, it might work? that might fix the issue without having to use IDs in the filename.

[D4rk4]

@greengeckowizard @ncw I give 250 USDT bounty to the person who implements the telegram backend for rclone when related MR is accepted.

[ElDavoo]

wow that's a plot twist

[D4rk4]

Any news?

[birdup000]

I've started a repository called RcloneTelegram I'm working on Rclone Support for Telegram

[birdup000]

If someone wants to make a backend for this, I don't have a problem provided

1. It uses only documented public APIs
2. It doesn't allow or encourage breaking the terms of service of the provider

I don't know whether an rclone backend would breach 2. above or not, some evidence either way would be nice.

Hi in my Rclone to Telegram which is the backend. I include these listed. https://github.com/greengeckowizard/RcloneTelegram

[musahi0128]

@ncw Please have a look at this and this

I am sure this telegram backend is better suited to run against a self-hosted telegram-bot-api. It's relatively easy to get one running using docker, for example:

docker -p 8081:8081 ghcr.io/bots-house/docker-telegram-bot-api:latest --local --api-id="2496" --api-hash="8da85b0d5bfe62527e5b244c209159c3" --max-webhook-connections=100000

I am no programmer but here is some of the public API I think relevant for this backend.

Uploading files: https://core.telegram.org/bots/api#senddocument

Downloading files: https://core.telegram.org/bots/api#getfile

We can use the caption to store the path information. Since there is 2gb file size limit (4gb for premium account), rclone should split file larger than that (or anything specified in the config) when uploading and append some information to the caption, so it is possible to differentiate between a whole file and file parts in a message and treat the files accordingly. Each API call require a bot, and one account is entitled to 20 bots max, that can be used in multi-threading.

I imagine the config entry will look like this

[telegram]
host = localhost
port = 8081
chat_id = XXXXXXXXXXXXXX
max_file_size = 2Gi
bot_01 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bot_02 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bot_03 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
...
bot_20 = XXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

There is this project that also utilize telegram unlimited storage and the dev also made a rclone version that work against its own API. The project is great, and I have been using it for a while. But I think using a direct API to Telegram will be much better and easier to manage for rclone, and this should require no database at all.

Please also note that I am willing to sponsor this backend.

Thanks!

[kkm]

+1