Bot api server general questions

[Fedorus]

1) Is it right that this is bot-api to client-api proxy for bots? 2) Can we extend this server api with methods that exist in client api and can be called by bots but does not exist in bot api?

[levlam]

1. Yes. And it ever was.
2. No. Bots have access only to methods required for the Bot API implementation.

[Fedorus]

This is list of methods from client api which can be called while logged in as bot: https://tl.telethon.dev/methods/botindex.html And among them there is GetFullUser method that does not exist in official bot API... but still can be used by bots...

[giautm]

Hey, Is there an docker version to deploy bot server to any cloud that support docker?

[levlam]

@Fedorus This is not true. GetFullUser is used for getChat implementation. All UserFull fields not included in getChat response are useless for bots: bots don't have peer settings, can't change chat notification settings, can't block users, can't call users, don't have privacy settings, don't have scheduled messages, don't have video calls and can't move chat with the user to Archive.

[levlam]

@giautm No. You need to build the Bot API server yourself using provided build instructions to use.

[gabbhack]

What about limits on sending messages and so on? As far as I know, the limits for bots using the client-api (telethon/pyrogram) are stricter than for bots using official bot-api. Does this depend on whether the app_id is "trusted"? If bot support has increased limits for my bot, will they still apply on my server?

[fskuratov]

So, briefly, as far as I understand no new methods for Bots are available? No Recent actions history, or chat members list, right? Any exclusions maybe, or someone can add them by forking and rewriting the current BotAPI server?

[levlam]

@gabbhack

What about limits on sending messages and so on?

Limits always was server-side and never depended on api_id or something similar.

As far as I know, the limits for bots using the client-api (telethon/pyrogram) are stricter than for bots using official bot-api.

This is false.

If bot support has increased limits for my bot, will they still apply on my server?

Yes.

[levlam]

So, briefly, as far as I understand no new methods for Bots are available?

Briefly, you can run Bot API server locally now.

No Recent actions history, or chat members list, right?

Yes, because that polling of those lists wouldn't work. You can't poll Recent actions for million chats in runtime. Stay tuned for the future updates, though.

Any exclusions maybe, or someone can add them by forking and rewriting the current BotAPI server? No, it isn't possible to add them currently.

[RememberTheAir]

No. Bots have access only to methods required for the Bot API implementation.

Just a clarification: as far as I remember, using any MTProto client and logging in as a bot account, the server doesn't forbid you to send a channels.getParticipants request. A similar method is not available in the bot api, as the bot api doesn't expose any method to fetch a chat's members list. What is currently stopping someone from forking this project and add a method to the current available methods to request the members list using tdlib's getSupergroupMembers? Maybe tdlib doesn't let you use such methods if you login as a bot - or underneath getSupergroupMembers uses some other mtproto method that is not available to bots (so one would have to fork this project and tdlib to add this functionality)?

[levlam]

@RememberTheAir Just a clarification: getSupergroupMembers is required for the Bot API implementation, and you can use it with filters not exposed in the Bot API. But flood limits for such usages aren't increased for bots, so you are likely to exceed them very fast.

[RememberTheAir]

@levlam ah, I see. One should expect mtproto/tdlib to accept just method requests strictly required to what the bot api allows to do - getSupergroupMembers is required to the current set of functionalities offered by the bot API (eg, getting a group's administrators), so it's not something Telegram left out "just because". Makes sense, thanks for the clarification. That also means that there are some MTProto methods that can be more broadly extended by a bot api fork (such as getSupergroupMembers), if you take in account the tight limits enforced by telegram's servers

[funnyflowerpot]

Thanks @levlam for proving quick and helpful answers here!

Question: What parts of the data transfer between Telegram client, Telegram bot, and local Bot API server are going through official Telegram servers?

In the previous version of the Bot API the data flows were pretty clear to me, but now I find it more difficult to see which network host is talking to whom and when. I was searching for details or maybe something like a sequence diagram, but I couldn't find related information so far.

I am also asking from a data privacy perspective. So in other words: What data (and metadata) could be still read by official Telegram servers, when a local Bot API server is used? I think including such information in the readme file of this repository would be great in terms of transparency.

[funnyflowerpot]

Here is how I understand the network communication between the relevant hosts Telegram client (C), Telegram bot (B), local Bot API server (BAS), and official Telegram servers (OTS). Is the following outline correct? If I missed a section in official documents describing this, please let me know!

1. BAS starts (no network requests here).
2. B starts and registers with BAS.
3. B tells OTS that it can be reached via BAS.
4. Using C a user attempts to start a chat with B.
5. C asks OTS for information on B.
6. OTS tells C that B's API server is BAS.
7. C sends data to BAS, BAS forwards data to B, and vice versa.

And so:

• All subsequent communication between C and B happen with BAS as a proxy.
• C and B never communicate directly with each other. (This also was the case before local Bot API servers.)
• OTS are only needed to establish communication between C and BAS in the beginning, but not afterwards.
• OTS and BAS never communicate directly with each other.

Is this description correct? Thank you!

[RememberTheAir]

Sorry for the low effort diagram. This is a map of how the various involved entities interact with each other.

Using your abbreviations, C never directly interact with BAS (that is, the user's Telegram app never directly connects to the server where an instance of telegram-bot-api is running). All the communications go through Telegram's servers (OTS), which forwards them to the two mtproto clients involved: the Telegram app the user C is using on their device, and the tdlib instance telegram-bot-api (BAS) is running under the hood.

BAS just acts as a front-end to the Telegram API, which is way more difficult to implement than a common Rest API (which is what BAS is supposed to be - an additional level of abstraction to interact with the Telegram API as a bot account). A rest api is more familiar and friendly to developers, so it's easier to pick up.

Note: in the image above, you can see only one telegram-bot-api instance, but there can be a number of these now, each one of them used by a number of bots. Also, the machine where the api is hosted can also be used to host the actual bot code, of course

[itsMoji]

Is there anyway to get file upload/download progress?

[funnyflowerpot]

@RememberTheAir Thank you very much for the explanation! The diagram also helps a lot, particularly the details that the Bot API server is a tdlib client and communicates via MTProto. This helped me to better grasp the idea behind tdlib and puts the benefits of a local Bot API server into perspective. Thanks again!

[micheleselea]

Why using the bot server api locally should improve performance? There is just one more layer instead to communicate directly with telegram servers.. I don't get why should be better instead using bot api as usual. If the architecture whould like @funnyflowerpot said There will be benefit otherwise I don't understand

[JrooTJunior]

Hey, Is there an docker version to deploy bot server to any cloud that support docker?

https://hub.docker.com/r/aiogram/telegram-bot-api

[kolayne]

Why using the bot server api locally should improve performance? There is just one more layer instead to communicate directly with telegram servers. I don't get why should be better instead using bot api as usual If the architecture whould like @funnyflowerpot said There will be benefit otherwise I don't understand

Absolutely agree. At first, I supposed it was created to accept PRs implementing new methods or something like that from the developers using Telegram API. Then I figured out that's not an instance of what is running on the Telegram side and started thinking something similar to what @RememberTheAir has described. After reading his explanation I felt like bingo!.. But that's wrong too...

@RememberTheAir, @funnyflowerpot, could you, please, help me understand, what are the advantages of a local bot API server?

[gabbhack]

@micheleselea

I don't get why should be better instead using bot api as usual.

The regular bot api is also an "extra" layer between bot and Telegram servers. A local bot api can improve performance due to a low ping and an extended limits.

[kolayne]

@gabbhack, does it mean that usual bots interact with Telegram servers via Telegram's instance of this project (global Bot API Server?)? Are files stored on bot API servers rather than on main Telegram servers (tdlib)? Will they send a request to my local Bot API Server each time a user wants to download a file sent by my bot?

[gabbhack]

@kolayne

does it mean that usual bots interact with Telegram servers via Telegram's instance of this project (global Bot API Server?)?

Yes.

Are files stored on bot API servers rather than on main Telegram servers (tdlib)? Will they send a request to my local Bot API Server each time a user wants to download a file sent by my bot?

All files are stored on Telegram servers, not on bot api server, but I think bot api can cache [1] [2] files (or something else) that the bot requested to download.

[micheleselea]

@gabbhack ok for the limitation extended with local server, but the lower ping time can be just from you and your bot api, and than you have the "ping time" between bot api server and telegram. I see only one more node that can be slow down communication, because you always speak with a bot api even in the standard configuration, but is a bot api that run on telegram server so the mtptoto communication I think it's faster than which one with local server api and telegram. I keep thinking that is for limitation extension and not for performance reason. @levlam what do you think about? that whould really improve performance? and if so why?

[gabbhack]

@micheleselea Popular bots have a problem - their servers can handle a large number of requests, but bot api sends maximum of 100 simultaneous requests (see setWebhok max_connections parameter). This causes some users to get a response from the bot later, but it's not about the power of the bot's server. Local bot api can send 100,000 simultaneous requests, which may increases the bot's throughput.

These results may not be relevant, but a simple query was faster with a local server. Of course, this depends on the server location: https://t.me/aiogram_ru/334063

[micheleselea]

ok understand, because the MTProto communication between local bot and telegram bot is not limited as direct bot API are. So the local-bot-api implementation is not restricted from the point of view of maximum number of https request?

[micheleselea]

So basically the main advantage of this infrastrutture is that the conversion between HTTP BOT API and MTPROTO is done locally and than MTPROTO si used to "multiplex" the BOT API CALL in a already established channel. That way can be interesting to test...I'll try

[funnyflowerpot]

@kolayne @micheleselea From another issue:

[A server run by means of this project] is exactly the same server as run at https://api.telegram.org.

This is where I see the advantages of a bot API server: In the diagram above we see an official Telegram server, the bot API server and bots. If the bot API server does not run on the same host as an official Telegram server, but bot API server and bots share the same host, then network communication between network hosts does not happen over HTTP, but MTProto. Compared to HTTP the MTProto protocol is most surely providing better performance for network communication with Telegram apps in most cases.

And as @gabbhack said web hooks for real-time communication might be a good reason for a local bot API server too. After all HTTP and in particular an additional TLS layer (HTTPS) can provide quite some overhead. Keeping this kind of communication within a host and using MTProto's (builtin) encryption can help a lot I guess.

[funnyflowerpot]

I think for certain use cases the choice of using a local API server or not is an advantage for the community. That being said, a decentralised version (as I wondered about above), where Telegram client and bot API server bypass the official Telegram servers, might speed things further up considerably. There would be other advantages too, for example lower load of the official Telegram servers or improved data privacy by giving more control to the bot providers (there are risks too though to be fair). But I am not sure whether MTProto would support that change in network communication anyhow.

[levlam]

@itsMoji No, there is no way to provide download/upload progress for an incoming HTTP request.

[levlam]

@micheleselea @kolayne This is exactly the same server bots communicate with on api.telegram.org. There are no additional or removed layers.

There can be various performance improvements, that can be achieved with the local server. For example, api.telegram.org is located in the Europe, but some bots are bound to Asia or America datacenters. These bots would work much faster if switched to the local Bot API server located in the corresponding region. Another performance improvements is ability to upload and download local files. This way the files don't need to be additionally transferred over HTTP. Also, responses to webhook requests can be sent much faster to a local server, which increases theoretical webhook throughput. Despite that improvements most of the bots don't need to switch to a local server and can continue using api.telegram.org as a gateway.

[levlam]

@gabbhack getMe must not be used to test server performance. getMe is answered immediately by the server without invoking any network requests, so the test effectively measured ping time to the Bot API server.

[micheleselea]

@levlam what I want to understand is if the HTTP requests to our local bot api server are faster than requests sent directly to the api.telegram.org. I mean: suppose I need to delete 5000 messages (sometimes happens in my bot), because the delete API can be called for one message at a time, I have to do 5000 call to api.telegram.org. Sometimes I get a message that told me to wait a while because I did to much requests. Do you think that from this point of view can be better to have a local server bot api server? Do you think I can have less flood reply?

[levlam]

The HTTP requests to a local Bot API server are definitely faster than requests sent directly to the api.telegram.org. But if they are faster for X milliseconds, then requests from the local Bot API server to Telegram servers are longer for exactly the same X milliseconds for bots boud to Europe DC. There should be no difference, where the server is located, for ordinary requests.

[micheleselea]

@levlam I was thinking that the connection between local bot api server and telegram server, because is MTPROTO, is a connection already established, so there's no the handshake time (at least). Than I was thinking to the delay that api.telegram.bot introduce in serving requests and send it via MTPROTO internally, an this server is one for all the bots. Because if the connections/requests limits are the same and the only benefit is the delay time for bot outside Europe I think probably I can stay with api.telegram.org

[levlam]

@micheleselea You are right that MTProto has no handshake time at all during ordinary usage. But HTTPS connections after initial handshake can also be reused. If the connections aren't reused, then this gives another performance improvement for local Bot API server.

But most bots still don't need to use a local Bot API server, because api.telegram.org server is more than sufficient for them.

[micheleselea]

Ah ok so I can use HTTPS connection in keepalive with the server? I think was not possible

[levlam]

@micheleselea Keep-alive is irrelevant. It is enabled by default in HTTP/1.1. Instead, you should be able to use a connection pool to send HTTPS requests to the server, but ability to do this depends on your programming language and other software used to send the requests.

[micheleselea]

@levlam I understand, we use c++ and the http class we use has that function, I just called it "keepalive" because usually the HTTP method for reusing a connection, so you don't have to "reconnect" all the times, it's Connection; Keep-Alive header. I'm going to check if it's works, I thought that api.telegram.org did not support that kind of connections

[levlam]

Connection; Keep-Alive is deprecated since HTTP/1.1, because this is a default behavior for servers. You don't need to specify the header. Instead your client HTTP library should be able to reuse network connections.

[micheleselea]

@levlam HTTP header is Connection: Keep-Alive and I don't think is deprecated, and so I don't think that reuse connection is the default behavior, instead some servers DO NOT allow reusing connections. My client HTTP lib can reuse connections, but usually I have to set that behavior. Anyway that is going to be an off topic, I'll try to reuse connections to api.telegram.org that I wrongly thought was not allowed

[makisukurisu]

Even though I can modify some parameter I can't change how much data can be passed trough callback_data parameter? Or can I?

If not, will it be increased in near future?

Thanks for reply in advance.

[levlam]

I can't change how much data can be passed trough callback_data parameter?

You can't.

If not, will it be increased in near future?

No. It should be as small as possible to minimize client traffic and 64 bytes is more than enough to store unique ID for a data of any size.

[gabbhack]

Why should the client see and transmit the date? Why not store the original date on the telegram server and send a minimal ID to the client? When you click on the button, the server would substitute the original date.

[jcmag]

currently bots are limited to 50MB when they send files; could we bypass this limit by running a local API server?

[RememberTheAir]

@jcmag yes: https://github.com/tdlib/telegram-bot-api#usage

[puppy0cam]

@micheleselea @kolayne This is exactly the same server bots communicate with on api.telegram.org. There are no additional or removed layers.

There can be various performance improvements, that can be achieved with the local server. For example, api.telegram.org is located in the Europe, but some bots are bound to Asia or America datacenters. These bots would work much faster if switched to the local Bot API server located in the corresponding region. Another performance improvements is ability to upload and download local files. This way the files don't need to be additionally transferred over HTTP. Also, responses to webhook requests can be sent much faster to a local server, which increases theoretical webhook throughput. Despite that improvements most of the bots don't need to switch to a local server and can continue using api.telegram.org as a gateway.

Is there any way to easily determine what datacenter our bot is bound to?

[micheleselea]

ping api.telegram.com and you will see that is somewhere in Nederland. For your bot, just look at datacenter IP where the your bot is running

[micheleselea]

@levlam about my question on "delete" requests I posted yesterday: what I want to understand is if the HTTP requests to our local bot api server are faster than requests sent directly to the api.telegram.org. I mean: suppose I need to delete 5000 messages (sometimes happens in my bot), because the delete API can be called for one message at a time, I have to do 5000 call to api.telegram.org. Sometimes I get a message that told me to wait a while because I did to much requests. Do you think that from this point of view can be better to have a local server bot api server? Do you think I can have less flood reply?

[gabbhack]

@micheleselea https://github.com/tdlib/telegram-bot-api/issues/1#issuecomment-721820998 If I understand correctly, the limits on the local server are the same as those indicated in FAQ, because these limits are regulated by Telegram servers.