search

LonamiWebs / Telethon

Pure Python 3 MTProto API Telegram client library, for bots too!
https://docs.telethon.dev
MIT License
9.46k stars 1.36k forks source link

events.NewMessage event is not triggered for certain channels #4345

Open M4jx opened 3 months ago

M4jx commented 3 months ago

Code that causes the issue

@client.on(events.NewMessage(chats = list(channels_hashmap.keys())))
async def main(event):
     print(now.strftime("%d/%m/%Y %H:%M:%S") + f" Received from channel: {event.chat_id}")

Expected behavior

When a new message is sent in a channel passed to chats, the events.NewMessage should be triggered.

Actual behavior

When a new message is sent, the events.NewMessage is not being triggered for certain channels.

I have tested to create a channel in an account A and monitor it in account B (where telethon is running) and the events.NewMessage was triggered instantly. However, for certain public channels that I do not own, the event is not being triggered.

Tested on a newly created channel with 2 subscribers and it worked. Tested on a public channel with 500K subscribers and it did not work.

Traceback

No crashes are happening.

Telethon version

1.34.0

Python version

3.10.12

Operating system (including distribution name and version)

Ubuntu 22.04.4 LTS

Other details

No response

Checklist

abhinavbajpai2012 commented 3 months ago

Same with me, doesn't work for some public channels

Lonami commented 3 months ago

Please react with the thumbs-up emoji instead of posting "me too" comments.

M4jx commented 3 months ago

Update: While Telethon is running, if a message is sent in the monitored channel and the events.NewMessage event is not triggered. It can be triggered manually if you open the channel and read the message in the Telegram app (from the same user account telethon is running on). Once the channel is opened, and the message is read, telethon triggers the event instantly.

khoben commented 3 months ago

Update: While Telethon is running, if a message is sent in the monitored channel and the events.NewMessage event is not triggered. It can be triggered manually if you open the channel and read the message in the Telegram app (from the same user account telethon is running on). Once the channel is opened, and the message is read, telethon triggers the event instantly.

Telegram doesn't always send all updates to the client. Some channels can only be updated after manually calling for updates. You can try restarting your app, creating a new session, changing the IP or the account itself.

iSte94 commented 3 months ago

Same problem for me, and it's not the library. Same problem with Pyrogram

bosslaiv commented 3 months ago

I noticed the same ,,, few months back i was runing multiple clients via mobile IP , and received 100% all updates ,,, few months later , after a pause , i started a single client over VPN , and not all updates where being registered .

Lonami commented 3 months ago

Some people have reported that some channels must be polled manually to receive updates as of recently.

Telethon does not have any built-in way to do this yet (but of course can be done via raw API; I just haven't checked so I don't know exactly how).

I don't know if this will ever be "fixed" to work automatically, because if the need for manual polling is true, Telethon would have no way to know what it needs to poll manually.

Tidalikk commented 3 months ago

I noticed the same ,,, few months back i was runing multiple clients via mobile IP , and received 100% all updates ,,, few months later , after a pause , i started a single client over VPN , and not all updates where being registered .

Since when did you start noticing this issue?

francestu96 commented 3 months ago

Update: While Telethon is running, if a message is sent in the monitored channel and the events.NewMessage event is not triggered. It can be triggered manually if you open the channel and read the message in the Telegram app (from the same user account telethon is running on). Once the channel is opened, and the message is read, telethon triggers the event instantly.

Actually, in my case, even the read of the message from the Telegram app does not trigger it. Is it only a problem of mine?

sushilronghe99 commented 3 months ago

4361

In my case the whole app just blackouts for randomly 1-2 minutes and then it receives the new messages updates properly. I have 10 channels subscribed and for 2 minutes app did not receive any update from any channels between 9:27-9:29 AM and 10:02-10:04 AM. Afterward, for the rest of the day, all updates were received properly.

So as suggested here NewMessage event is not reliable and we should poll those channels manually. I will try that and let you know how it goes.

arvindavoudi commented 2 months ago

I've encountered a peculiar situation with a message delivery order, and I'd like to explain it in hopes of shedding light on the issue. Assume we have three messages (A, B, and C) sent in the following order with some minutes delay in between:

In my case, the following occurred:

To better understand and debug this issue, I have the following questions:

Checking the behavior of other packages written in other languages can also help us to confirm where the issue is from and why.

Lonami commented 2 months ago

Out-of-order messages shouldn't occur because the library checks that the incoming pts come in order:

https://github.com/LonamiWebs/Telethon/blob/47673680f4f615aac15d7e8c1886abea2fe40d9e/telethon/_updates/messagebox.py#L561

Make sure the message isn't being deleted and getting resent.

I have observed official clients displaying messages that had been deleted multiple times, and not too long ago.

If you really think Telethon is processing updates out of order, I either need someone with that problem to debug it themselves, or a way in which I can reliably reproduce it myself. Although I've already spent a significant amount of time going over the update handling code myself, so I doubt I would uncover anything.

As far as I know https://core.telegram.org/api/updates doesn't say anything Telethon isn't doing already (at the time of writing).

As for missing messages, or the library fetching them after a while. Yes, there is a default timeout after which the library will attempt to fetch updates:

https://github.com/LonamiWebs/Telethon/blob/47673680f4f615aac15d7e8c1886abea2fe40d9e/telethon/_updates/messagebox.py#L46

This is in line with Telegram's documentation above:

Manually obtaining updates is also required in the following situations:

  • Long period without updates: no updates for 15 minutes or longer.
arvindavoudi commented 2 months ago

Dear @Lonami, I'm sure that I received message (C) before (B) but I'm not sure if it was deleted and sent again or not. If it was deleted, I expected to receive that exact same message again after a while, right? but as you know, sometimes messages do not get delayed, they simply get ignored and never get transmitted. Even after opening that channel on my phone, message (B) was not received "twice", just only once and I also didn't receive message (C) again (meaning that message (C) was not deleted and got resent). As far as I consider these messy updates complicated and not quite trackable, I cannot really guarantee that this unordered updates issue is valid but It was my observation that I decided to share with you because my whole project is now unusable after a few months of not being on the server. It's not working and I'm very concerned. :) With the all experience you have, can you consider submitting a bug to https://bugs.telegram.org/ as Telethon owner? I think it would help if you're sure there is no issue with Telethon itself since Pyrogram also does not work as expected and a working solution is needed widely in the industry.

P.S.: If you haven't any channel suffering from this updating issue (not the ordering one), I know one that I can send only privately to you.

Lonami commented 2 months ago

I do not have interest in submitting bug reports to Telegram. You can certainly message me privately (or include the channel in a GitHub comment between <!-- HTML comments --> so that only maintainers can read it), and I may give it a go… eventually.

Perhaps trying this out would be a good excuse to make sure update handling works correctly in the (not ready for use) v2 version.

Alex11381 commented 1 month ago

Has anyone found a solution to this problem?

alan-insam commented 1 month ago

Is there any workaround yet?

sushilronghe99 commented 1 month ago

Hi There,

There is no solution to make the update function work the way we want. I also tried another library but no luck. It is already established its an issue with Telegram API and not with libraries.

Here is the workaround that might help your case. (It requires heavy usage of Asyncio)

Run a task every 30 sec (or less) to pull the messages from Telegram using iter_messages. Use offset_id instead of offset_date to fetch messages. The function should effectively use the message ID to track the current state of every channel you listen to.

Few important points

  1. The logic will get complicated if you have multiple clients running on the same server use dictionaries or persistent storage to keep track of each channel state.
  2. You can also use the New Message Trigger - In my experience this has a success rate of 20%. If we are keeping both then we need to make sure the message ID is used effectively to identify duplicate messages.
  3. My Bot is running straight for 10hrs daily with an interval of 30 sec - so far there are no complaints from telegram or telethon. If you are going to run this for a lesser interval and longer time please run proper tests before going to production.

This works for me like a charm - not even a single message missed in the last 30 days for almost 5 clients running on one server. Not sure if this is the right solution but at least for my use case it's working.

Hope this helps.

Thanks

galaSYT1 commented 1 month ago

I noticed this issue over 6 months ago. If Telegram hasn't fixed anything until now I don't see them fixing it at all. Afaik there is no easy fix for it.

As already mentioned, if the App is open NewMessage works correctly.

fabianski7 commented 4 weeks ago

for me this problem only happens on channels and only on those with a lot of subscribers. everything else works normally

this channel with over 100,000 subscribers is working https://t.me/apostasepalpites this one with over 100,000 too doesn't work https://t.me/promocoesecuponsglobais

sabitm commented 3 weeks ago

I wonder if anybody has done reading the source code of official Android/iOS telegram app and find out what (hack) they use for new message event instead? It seems that the official mobile app didn't suffer from this problem at all

khoben commented 3 weeks ago

I wonder if anybody has done reading the source code of official Android/iOS telegram app and find out what (hack) they use for new message event instead? It seems that the official mobile app didn't suffer from this problem at all

Actually, with official clients exactly the same problem, some channels are updated only when you open them.

sabitm commented 3 weeks ago

Actually, with official clients exactly the same problem, some channels are updated only when you open them.

Oh yeah, totally missed that. I opened my Telegram when watching them. Now, what do they use instead? Manual fetch?

khoben commented 3 weeks ago

Actually, with official clients exactly the same problem, some channels are updated only when you open them.

Oh yeah, totally missed that. I opened my Telegram when watching them. Now, what do they use instead? Manual fetch?

I assume this is done for resource optimization purposes. Some channels get real time updates, some only manually. Sometimes with new sessions or a different client IP, the channel starts to get updates.

Official clients also listen for updates just like telethon, but it's not as important for the user to get necessarily all the updates from 500+ channels. Further, when the channel is opened, the updates will be pulled up.