Not receveing messages from all chats

[LeonardoRick]

def tlgMsgHandler(update):
    # only text messages
    message_content = update['message']['content'].get('text', {})
    msg = message_content.get('text', '')
    chat_id = str(update['message']['chat_id'])
    print(f'chat Id: : {chat_id}\nmessage: {msg}')

constants = tlg_constants.TelegramConstants()

tlg: Telegram = Telegram(
    api_id=constants.api_id,
    api_hash=constants.api_hash,
    phone=constants.phone,
    database_encryption_key=constants.api_hash
)

tlg.login()  # connect to Telegram
tlg.add_message_handler(tlgMsgHandler)

tlg.idle()

Am I doing something wrong? I receive messages just from some channels but others don't.

I even tried to add

result = tlg.get_chats()
result.wait()

before tlg.add_message_handler(tlgMsgHandler), but I don't know if it's the case to use that

[alexander-akhmetov]

I even tried to add

result = tlg.get_chats() result.wait()

When you get all chats, you have to pass offset_order, see #58 and https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1get_chats.html .

If it still happens after you loaded all chats, you can try to enable debug logs (tdlib_verbosity to 5 or higher) to see what's happening. tdlib should print log messages every time it gets a new update from Telegram.

[LeonardoRick]

I tried result = tlg.get_chats(offset_order=2 ** 63 - 1, offset_chat_id=0) and retult = lg.get_chats(offset_order=9223372036854775807) followed by result.wait(), but it still not receving all messages. I tried @melkor217 sollution too and no success too.

I tried debug it with tdlib_verbosity to 5 and when the message is not recognized, log doesn't show neither updateChatLastMessage or updateNewMessage objects. So when I'm receiveing a message from a group that my code is not recognizing, I'm not receveing those objects. When I receive them, it works properly. Some other info are logged when this problematic chats send messages.

I'll put the log here in case It helps to find something. This is the log I receive when I DON'T receive those first updateChatLastMessage or updateNewMessage

 3][t 4][1600130975.339170694][FileManager.cpp:1128][#1][!Td]  Register file data [[remote_name:107917777_304636.jpg] chat 0 [size:0][expected_size:0] NoKey{} remote [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]]{}] as 436(0) from register_remote
[ 3][t 4][1600130975.339195251][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Binlog
[ 4][t 4][1600130975.339210749][FileManager.cpp:1364][#1][!Td]  Merge new file 436(0) and old file 256(0)
[ 3][t 4][1600130975.339221954][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Database
[ 4][t 4][1600130975.339235544][FileManager.cpp:1456][#1][!Td]  Have x_node->pmc_id_ = 0, y_node->pmc_id_ = 52, x_node_size = 1, y_node_size = 1, node_i = 1, local_i = 2, remote_i = 2, generate_i = 2, size_i = 1, remote_name_i = 1, url_i = 1, owner_i = 2, encryption_key_i = 2, main_file_id_i = 1, trusted_by_source = 1, x_source = Binlog, y_source = Database
[ 3][t 4][1600130975.339250088][FileManager.cpp:2514][#1][!Td]  Skip run_generate, because file 256(0) can be downloaded from server
[ 3][t 4][1600130975.339261532][FileManager.cpp:2138][#1][!Td]  Cancel downloading of file 256(0)
[ 3][t 4][1600130975.339272499][FileManager.cpp:2625][#1][!Td]  File 256(0) upload priority is still 0
[ 4][t 4][1600130975.339283705][FileManager.cpp:1011][#1][!Td]  Forget file 436(0)
[ 3][t 4][1600130975.339294910][FileManager.cpp:1128][#1][!Td]  Register file data [[remote_name:107917777_304638.jpg] chat 0 [size:0][expected_size:0] NoKey{} remote [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]]{}] as 436(0) from register_remote
[ 3][t 4][1600130975.339310646][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Binlog
[ 4][t 4][1600130975.339323759][FileManager.cpp:1364][#1][!Td]  Merge new file 436(0) and old file 257(0)
[ 3][t 4][1600130975.339332819][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Database
[ 4][t 4][1600130975.339345455][FileManager.cpp:1456][#1][!Td]  Have x_node->pmc_id_ = 0, y_node->pmc_id_ = 53, x_node_size = 1, y_node_size = 1, node_i = 1, local_i = 2, remote_i = 2, generate_i = 2, size_i = 1, remote_name_i = 1, url_i = 1, owner_i = 2, encryption_key_i = 2, main_file_id_i = 1, trusted_by_source = 1, x_source = Binlog, y_source = Database
[ 3][t 4][1600130975.339358091][FileManager.cpp:2514][#1][!Td]  Skip run_generate, because file 257(0) can be downloaded from server
[ 3][t 4][1600130975.339368582][FileManager.cpp:2138][#1][!Td]  Cancel downloading of file 257(0)
[ 3][t 4][1600130975.339378357][FileManager.cpp:2625][#1][!Td]  File 257(0) upload priority is still 0
[ 4][t 4][1600130975.339387894][FileManager.cpp:1011][#1][!Td]  Forget file 436(0)
[ 3][t 4][1600130975.339413404][FileManager.cpp:1128][#1][!Td]  Register file data [[remote_name:107917777_304636.jpg] chat 0 [size:0][expected_size:0] NoKey{} remote [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]]{}] as 436(0) from register_remote
[ 3][t 4][1600130975.339430094][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Binlog
[ 4][t 4][1600130975.339442968][FileManager.cpp:1364][#1][!Td]  Merge new file 436(0) and old file 256(0)
[ 3][t 4][1600130975.339451551][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304636]] from Database
[ 4][t 4][1600130975.339463472][FileManager.cpp:1456][#1][!Td]  Have x_node->pmc_id_ = 0, y_node->pmc_id_ = 52, x_node_size = 1, y_node_size = 1, node_i = 1, local_i = 2, remote_i = 2, generate_i = 2, size_i = 1, remote_name_i = 1, url_i = 1, owner_i = 2, encryption_key_i = 2, main_file_id_i = 1, trusted_by_source = 1, x_source = Binlog, y_source = Database
[ 3][t 4][1600130975.339476109][FileManager.cpp:2514][#1][!Td]  Skip run_generate, because file 256(0) can be downloaded from server
[ 3][t 4][1600130975.339487076][FileManager.cpp:2138][#1][!Td]  Cancel downloading of file 256(0)
[ 3][t 4][1600130975.339496851][FileManager.cpp:2625][#1][!Td]  File 256(0) upload priority is still 0
[ 4][t 4][1600130975.339507341][FileManager.cpp:1011][#1][!Td]  Forget file 436(0)
[ 3][t 4][1600130975.339517117][FileManager.cpp:1128][#1][!Td]  Register file data [[remote_name:107917777_304638.jpg] chat 0 [size:0][expected_size:0] NoKey{} remote [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]]{}] as 436(0) from register_remote
[ 3][t 4][1600130975.339531660][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Binlog
[ 4][t 4][1600130975.339544773][FileManager.cpp:1364][#1][!Td]  Merge new file 436(0) and old file 257(0)
[ 3][t 4][1600130975.339553356][FileManager.cpp:1223][#1][!Td]  Choose between [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Binlog and [profile_photos, DcId{1}, location = [id = 2375007367708387251, access_hash = 0, volume_id = 107917777, local_id = 304638]] from Database
[ 4][t 4][1600130975.339565754][FileManager.cpp:1456][#1][!Td]  Have x_node->pmc_id_ = 0, y_node->pmc_id_ = 53, x_node_size = 1, y_node_size = 1, node_i = 1, local_i = 2, remote_i = 2, generate_i = 2, size_i = 1, remote_name_i = 1, url_i = 1, owner_i = 2, encryption_key_i = 2, main_file_id_i = 1, trusted_by_source = 1, x_source = Binlog, y_source = Database
[ 3][t 4][1600130975.339578629][FileManager.cpp:2514][#1][!Td]  Skip run_generate, because file 257(0) can be downloaded from server
[ 3][t 4][1600130975.339588881][FileManager.cpp:2138][#1][!Td]  Cancel downloading of file 257(0)
[ 3][t 4][1600130975.339598656][FileManager.cpp:2625][#1][!Td]  File 257(0) upload priority is still 0
[ 4][t 4][1600130975.339608669][FileManager.cpp:1011][#1][!Td]  Forget file 436(0)

It doesn't look like an error. I thought it was just some other log stuff, but what is weird is that I do not receive this info when the message works properly.

Its not something that happens just sometimes, this kind of different log is being logged everytime I receive a message on Telegram that is not well handled by my program.

Something valid to metion is that I have around 35 channels. Should that be a problem? I just want to receive new messages and I'm not storing it anywhere so maybe it shouldn't be.

[melkor217]

I tried result = tlg.get_chats(offset_order=2 ** 63 - 1, offset_chat_id=0) and retult = lg.get_chats(offset_order=9223372036854775807) followed by result.wait(), but it still not receving all messages. I tried @melkor217 sollution too and no success too.

I'm not sure if you really have to get chats to make make your stuff work.

If so, check if get_chats returns chats you need.

If number of chats is somewhat round like 100, you probably want to load more.

Something valid to metion is that I have around 35 channels

Not really, but keep in mind that "chat" entity could be a direct message, bot chat, group, channel etc.

[LeonardoRick]

@melkor217, Thanks a lot for answering. I think some things seems obvious to you but maybe I'm missing some gaps of understanding here.

I'm not sure if you really have to get chats to make make your stuff work.

If I don't need to get, how am I supposed to make it works normaly? Because some chats shows messages normaly. Others don't.

If so, check if get_chats returns chats you need

I used:

result = tlg.get_chats(offset_order=2 ** 63 - 1, offset_chat_id=0)
print(result.wait())

Is that it? It's printing None, so probably I'm doing something wrong. Should I do something with this return? I'm just waiting it with .wait().

If number of chats is somewhat round like 100, you probably want to load more.

The return of get_chats should show that? If not, where can I find this info.

Again, thanks for answering and clarifying a lot of things.

[alexander-akhmetov]

Is that it? It's printing None, so probably I'm doing something wrong. Should I do something with this return? I'm just waiting it with .wait().

I think there is a typo in your code, .wait() returns None. You need to call wait() to wait until Telegram returns the response, and then print the result:

result = tlg.get_chats(offset_order=2 ** 63 - 1, offset_chat_id=0)
result.wait()
print(result.update)

If number of chats is somewhat round like 100, you probably want to load more. The return of get_chats should show that? If not, where can I find this info.

get_chats returns you a response in this format:

{"@extra": {"request_id": "..."},
 "@type": "chats",
 "chat_ids": [1, 2, 3, 4]}

You can specify limit when you call get_chats, but in tdlib's documentation they say For optimal performance the number of returned chats is chosen by the library. This issue and this description of how to get list of chats also might be useful.

When you need to load all chats, you have to call get_chat until there is no more chats to load (response.update['chat_ids'] is empty). get_chat accepts offset_order and offset_chat_id, both are necessary to get the full list. Each chat has its own order and id. You start with offset_order=2 ** 63 - 1 and offset_chat_id=0, and after you receive the first result, you get the latest chat_id, call get_chat and use this information to calculate next (offset_order, offset_chat_id) pair.

Something like that should get all chat ids:

    ...
    tg.login()

    offset_chat_id = 0
    chat_ids = []
    offset_order = 2 ** 63 - 1
    chats_received = True

    while chats_received:
        result = tg.get_chats(offset_order=offset_order, offset_chat_id=offset_chat_id)
        result.wait()

        if result.update['chat_ids']:
            chat_ids += result.update['chat_ids']
            chat_info = tg.get_chat(chat_ids[-1])
            chat_info.wait()
            offset_chat_id = chat_info.update['id']
            offset_order = chat_info.update['order']
        else:  # no more chats to load
            chats_received = False

    print('Chats: ', chat_ids)

[alexander-akhmetov]

But I'm not sure that you have to retrieve all chats to receive messages, to be honest. Is there anything specific about those chats, is it always the same chats or they are different from time to time?

[LeonardoRick]

Nice. I tried this implementation and it returns a list of 222 items, so I guess the list is working. How am I supposed to use this list on my message_handler()? For now, it's just a list of IDs, so I'm not acctualy using it, right? Just getting them didn't solved the problem

But I'm not sure that you have to retrieve all chats to receive messages, to be honest. Is there anything specific about those chats, is it always the same chats or they are different from time to time?

No it's just regular chats, I can't think of something special about them. Should I concern about muted chats? Because some of them are muted but I couldn't find a pattern with that, so it's just an observation

[alexander-akhmetov]

Nice. I tried this implementation (you should just change while True to while chats_received for the future) and it returns a list of 222 items, so I guess the list is working. How am I supposed to use this list on my message_handler()? For now, it's just a list of IDs, so I'm not acctualy using it, right? Just getting them didn't solved the problem

No, I think not having chats locally should not affect receiving new messages. And it didn't work for you, so it doesn't.

No it's just regular chats, I can't think of something special about them. Should I concern about muted chats? Because some of them are muted but I couldn't find a pattern with that, so it's just an observation

Then I think it's better to ask in tdlib repository, sorry. When you pass a function to Telegram.add_message_handler, python-telegram calls your handler every time there is an update with updateNewMessage type. But you don't see these updates even in tdlib logs, maybe they need to be processed in some specific way, something is misconfigured or they are special somehow.

[LeonardoRick]

What's weird is that it's not just some cases. The majority number of chats are not sending me this updateNewMessage. What is even stranger is that it's not consistent. Some chat's not showing updates used to show sometimes. I now that because I writed down it's chat_id's and now they are not sending updates over add_message_handler. Is there a way that it can be blocked by the chat owner? Or maybe some privace setting I'm settled...

Anyway, thank's for helping and I will keep trying to find a way to fix this. I'm surprised that I can't find anyone that had this issue before. I'll keep looking for it too

[LeonardoRick]

Just for the record, I tried a basic pyrogram quick start and it's working ok for me. That's sad because I was really confotable using this library and probably i'll need to make a couple changes om my code. But If any ideia comes to my mind on how to fix it, I'll certainly let you guys know about that.