Problem arriving new messages on 5k+ members channels

[robertoenr]

Hi, I'm using your onixred 66.14 TelegramApi and working with TelegramBot with an UpdatesHandler.

I'm implementing a program based on Deepthrought and I'm implementing also the method onTLUpdateChannelNewMessageCustom(TLUpdateChannelNewMessage update), but it doesn't notify in real time all messages, only for some channels/groups.

Some channel/groups get notified instant, other after 15-20 min delay. and others (the most one) are never notified.

I've noticed that when start my java program, it collects all pending posts and it notifies all. But after a while when it gets uptime, it keeps silent and does notify nothing.

On my Telegram account I have ~80 public channels/groups subscribed (some <5000 members, others more than 5000, others with 30k, 40k... members).

It seems doesn't get notified with channels/groups with more than ~5k members, because I receive instant notifications for channels with less than ~5k members. From the rest I don't get notified, only re-executing the java program.

Do you have any solutions for this issue? Thanks.

[onixred]

I noticed that the program loses touch after 30 minutes of work

[onixred]

if your program will not only receive but also send messages then it will not lose contact

[onixred]

    try {
        selector.close();
    } catch (IOException e) {
        Logger.e(TcpContext.this.TAG, e);
    }

this code was suggested by one of the developers @talebipour the code was supposed to solve the problem with a lot of connections

[onixred]

• first option " if tasks empty" I think I need to add a check. If the tasks list is not empty, then do not close the thread

  public void closeEmptytasks() throws IOException {
  if (tasks.isEmpty()) {
      this.networkThread.interrupt();
      this.nioSelector.close();
      this.networkBuffer.clear();
  }
  }

• second option "delayed thread closing" Add a task that will close the thread when it is empty

[onixred]

I changed the code, I hope this solves the problem

[robertoenr]

Thank you for the help, I’ve tried it but sadly doesn't solve the initial problem. Should solve the lost connection issue.

Any solution for “no notifications” on some channels? I’ve noticed that the program is still conected, because if I send a TG message from other phone to the telegram account which is connected to the program, it notifies instant. Also tried with a public channel created by me, and, with the same telegram account as program, joined to it. It notifies the message instant on the channel. And for some channels it notifies instant. But not for the majority of them, I suspect the problem is only for channels which have 5000+ members.

Sometimes it notifies a channel, and it refresh all the pending messages, so it get also the “missing” messages, but not for real time, and not because the message itself was notified, caused by other message notification (it's a false sensation of getting all channel updates). If the message notified correctly on real time, would not be notified on this moment, the rest of missing messages would be lost.

I’ve checked the UpdatesHandlerBase, and the method processUpdate() is invoked instant when a message that is notified correctly arrives. But with the messages that arrive and aren't notified, the method processUpdate() is not invoked.

Also I've checked that, if you have running the same Telegram account on a Telegram app on smartphone, along with the java program, it gets more messages, like the mobile app was forcing to refresh the Telegram account on all programs connected (sync between sessions).

On the MTProto class, there are a TcpListener. It has a method onRawMessage(). But, on this method, I don't receive nothing on the moment where a "bad" channel receives a new message, so it does nothing. When a "good" message arrived it gets correctly a raw message. I notice that receives many "updateShort", but I think aren't channel new messages.

So, to make autonomous tests, I have 2 telegram accounts. On each account I have the same channels subscribed (about 80). With 2 smartphones (A and B) with Telegram App connected to each account I see when a channel receive a new message it’s notified on both devices on real time. All the channels are notified instant by using TG mobiles apps.

In base of previous paragraphs, the autonomous running test is to open 1 telegram account on the A smartphone. After that I close the other telegram account on the B smartphone and switch off the B smartphone (to avoid the synchronize while app is opened, before explained). After that, I execute the java program linked to the B Telegram account. While that, I keep observing the A Telegram account on the A smartphone. When run the java program, initially collects all the pending messages without no problem until it gets up to date with all messages until the now moment. After that, it seems to keep listening for updates, but the method onTLChannelNewMessage() is only invoked with some channels. Others channels are never notified (while on the A smartphone I see new messages arriving on several public channels, but the java program doesn’t receive them). After check that fail, in order to check that the program hasn’t lost connection from Telegram severs, I try to send a message from smartphone A to account B, and the message is notified instant. Also I tried to send a message from smartphone A to a public channel created by me where the B account is subscribed, and the message is notified instant: the program hasn’t lost connection.

I don’t know if the MTProto related code have a missing routine to get notified for channels with more than 1k members or similar issue.

Example channels that get notified correctly and instantly: This, This, and This

Example channels that not get notified: This, This, and This (just some examples, most channels not get notified)

[EDIT] I think it's the same problem, described here: https://github.com/vysheng/tg/issues/1067

But I don't know what to edit on the TelegramApi project to solve.

Any help is really apreciated. Thanks.

[onixred]

i think changed the code, BuffersStorage

MAX_COUNT_FREE_BUFFERS_BIG = 6;

on

MAX_COUNT_FREE_BUFFERS_BIG = 10;

[onixred]

Perhaps you need to add another buffer

freeBuffersBig = new ArrayList<ByteBufferDesc>(); // <= 280000

add

freeBuffersExtralarge = new ArrayList<ByteBufferDesc>(); // <= 320000

[onixred]

you need analyze class HTtpContext

private void readData(ByteBuffer buffer) throws Exception {

[robertoenr]

Hi, I've made the suggested changes on the BuffersStorage class and put a log on the first line of function readData(ByteBuffer buffer) on the TcpContext class. In order to monitor the new data coming.

private int MAX_COUNT_FREE_BUFFERS_OTHER = 10; private int MAX_COUNT_FREE_BUFFERS_BIG = 10; private int MAX_COUNT_FREE_BUFFERS_EXTRALARGE = 6;

private final ArrayList<ByteBufferDesc> freeBuffersBig; private final ArrayList<ByteBufferDesc> freeBuffersExtralarge;

freeBuffersBig = new ArrayList<ByteBufferDesc>(); freeBuffersExtralarge = new ArrayList<ByteBufferDesc>(); // <= 320000

} else if (size <= 280000) { arrayToGetFrom = freeBuffersBig; byteCount = 280000; } else if (size <= 320000) { arrayToGetFrom = freeBuffersExtralarge; byteCount = 320000; } else { buffer = new ByteBufferDesc(size); }

private void readData(ByteBuffer buffer) throws Exception { System.out.println("[TcpContext] readData");

As I can see, when a message from a 5k+ members channels arrives, the log doesn't prints, so no data arrives. I don't know why.

The problem is exactly reported on this issue (other project): https://github.com/vysheng/tg/issues/1067 In order to correct, they changed a file called "lua-tg.c", but I can't find any similar in our TelegramApi.

[onixred]

see class TLPeerChannel TLPeerUser TLUpdateChannelTooLong

[robertoenr]

The strange thing is that on the readData(ByteBuffer buffer) method, on TcpContext class, doesn't come anything when a supergroup pubs a new message. I don't know if changing the layer will solve this problem

[robertoenr]

I think it's more a problem with the receiver socket connection, at low level api. But don't know what...

[robertoenr]

see class TLPeerChannel TLPeerUser TLUpdateChannelTooLong

I've see it, but doesn't tell me anything. I've done a global search for these classes, but can't find anything relevant that reference the issue

[robertoenr]

Solved by replacing UpdatesHandlerBase with this from the @astuteficus fork. Thank you!

https://github.com/astuteficus/TelegramApi/blob/master/src/main/java/org/telegram/bot/handlers/UpdatesHandlerBase.java

[onixred]

I think you need to merge this branch

[robertoenr]

After the yesterday Telegram fall and reboot the program, the issue is back again.

I suspect it’s a problem of differences cache or chat/user cache, because yesterday I didn’t clear these tables, and before yesterday (when the program was working ok) with the replacement of UpdatesHandlerBase I also cleared these 3 tables, and was working ok.

[robertoenr]

I have stopped the program, empty the differences, users and chats tables on MyQSL, and re-run the program. And now starting to receive all messages. My project is based on Deepthrough, I don’t know what’s wrong...

[robertoenr]

It’s crazy, now the program is notifying my bot only when I open the TelegramApp, when I open it, throws many messages. When I close the app I don’t receive anything

[astuteficus]

Be careful using my code, it is really optimized for my bot's workflow and I can't give any guarantee about it's compatibility with original idea of this library

[robertoenr]

Thank you @astuteficus. I'll keep in mind it

[robertoenr]

Yesterday, I re-start my project with the clean 66.14 library from @onixred. Before run the program I clean the chats, users and differences tables from DB. When run it, it started notifying all messages from all channels, but at night it only has sent messages of 2 channels. I suspect that is something wrong on differences logic or similar. My project is exactly like this: https://github.com/rubenlagus/Deepthought/blob/master/Deepthought/src/main/java/org/telegram/Deepthought.java

And the internal logic is exactly this: https://github.com/rubenlagus/Deepthought/tree/master/Deepthought/src/main/java/org/telegram/plugins/echo

On the CustomUpdatesHandler I also have implemented the onTLChannelNewMessage() method to receive all channel messages.

On MessageHandler I commented the line on bottom function where it echo the message, keeping the line which mark it as read.

The DifferencesHandler is internal on library, so I don’t know whats wrong...

Any help is appreciated.

[robertoenr]

I can see when I try to auth, this error:

org.telegram.api.engine.RpcException: PHONE_MIGRATE_4

just after the authSendCode, and I can see it on my 2 accounts. I don't know if matters, but I've see that if I keep closed all the Telegram Apps linked to any of my account, my java application doesn't retrieve new messages and I can't see the log "getting differences". Only get instant messages from an unique channel with 978 members (@baratukin_es). If I open one of my Telegram account on any Telegram app, the program start getting differences. If I close again, again keep silent until I open any app.

[Edit] I've noticed that the "not receiving messages problem" it only happens on one of my Telegram accounts. And it seem I only receive MTPong data. This a custom logs on the MTProto class, with the account which doesn't receive messages:

[MTProto] onRawMessage()
[MTProto] decrypt()
MessageArrived (#2): time: 1530984582
MessageArrived (#2): seqNo: 28, msgId:6575528711837272065
[MTProto] messageClass == else
[MTProto] onMTMessage()
[MTProto] onMTMessage() -> before deserializeMessage
[MTProto] onMTMessage() -> before onMTProtoMessage
[MTProto] onMTProtoMessage()
[MTProto] onMTProtoMessage() -> MTPong
[MTProto] onMTMessage() -> after onMTProtoMessage

If I keep running the program with the "cursed" account I only receive messages instant from some channels with <1000 members. From the rest of channels I receive nothing.

Other strange behaviour is when I open the Telegram App on iPhone (which has linked other different Telegram account), the "cursed" account and program get refreshed and receive all pending messages. I keep open the app on iPhone and again doesn't receive anything. After, I close the Telegram app on iPhone and receive again the pending messages. I don't know why because they are different account with different tokens and api keys... Note: I've asked the official @SpamBot to check if the "cursed" account is spam locked, but it says that the account is free and working perfect. To reproduce the "not receiving messages" problem, you must switch off all the phones and close all the Telegram applications, to avoid they interference on the standalone working of api.

Any tips to workaround this? Thanks.

[onixred]

I'm back and ready to solve the problem

[gabrielbandeira80]

I have the same problem.

[Obi-wan-koban]

Has this been fixed in the newest version yet? Can this not be related to a setting from the telegram app itself? (maybe if you disable notifications, no messages will be pushed until a logon appears?