More docs required - handling incoming updates

[leonerd]

I'm currently attempting to make use of incoming "updates" messages from telegram, and finding no examples for this.

Specifically, nothing in the toplevel readme, nor any of the files under the examples/ dir, show anything about, for example, how to react to an incoming UpdateShortChatMessage which contains a newly-received message in a chat.

By sourcediving, I do see that the toplevel MTProto appears to have a pair of on+emit methods, assigned by the constructor in lib/service/main/index.js itself but this exercise in reading the source hasn't so far lead me to discover what event names might be emitted, that I could register an interest in, nor what the shape of arguments they'd provide.

For reference, this is part of a code rewrite on matrix-appservice-tg to swap from using telegram.link to telegram-mtproto; the part of code currently in question starts with the call to client.registerOnUpdates at

https://github.com/matrix-org/matrix-appservice-tg/blob/master/lib/TelegramGhost.js#L81

[goodmind]

20

[leonerd]

Ohsure; I'm happy that telegram-mtproto doesn't do a lot to help me with it; but there's literally no starting point whatsoever in the docs here. I presume (perhaps falsely?) that there is some name I can on wait, and some code I could write like the following:

client.on("BANANA", (event) => {
  console.log("I just received an event", event.BANANA);
});

It's just hard to know where to begin right now. If you could suggest suitable actual things to write in place of my BANANA placeholders above, I can make some progress

[goodmind]

There is no working update manager now

/cc @zerobias

[leonerd]

I see. That's upsetting. So what you're saying is that currently, this entire library is unable to handle incoming messages from Telegram in any way?

Is there anything I could do to assist in some way? Perhaps writing the minimal viable implementation that will suffice for my purposes, and submit it to you?

[wfjsw]

FYI A code stub is available here: https://github.com/zerobias/telegram-mtproto/blob/develop/src/plugins/updates.js

[leonerd]

That's some exciting looking code, but I'm not seeing how I make use of it. The key line appears to be the call to setUpdatesProcessor, but I can't make sense on reading this, of how I apply it to a client instance I already have.

[zerobias]

@leonerd Yes, some modules are actually too complicated and confusing, so I did not implement an update manager yet, so as not to increase the bad code coupling

By the way, work with updates is described here

[leonerd]

@leonerd Yes, some modules are actually too complicated and confusing, so I did not implement an update manager yet, so as not to increase the bad code coupling

Ohsure; I don't mind that. I just don't see a single starting point anywhere in any of the code. See again the BANANAs in my comment above; https://github.com/zerobias/telegram-mtproto/issues/40#issuecomment-294008076. If you can suggest me something, anything that I can write, that somehow attaches code onto the client instance that gets invoked on receipt of an incoming update message, I'm sure I can figure the rest out myself.

By the way, work with updates is described here

Sure. I know how the Telegram API works in terms of bytes over the wire. The point of my docs request here, is how I may use your telegram-mtproto to send and receive those bytes.

[wfjsw]

In fact, I can see updates coming in the networker debug log, but I can't catch and process them.

[wfjsw]

@leonerd something like this:

client('updates.getState', {})
client('updates.getDifference', {
    pts: 0,
    date: 0,
    qts: -1
})

[wfjsw]

But if we don't have a hook on networker, wr probably have to manually pull updates every time.

[zerobias]

@wfjsw Usage example https://github.com/goodmind/treact/blob/development/src/app/containers/InstantMessages/index.tsx

Available in version 2.2.1 and disabled by default later

[zerobias]

@wfjsw I'll add hooks in the next release

[leonerd]

I'm still struggling to make headway here.

I now have a debugging on anything callback to just print stuff.

    client.on("*", (ev) => {
        console.log("TODO: Received telegram update", ev);
    });

So far, nothing. I suspect this won't work until I attach the "UpdatesManager" that I can see in code.

For example, I see there's a file lib/service/updates.js that contains a bunch of code, including the attach function, which sets something about an updates processor, and starts the code that I feel is the right thing to be doing. This code really does appear to exist.

https://github.com/zerobias/telegram-mtproto/blob/master/src/service/updates.js#L434

Maybe it's just my unfamiliarity with javascript to be able to read exactly what's going on in this file, but I've yet to work out how to make use of this file. It appears to export a thing called UpdatesManager. Is this a class? It doesn't look like it - I can't see what I recognise to be a constructor function. Maybe in some other way I can create one and invoke this attach function/method/whatever-it-is against my API instance, to make it work.

One thing reading this code still troubles me. As far as I can tell by reading it, the call to networker.setUpdatesProcessor appears to be affecting process-global state, rather than anything that might apply to one particular instance of the API client. Am I given to understand this is the case? The program I am writing could potentially handle hundreds of telegram connections from one process - is this going to work properly between them all?

[wfjsw]

@leonerd The event 'seq' probably got removed after refactor. Wait till next release as it will contain some events.

[leonerd]

The event 'seq' probably got removed after refactor. Wait till next release as it will contain some events.

I have no idea what this is in reference to. Are you able to assist me with how I can invoke this attach code?

[wfjsw]

@leonerd The UpdateManager mainly listens on the event seq to get updates from the lib. But this piece of code is relatively old, so things may not work as expected. I think we should wait ATM. Meanwhile you can still invoke updates.getState and updates.getDifference manually (this part of docs in Telegram Core is not too obsolete yet, see also https://core.telegram.org/api/updates#obtaining-differences )

[leonerd]

Yes I know how to invoke updates.getState and getDifference, but those are polling mechanisms. They aren't the eager event-pushing that comes directly from Telegram via update messages, as used to happen when I was using the telegram.link library.

I'm still having trouble understanding this - are we saying that, as of right now, telegram-mtproto literally does not have any way to handle the incoming updates messages? If not, how on earth is anyone building a client based on it? I feel there's something I'm still confused about here.

[wfjsw]

@leonerd

@zerobias had said

@wfjsw I'll add hooks in the next release

[zerobias]

Yes, I'm switching to event-driven architecture, needs some time to implement but it will allow subscribing to literally everything. It's more obvious way than current one

[wfjsw]

And this lib is not completed yet

[leonerd]

Yes, I'm switching to event-driven architecture, needs some time to implement but it will allow subscribing to literally everything. It's more obvious way than current one

I see.

Any way I can assist with that effort then? This seems to be an important feature that I'm in need of, so if I can help bring it to life...

[leonerd]

Is there any hacky workaround I can apply right now, to extract the update messages out of the telegram wire? Doesn't matter how neat or horrible it is, just something that will do for now as a temporary hack, until a better solution exists?

The reason I ask is that I already have code working using telegram.link, which includes all its own processing of incoming update messages. But telegram.link is terrible and buggy, slow to start up and other problems, and updates-aside I'm finding your telegram-mtproto a lot better.

I don't need anything like a full processor - I already have that. All I need is the ~one-line redirection bit of logic to hook in, so that when an update-style message is received over the wire, it gets piped into my update processing code I already have.

Any kind of workaround could be considered - some sort of gut-wrenching into the innards of my api client object, or actually patching the source code to have it emit an event or anything. Some small start to get my code off the ground, while I await a better solution in the form of a full Updates Manager.

[wfjsw]

Take a look at DEBUG=* log, find the right logger(should be networker) and place a emit there.

[zerobias]

I will try to publish relevant changes on a weekend

[leonerd]

Ahah. So after a day and a half of code diving I have come upon this line:

https://github.com/zerobias/telegram-mtproto/blob/master/src/service/networker/index.js#L949

Helpfully, already a pre-commented console.log line that I just have to uncomment, and sure enough it prints to the console all the updates messages I wanted to see. :)

So now all I have to do is wire that up to the rest of my client code. One slight complication appears to be that I can't just use the updatesProcessor global variable, because there's only one of it. Whereas in my program I have tens, possibly hundreds, of client instances.

However, it appears to work if I simply insert

this.emit('update', message);

So now at least I can continue to make progress with my application by reusing my old code that used to work against telegram.link, while I await a better "updates processor" from telegram-mtproto itself.

[mjkaufer]

@zerobias can you add @leonerd 's this.emit('update', message) to the actual release - I'm sure many devs would find it useful

[incubus]

Hi!

Does anyone know how to use setUpdatesProcessor in 3.0.8 version?

[alimhv]

Hi How to use this method ?? I get this error [1.327] Error undefined undefined 2 4 @leonerd

[webworker01]

So I found where to add the emit call, but was wondering if there's any reason this has not yet been added to the actual codebase? Too hacky?

[JosefJezek]

@zerobias any solution?

[AndreasGassmann]

Is there another way to do this in the current release? (other than editing the source code and adding this.emit('update', message);)

By the way, thanks @leonerd for all the effort you put into this. Are you still using this library and are you still using your "workaround" or did you find something better?

EDIT: @webworker01 Where do you have to add that line now? I couldn't find it in the newest v3 release

[webworker01]

@AndreasGassmann Right around this line does the trick

[tulir]

@AndreasGassmann v3 does not need the emit update patch. You can just use client.bus.untypedMessage.observe(func)

[AndreasGassmann]

@tulir perfect, that's what I was looking for. Thanks!

[homjke]

@tulir, will this work to get new messages in real-time (from chats and groups) ? If answer "Yes", could you please give me some code example, how it works for you?

[tulir]

Yes, I use it for all incoming updates.

The callback function gets the event data in one argument. The actual update data is in data.message. The update data is one of these types.

client.bus.untypedMessage.observe(data => {
    console.log(data.message)
}

Also, I'm pretty sure that updates.getState needs to be called continuously for it to actually work. I haven't tested it thoroughly, but when I increased the time in the following interval setup, it took longer for messages to come.

setInterval(() => client("updates.getState", {}), 1000)

[leonerd]

This is all a useful chat about how to perform the activity, but it exactly demonstrates the original point in my original bug report. This detail isn't documented anywhere in the docs for the library, so people who want to know currently have to read this issue thread. It needs to be written down somewhere that people can find it.

[Fuzzyma]

Coming across this thread with my issue. Do you know how I can get Photos with the api which are send to me? That would be a very important feature for me but I am not sure how to do it. Maybe you can throw me in the right direction^^

Also, I'm pretty sure that updates.getState needs to be called continuously for it to actually work.

That basically means you dont get updates pushed but have to pull them

[goodmind]

@Fuzzyma use

messages.getHistory#dcbb8260 peer:InputPeer offset_id:int offset_date:int add_offset:int limit:int max_id:int min_id:int hash:int = messages.Messages;

method as in example https://github.com/zerobias/telegram-mtproto/blob/feature/mtproto3/examples/chat-history.js

but it's not push updates

[Fuzzyma]

@goodmind I finally made it working. I call await client('updates.getState', {}) in intervals (which is basically ugly pulling) and subscribe to client.bus.untypedMessage.observe(...).

My mistake was, that muted channels do not emit messages - at least not the important one. You still get all the updates about who is tipping and so on - pretty annoying and such a traffic waste...

However, having tested it with an unmuted channel I got all updates and also all media. The media attribute contains photos and other stuff. You get an access hash and an id which you can theoretically use to download the media.

I say theoretically because it does not work. The docs say upload.getFile expects an InputFileLocation. There are several constructors to get such a location. I have an access has and the id so I use the inputVideoFileLocation like this:

  var media = await client('upload.getFile', {
    location: {
      _: 'inputVideoFileLocation',
      id: 'yourid',
      access_hash: 'andyourhash'
    }
  })

However, doing this yields a No predicate inputVideoFileLocation found error. This is also true for inputAudioFileLocation. For the types inputDocumentFileLocation and inputEncryptedFileLocation I get FILE_ID_INVALID (sure - its neither encrypted nt a document) and finally I cant use the normal fileLocation constructor because it does not work with id and access hash.

So - any idea? I really might want to open a seperated issue for that :D.

// EDIT: I manually added the inputVideoFileLocation. It now throws FILE_ID_INVALID, too // EDIT2: Finally figured out, that you need to look into sized which you also get bedide the id and the hash. Every size have his own location object which you can use to get your file. e.g.

  var media = await client('upload.getFile', {
    location: {
      _: 'inputFileLocation',
      dc_id: x,
      volume_id: 'xxx',
      local_id: xxx,
      secret: 'xxx'
    }
  })

[koloboid]

@Fuzzyma

I really might want to open a seperated issue for that

it's a good idea!

Anyone, is there a way to work with client.bus.untypedMessage. without manual polling using updates.getState/getDifference? It seems 2.x version does it (but did not fire any event), where as 3.x does not poll at all (but fires event when you poll manually).

[0x6c23]

I am using 3.2.11 and it does not fire any event for me, even with polling manually. Using this snippet:

// 2.x
  client.on("update", handleUpdate);

  // 3.x
  client.bus && client.bus.untypedMessage.observe(data => {
    console.log(data.message);
  });

  setInterval(() => {
      client("updates.getState", {}).then((state) => {
        console.log(state);
      });
  }, 1000);

[mortazakiyani]

hi i have vps that mtproto is running on that how i should manage about 3000 connections cpu 2 core 2 ghz ram 512 mb

[MauriPastorini]

Help: Is there a limit of the pulling data with "getState" ? Also, which version are you using for this updates? as I am not receiving anything with the last 3.2.11 version

[LuckStock]

Seems the above still doesn't work. Tried both @beta (2.2.2 version) and @3.2.11

updates.getState - method not working (promise doesn't resolve/reject). client.on("*") - listener not returning anything on update event. client.bus.untypedMessage.observe - not resolving/rejecting. client.bus.rpcResult.observe - not resolving/rejecting.

The only way to get new posts is to manually fetch the getHistory for every channel/chat using interval wrapper. @zerobias Do you plan to implement this highly required feature as an event?

[AlekseiBuzlov]

Still have no luck to get new messages on both versions 2.2.2 and 3.2.11 Does anybody have any working examples?

[StarNeit]

Hello @LuckStock , @GrinderBrad Did you solve your issues? I also want to get new messages, but didn't find good solution yet. Please let me know if you have found good idea.

[LuckStock]

@StarNeit , We have completely removed this library due to buggy behavior we were unable to solve or find a fix. We had to write a Python script based on Telethon and embed it in NodeJS as a shell script. It completely resolved our requirements for now.

[soapbravo]

@LuckStock That sounds interesting. Would you mind share your project public url? or github repo if that is open source? I really want to know / see how to use telethon for webapp including authentication. Actually I am developing something similar to webogram using this mtproto, but stuck on receiving updates. Thanks.