Open lefuturiste opened 2 years ago
Getting similar issue, even 2 minutes wait sometimes. I was told it's maybe because I'm using matrix.org homeserver which is known to be slow, but I don't see that slowness when sending messages using element.
Getting similar issue, even 2 minutes wait sometimes. I was told it's maybe because I'm using matrix.org homeserver which is known to be slow, but I don't see that slowness when sending messages using element.
yes and this is in a matrix.org server
I tried to debug a little bit and dig deeper in the code but I didn't found any evident issue, I think I need to profile the python code
Also might be worth bench-marking using other homeservers or even a local one.
@ users
See comments below line 5805. Syncing means updating the on-disk cache, by downloading all events from all rooms since the last app execution. This typical workflow of any Matrix client goes unnoticed in a graphical client, because of user interaction (browsing to the room, typing the message).
If you want encryption, syncing is necessary (encryption requires downloading state events, you need to know which devices are trusted or not, and matrix-nio is designed to require a sync if you want encryption).
@8go
One can send non-encrypted in matrix-nio without syncing. What about a command-line flag to disable the sync explicitly, for a single execution?
This is not only to send a message, but also any write-only operation (banning, ...) For example, OpenStreetMap Humanitarian wants to automate banning with matrix-commander
across 20 rooms (like a home-made Mjölnir).
When the sync is running, what about a descriptive message Syncing...
@ myself
There is an unsupported way in matrix-nio to send encrypted without syncing. On my test setup, I did the initial sync. For the following executions, instead of the sync, async_client.rooms[room_id] = nio.rooms.MatrixRoom(room_id=room_id, own_user_id=user_id, encrypted=True)
. I'm writing it for anyone who reads this and wants to try locally (or open a matrix-nio feature request).
Note from my testings: Performing a bulk ban in different rooms is just taking a few seconds per room, nothing compared with sending a message.
@opk12 @nukeador @lefuturiste
3 When the sync is running, what about a descriptive message Syncing...
OK, that can be done.
1 What about a command-line flag to disable the sync explicitly, for a single execution?
Doable. Yes. But this allows the user to shoot oneself into the foot. I am not sure what can go wrong if used incorrectly. Let's discuss option (1) a bit more. I really do not know what the worst case scenario is. Can the event database be somehow corrupted? Can one lose messages or other events? Will events just be out-of-order? First matrix-nio, then matrix, synapse, then the DB. Many dependencies/layers and I am not sure of the outcome.
What is your opinion? Please share!
I search for sync
in the documentation's page Examples
. The examples send room events before syncing:
A basic client
in main()
Log in using a stored access_token
in main()
's else
branchSending an image
uploads a file and sends a message, and it does not syncThe only case where the examples sync before sending a room event is when encryption is needed, in Manual encryption key verification
.
The specification to send an event is 7.8 Sending events to a room
. I successfully call it with curl as in:
#! /bin/bash
curl \
-X PUT "https://$HOMESERVER/_matrix/client/v3/rooms/$ROOM_ID/send/m.room.message/$(date +%s%N)" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
-d '{"body": "'"$BODY"'", "msgtype": "m.text"}'
The parameters are: homeserver, room id, transaction id, access token, message body. No parameter requires updating the local state and this curl is an example of a client that does send without syncing.
Uploading a file does not touch a room (you upload to a server, not a room) and is at 1.1.9.2 under 1.1.9 Content repository
and does not require storing local state.
A client's ability to make the server (Synapse, DB, ...) not follow the protocol (corrupt, alter the room's past, alter future behavior, ...) is a security issue on the server and not a fault of the client.
That the documentation sends before syncing also implies that you can send without syncing at all, because a network error during the later syncing would otherwise break / corrupt matrix-nio.
Thank you @opk12 for your research and your summary.
The matrix-nio examples do not convince me (as I wrote some of these examples myself) and they were written to keep things extremely simple, not to be best-practice code.
The Protocol example convinces me a lot more.
I take it, you @opk12 are voting in favor of a --skip-sync
option.
Should there be a --skip-sync
option and therefore the user decides if to use it or not by setting the option (default is sync==on)?
Or should there be no user option, no command-line option, but matrix-commander
decides when to sync and when not to sync?
Should it automatically skip syncing if there is only (a) sending, (b) xxx, ... ?
Please share your Thoughts? Input? Opinions?
The matrix-nio examples do not convince me (as I wrote some of these examples myself) and they were written to keep things extremely simple, not to be best-practice code.
If that's the case, this should be written next to each example, because people are jumping to the examples through links (not read the whole page) and use them as reference (copy-paste).
But then, is it possible that matrix-nio needs syncing to send? For example, in theory it could
Since I do not dare to give a definite answer, in short I am not sure, the best way forward might be to raise the question as an issue on matrix-nio
, as an open-ended questions:
plus additional narrow questions:
Other possible questions that come to mind:
Why would one want to sync before sending
What does one lose by sending without sync
Because it might be possible, but with various levels of degraded functionality or security (which may be relevant or not, depending on the use case).
I don't know who maintains matrix-nio, but pointing out that it was you who wrote some of those examples might help if anyone who answers got a misconception based on reading the documentation. However, I hope that the fact that the examples were kept in the docs until today implies a partial acknowledgement.
I did not test encryption with matrix-commander, but do I understand it correctly that this message should be sent encrypted?
If yes, then sending cannot default to auto-detect, and requiring an option for all operations would be consistent and more readable (when reading a command, I do not need GitHub just to remember whether it auto-skips) and would give the feeling that the program defaults to safety. What about --sync on|off
to leave our future selves the choice of a future auto
option or multiple safety levels.
I just looked at my own source code, and back in the day I put in some comments as I wrote the code for send
.
Around line 5881 you will find comments like:
# Sync encryption keys with the server
# Required for participating in encrypted rooms
...
# must sync first to get room ids for encrypted rooms
...
There are 2 API calls: keys_upload()
and sync()
. Not sure how slow keys_upload()
is in comparison to sync()
.
So, just blindly sending without a sync
will not work in all cases. Things that could go wrong when doing a send
without a sync
:
This is not a complete list of what can go wrong.
From this, I deduce that sync
cannot just always be removed for send
. By default we must keys_upload()
and sync
.
But send
might work in (some) cases, when there is no new state, so nothing needs to be sync-ed.
At the same time, the code is written such that even if the sync fails, the send will take place. Meaning, right now there is no 100% guarantee the sync will be successful. What I am implying is what @opk12 said before: sending without syncing might lead to some error (room id not found, key not found) but it will not lead to corruption on the server. This is another way of saying: it might not be so dangerous to skip the sync, especially in cases where no new rooms where created, no new keys were created, etc.
--sync on|off
might be the way to go. A value of off
would skip for sending (just sending, not receiving). Should we name is --send-sync on|off
to be more clear?
What is your preferred option name? --sync
? --send-sync
? --sync-send
? Opinions?
Then to make it even more complicated: the sync()
API call has a boolean parameter full_state
, so one could even envision: --sync none|partial|full
. none
meaning sync()
will not be called. partial
meaning sync(full_state=false)
. full
meaning sync(full_state=true)
. But the documentation does not explain well what the difference between full_state
true or false is. The docu just says:
full_state (bool, optional) – Controls whether to include the full state for all rooms the user is a member of. If this is set to true, then all state events will be returned.
As you can see I have opened this issue on nio
:
https://github.com/poljar/matrix-nio/issues/356
# Sync encryption keys with the server # Required for participating in encrypted rooms ... # must sync first to get room ids for encrypted rooms ...
- room id not known, send fails
This is not a complete list of what can go wrong. From this, I deduce that
sync
cannot just always be removed forsend
. By default we mustkeys_upload()
andsync
.
But note that all this is still about encryption. (And I wonder whether "get room ids" was related to some error message arising from async_client.rooms[room_id]
not being populated automatically without a sync, as described above.)
--sync on|off
might be the way to go. A value ofoff
would skip for sending (just sending, not receiving). Should we name is--send-sync on|off
to be more clear?
Once the local cache is updated, all operations will be affected. If "sync after X but before Y" is wanted, then it's clearer to call m-c once for X and once for Y.
Then to make it even more complicated: the
sync()
API call has a boolean parameterfull_state
I think it mirrors the /sync
endpoint's full_state
parameter from the C-S API. It helps think about an extensible CLI, but I'd focus on getting the sync vs no sync thing, to avoid feature creep.
I did some testing of my own today.
The setup matters a lot, so your mileage may vary drastically.
I tested only short send actions, like matrix-commander -m test
or similar. Sending a small image, sending to 2 rooms, etc. I only tested encrypted messages/rooms.
What I found is:
--sync off|full
has been implemented. See commit 6a752f2
So, even turning sync off, for sending single messages, this only reduces the total time by at best 33%. Again, mileage may vary drastically. But it is not a magic bullet and no, now it does not go in the blink-of-an-eye.
For the time being the default is --sync full
. If many people use --sync off
without problems we might switch default to off
in the future.
See https://github.com/8go/matrix-commander-rs
An idea to improve performance :smile:
Talk about it to your friends, post about it, spread the word and provide PRs. :pray:
And give it a :star: if you like the idea.
Eh but I fear that the network is the problem. The language impacts CPU usage, but m-c is spending a negligible time on the CPU; the usual symptoms of being CPU-bound are heating the computer, slowing down the system, or that the stack trace that appears at Ctrl + C contains some function that does a very long and tight loop (but that's not the case for m-c or for matrix-nio).
That said, it's not impossible that matrix-nio (as opposed to matrix-commander) does additional (necessary, or unnecessary) HTTP calls, so while I would not bet that switching the library could improve performance, I'd be happy to be proven wrong.
--sync off|full has been implemented. See commit 6a752f2
Thank you! I updated my script to use it.
Another tip to avoid performance penalty: do NOT use room aliases, but use room ids instead. Room aliases need to be looked up, adding another REST call to the server.
If you just want to login, verify and send a text or a file, you can now use the Rust version. Rust version cannot listen (hoping for PRs). Rust version is also not optimized (--sync off
does not yet exist), but still you can try and see if it makes any difference at all in performance.
Feedback is welcome.
Check out:
Hi,
I'm testing this software and no matter what I try to do it always freeze for one minute after finally doing what I want.
I wanted to send a message to a specific room, I issued this command :
However this is taking 67 secs! for a single message? Where it did go all wrong? It's seems to be waiting for some thing between
2022-08-22 08:14:01,495: DEBUG: matrix-commander: Rooms are: ['!REDACTED:matrix.org']
and2022-08-22 08:15:37,089: DEBUG: matrix-commander: stdin is not ready. A pipe could be used, but pipe could be empty, stdin could also be a keyboard.
My environnement:
Anyway thanks for your work