Open BillCarsonFr opened 2 years ago
Interesting related blog post https://blog.neko.dev/posts/unable-to-decrypt-matrix.html
@BillCarsonFr Wow what a great write-up you found at https://blog.neko.dev/posts/unable-to-decrypt-matrix.html
Please correct me if I'm wrong, but having looked at that, it seems that just about all of those errors could be resolved via wizard/options provided to the user by the client which could try to resend encryption key with help of server or to even create a new one for the channel/room. I don't see any 'technical' issue/blocker other than currently missing functionality client-side to prompt user for permission to, with the help of in-between server, do what's necessary to fix room encryption/decryption.
If there's a 'security' issue with properly re-identifying the correct user, for example to re-provide keys to the other user while they've gone offline so they can pick them up from server when they are online again, can't we have a simple "challenge/response" ie question/answer to "pick-up" the keys? (I'm trying to avoid creation of a "new" room/channel and instead fix one already created but giving errors of unable to decrypt. The sender's device has not sent us the keys for this message.)
It seems the reason certain steps aren't by default taken to automatically fix such issues, is that it could be a security risk in certain situations. But if the "creator" of a room gives needed permission, it seems in a sense trivial to resolve such decryption/key errors etc. Or am I wrong and missed something?
And to the author of: https://blog.neko.dev/posts/unable-to-decrypt-matrix.html Many thanks!
Please correct me if I'm wrong, but having looked at that, it seems that just about all of those errors could be resolved via wizard/options provided to the user by the client which could try to resend encryption key
FTR, there used to be such UI, but it was prone to social attack and annoying in the UI.
It's something explored, but it's a bit hard to have a fix all encryption problems
button. We are trying to go to the bottom of the root cause for distribution failure by moving to the rust sdk
@BillCarsonFr Ah ok guess I'm relatively new so didn't know of the older UI. Personally, I'd be ok with the "social attack" risks and UI annoyance versus the embarrassment of inviting users to my new chat and getting decryption errors.
My concern was that it may be impossible to fix the root for all cases of decryption issues, especially if some are due to security measures which may need to be over-ridden by user consent in order to fix, which brings us back to that UI annoyance.
But regardless, it's great to hear the underlying sdk is being improved to hopefully eliminate or greatly reduce these issues. I had heard of rust in conduit (I run go-dendrite) but didn't know Kotlin SDK is being all redone in rust, is that right?
Anyway, thanks again to everyone working on this! (I look forward to some beta-testing with the new sdk when its ready for that.)
The link to send debug logs does not appear on Element Web on my server. Did this UI change, or is there some option that an admin needs to enable to get that to show up?
@anon8675309 your config.json must have the URL to send debug logs to, like the example https://github.com/vector-im/element-web/blob/develop/config.sample.json#L25
Can you confirm that this is still the case? I have the bug_report_endpoint_url entry from the sample and the link to send debug logs does not appear. If it's working as expected for you, I'll set up a new server and open a new ticket with the minimal steps to reproduce. (I searched for a report of this issue and didn't find anything, but I'll do so again before opening a new issue).
Was able to send feedback from Element desktop... and have this issue with one single user on my home server, both users on the same server. Upgraded the room to version 10 and got the error again after not even 10 messages
Hi there, I have been encountering this problem a lot recently. I've already sent logs from element-web and element-android as the person who received encrypted messages that cannot be decrypted, but I could not yet send logs as somehow sending such problematic messages.
Question: when this situation happens in a matrix room, where a given user ends up only sending messages that cannot be decrypted by other room members, is there any (even intricate) known workaround? The only "workaround" I used so far was to upgrade the room to a newer room version, which solves the issue by creating a new room, but I can't really call this a proper solution… and right now I have this problem on a room that is already using version 10…
@Ezwen I've found that asking sender of messages to run /discardsession
usually fixes any messages moving forward. Though it does not solve the messages with the encryption issue.
@BillCarsonFr just so that I'm on the same page: is there a lack of sender / receiver logs to review?
Will be fixed with Element R
For the uninitiated, could you please link to something explaining what this is? It's ungoogleable as R is an element on the periodic table, and exists in matrixes sometimes :)
I agree with @theelous3 that adding context on Element R would be helpful. From my quick search it seems it's probably short for Element Rust which is Element using the matrix-rust-sdk. Related links:
Anyway, good job team on getting this rolling :)
is there any ETA? I'm living in an area with really bad cellphone coverage, we're getting UDT messages as soon as somebody leaves the village (well, the ones which didn't switch back to whatsapp)
@yennor Not an ETA, but perhaps a short-term remedy. I've found that installing the newer Element Android v1.6.3 version that's based on the Rust SDK has greatly helped in my rooms.
You'll likely need the "vector-gplay-rustCrypto-arm64-v8a-release.apk" file.
@lousando thanks, I'll give it a try and hope for the best. What about the element-desktop version? can't find anything about rust there.
@lousando thanks, I'll give it a try and hope for the best. What about the element-desktop version? can't find anything about rust there.
Yeah I'm not entirely sure if the desktop app has been swapped out to use the Rust SDK. I usually only keep up with the Android repo as there is usually where the unencrypted problem begins for myself and my recipients.
element web/desktop hasnt been changed to use the rust sdk for encryption yet, its a labs option (not one of the beta's) which has to be enabled in the config.md not just in the labs menu
Every friend group or company that evaluted Element/Matrix that I've been a part of has run into this issue and has eventually given up on Matrix with this app. It's completely unrealistic that a non technical person could use it at the rate which the basic functionality of reading a message breaks and how hard it is currently for users to recover from a broken session in an encrypted group chat with multiple people. You're spending more time on tech support than actually communicating. I can't understate how important it is that this issue finally gets fixed after multiple years.
Agree with @kwinz's comment above. I also can't stress how critical this UX problem is. I can't count the number of people I've encouraged to try Element/Matrix that was turned off by this baffling problem. There are group chats that work fine for a few weeks/months, but suddenly one (or a few) person(s)'s messages would show up as "Unable to decrypt" for everyone else.
This is an incredibly big problem, and I've heard from many people across the world that this is THE REASON they gave up on Element/Matrix.
Agree with @kwinz's comment above. I also can't stress how critical this UX problem is. I can't count the number of people I've encouraged to try Element/Matrix that was turned off by this baffling problem. There are group chats that work fine for a few weeks/months, but suddenly one (or a few) person(s)'s messages would show up as "Unable to decrypt" for everyone else.
This is an incredibly big problem, and I've heard from many people across the world that this is THE REASON they gave up on Element/Matrix.
I concur, if I have to explain a friend why their messages behave like this they'll just go back to whatsapp and consider me annoying
We're actively working on this, despite the lack of activity on this particular issue.
In particular, within the last few months we have a dedicated test suite now to identify these sorts of failure modes, which will ensure that clients using the rust SDK FFI bindings (e.g Element X) or JS SDK with rust crypto (e.g Element Web) work correctly going forwards. We're also working through the causes we've identified and fixing them with regression tests where appropriate.
This is going to take some more time I'm afraid, and we all understand how frustrating it is when things break. In the mean time, if you do happen to be using Element X and/or Element-Web and happen to see a message which is undecryptable, please send a rageshake: I actively review unable to decrypt bug reports when they come in on those particular clients.
if you do happen to be using Element X and/or Element-Web [...] I actively review unable to decrypt bug reports when they come in on those particular clients.
@kegsay Does this include Element-Desktop?
Currently this does not include Element Desktop as rust crypto isn't enabled by default (yet). We expect this will change on a timescale of weeks not months. We've been discussing remaining blockers literally yesterday. I'll edit this issue when it's enabled by default on Element-Desktop.
2024-03-18: Element Desktop still doesn't have rust crypto enabled by default yet, but I will update this issue when it is. 2024-04-11: We are beginning to roll out to Element Desktop incrementally.
Created a new issue that could cause UTDs https://github.com/element-hq/element-meta/issues/2374
2024-03-18: Element Desktop still doesn't have rust crypto enabled by default yet, but I will update this issue when it is. 2024-04-11: We are beginning to roll out to Element Desktop incrementally.
Element Desktop received Rust crypto for new sessions back in February, for what it's worth. Rollout of migration for existing sessions is tracked at https://github.com/element-hq/element-web/issues/27001.
Updates on recent progress here:
Updates on recent progress here:
...
Just want to say a bit thank you to @richvdh for this progress update. ❤️ There is still MUCH that needs to be fixed in terms of bugs and UI/UX when it comes to encryption/decryption weirdness, and so many people I introduce to Element still get bounced off because of these problems. But at least getting these progress updates is helpful!
Recently I've been giving updates for this on This Week in Matrix. If you fail to decrypt a message please:
We often need both sides of the conversation to fix the issue.
It would also be helpful for us if you can opt-in to analytics, as that feeds into our graphs which plot UTDs in aggregate. The general trend of the past few months has thankfully been fewer UTDs across clients that opt-in, but there is more work to be done here.
I usually got that kind of problem when me or the peer beeing in an area with bad mobile phone connection. With bad I mean really bad. You can get disconnected from the network for several minutes all the time, randomly get connected again for a few seconds, or beeing connected, but almost no data gets through. I haven't been there (rural area of Colombia) since last year and won't for a few months. So So I can't tell if the situation improved with the new clients. But maybe for your test-suite, simulate random tcp package drops (very high percentage) with high RTT (Several seconds, sometimes I measured up to 20 seconds. around 5-8 seconds is normal). And sent a few thousand messages there and back.
Does https://github.com/matrix-org/matrix-js-sdk/pull/454 already "implement" solution for: https://github.com/element-hq/element-meta/issues/647 ? And if so how would my friend request keys from me (on Desktop Element) on a new fresh install of his?
Are we able now to send decryption keys to whomever we wish?
@jittygitty yes but Rust Crypto does not support that and that is what Element uses, so the js-sdk PR is unrelated
@t3chguy That's good news, so I guess if I wanted to make use of this feature, I would have to install the "Web" version client of element chat on my webserver? (I currently run Dendrite already, but with mobile and Desktop clients only.)
Or is there another client app that uses js-sdk? (Element Desktop does not use it?) Or will rust crypto gain it soon?
I use another account on 'matrix.org' using desktop+mobile, for that acct. we can simply login at chat.element.io and if I'm logged in via my other devices, I can send keys to my web chat.element.io and resend to friend who lost his? thx
Element Desktop = Element Web + Electron. Element Web only supports Rust Crypto at this time. As for other matrix-js-sdk consumers I suggest finding a place discussing Matrix rather than Element specifically.
Apologies, indeed, seems I'm confused as to what matrix SDK is used by what "client", and I had not noticed the https://github.com/matrix-org/matrix-js-sdk/pull/454 was outside of element-hq repositories. So, I guess you are saying that likely none of the clients under https://github.com/element-hq are able to leverage that matrix-js-sdk pull 454?
Is there a place I can find all such pertinent information on the various clients available? If anyone can point me in the right direction with a link would be appreciated. Otherwise, I'll try some search engine lookups to dig for such info. thx
It seems Cinny uses the matrix-js-sdk, and maybe there are more, but I havent found another client that does. Then, its an other question if Cinny makes use of this feature of the SDK. You could ask about that in Cinny's support room.
Another round-up of recent updates. (Most of these have already been reported in TWIM, but I think it's handy to have a record here too.)
A reminder that we're still at war with this issue, and it's incredibly helpful for people to send debug logs when they come across UTD errors.
That said, our analytics show that we are starting to make progress here:
Unable to decrypt Epic Issue
Meta issue relating to all "Unable to Decrypt" problems, a.k.a "Waiting for this message, this may take a while".
Needed information to resolve a reported issue
In order to properly debug an Unable To Decrypt error, we need logs from the receiver of the message (the one seeing the issue) and those from the sender. We can't debug issues without logs from both sides.
In your bug report, please identify which event you can't read. Either include the event ID, or something like "message from Dave at 10:10".
How to send rageshake from Element Web:
submit debug logs
buttonHow to send rageshake from Element Android:
Report Bug
How to send rageshake from element iOS:
Feedback
button at the bottomCauses of Unable to Decrypt errors
🟢: We believe that this will be fixed with Element R 🕛: Affects a feature which is not yet supported in Element R
We categorize the main sources of UISI errors as follow:
Client Issues
Sender Side
The sender failed to send the keys. There’s no consistent retry logic on network error, so key sending is fragile. https://github.com/element-hq/element-meta/issues/673. Test: TestClientRetriesSendToDevicehttps://github.com/element-hq/element-web/issues/20962The keys may be sent slowly:https://github.com/element-hq/element-web/issues/24680 Test: complement-crypto#34https://github.com/element-hq/element-web/issues/24681Sharing historical room keys on invite is slow and should prioritise recent messages https://github.com/vector-im/element-web/issues/23778https://github.com/vector-im/element-web/issues/24138https://github.com/vector-im/element-web/issues/23792https://github.com/matrix-org/matrix-rust-sdk/issues/3127Receiver Side
https://github.com/vector-im/element-meta/issues/762 Test: complement-crypto#37https://github.com/element-hq/element-web/issues/23113https://github.com/vector-im/element-android/issues/5905 (No test since no specific testcase is known - closed because Rust is sufficiently different)https://github.com/vector-im/element-web/issues/24450https://github.com/vector-im/element-web/issues/24682 Test: complement-crypto#38https://github.com/element-hq/element-x-android/issues/2520https://github.com/vector-im/element-ios/issues/7479 (Cannot reproduce, so no test)https://github.com/vector-im/element-web/issues/25723 Test: complement-crypto#35Element X iOS: https://github.com/matrix-org/matrix-rust-sdk/issues/3110Server Issues
Key Backups
Protocol Issues
Missing features
UX
Expected UTD
User Config