search

element-hq / element-web

A glossy Matrix collaboration client for the web.
https://element.io
Apache License 2.0
10.74k stars 1.89k forks source link

"Unable to decrypt: The sender's device has not sent us the keys for this message." (The UISI bug) #2996

Closed richvdh closed 1 year ago

richvdh commented 7 years ago

Update 2023-02-27

This issue is now superceded by https://github.com/vector-im/element-meta/issues/245

Historical information follows.

This message (or, less often, the closely related "OLM.UNKNOWN_MESSAGE_INDEX") can be caused by a number of things. This bug serves as a reference to the reasons we know about.

Client-specific bugs:

Protocol/server things:

Unexplained:

ara4n commented 7 years ago

For anyone reading this looking for a workaround, the best advice currently is that if you suddenly find yourself unable to decrypt messages from someone, ask them to open up your contact details on their client. This should force their Riot to resync its copy of your device list, increasing the chance that the next message sent will actually be encrypted for your current device.

If this doesn't work, you have no choice but create a new room (or worst case, export your session keys, logout, login and import your session keys), until the workaround in vector-im/element-web#3553 is implemented (or this meta-bug is finally closed up)

lampholder commented 7 years ago

Making forward progress on this requires either @richvdh to hop back into it, or for him to hand over to somebody else.

ara4n commented 7 years ago

I just spent a while reviewing all of the known remaining UISI causes with @richvdh. UISI bugs fall into two broad categories:

Wedged olm sessions:

Missing megolm keys:

Rich estimates UISIs to roughly be caused 50/50 split between the two.

However, all of the 'missing megolm key' class of bugs can be worked around by giving users a way of recovering missing megolm keys - and in some cases (broken federation; matrix-org/synapse#2165) this is the only plausible solution. In turn, if we had a way of recovering missing megolm keys, we'd also have a way to share history to new devices - the infamous 'share history' bug (#2286).

Therefore the suggestion is to focus entirely[1] on solving the problem of sharing megolm keys, given the value of solving both the 'missing megolm key' bugs as well as the 'sharing history' feature is greater than the value of solving the individual bitty 'wedged olm session' bugs (which all have different solutions). This means setting aside UISI hell and forging ahead and solving history sharing (#2286) or at least a subset of it. This could well include improving the UX for sharing history by supporting cross-signed keys (#2714).

[1] We can probably progress the 'multiple tabs' problem (#2325) in parallel. And the plan is to finish the devicelist race vector-im/element-web#3796 first.

richvdh commented 7 years ago

vector-im/element-meta#1143: If you reuse node-localstorage and share curve25519 keys between different device IDs, Olm wedges <-- NOT A BUG FOR RIOT.

Well, it is a bug for riot, in that if anyone uses the js-sdk in the obvious manner, riot fails to talk e2e with the resultant client. It's arguable whose fault that is - ideally both ends would be fixed. But it's not a bug for riot inasmuchas it doesn't affect riot<->riot comms.

...possibly other bugs where the act of loading the target's MemberInfo kicks the sender into refreshing their device list?

AFAIK the only thing that loading the MemberInfo would solve these days is vector-im/element-web#3796.

ara4n commented 7 years ago

writing https://github.com/vector-im/riot-web/issues/2286#issuecomment-299605919 made me realise that perhaps we can also improve the experience here in general with better error messages. For instance, do we have any way of detecting when an Olm session has got wedged, such that we can complain about that (and perhaps reset it?) rather than just whine about missing megolm keys?

richvdh commented 7 years ago

Yes, we can certainly improve this. We can give the user feedback about failing to decrypt to_device messages (though they tend to get replayed at initial sync, so we'd have to think how to avoid false positives). https://github.com/vector-im/riot-android/issues/800 randomly, covers that. If we can get it reliable, we can start a new Olm session to try and unwedge things. We can also consider giving better feedback from the sender's end (https://github.com/vector-im/riot-web/issues/2494).

In general it's hard to tell the cause of any particular UISI, because you can't correlate it to a to_device you couldn't decrypt.

pik commented 7 years ago

If users restore a device backup or tab whilst the existing device is still around and active, Olm will wedge. If you reuse node-localstorage and share curve25519 keys between different device IDs, Olm wedges <-- NOT A BUG FOR RIOT<->RIOT.

While it is possible at the moment I don't really like the idea of the same keys on multiple devices (this is surely a security risk). Notably also device_keys changing while device_id remains the same is another way to wedge olm sessions (the client sharing the megolm session key thinks they sent keys to the associated device, but that device has new keys and cannot decrypt the olm session).

richvdh commented 7 years ago

@pik: please take your questions about vector-im/element-meta#1143 and vector-im/element-web#3822 to the relevant bugs.

Another thing we should consider on this bug is a way for the recipient to distinguish "the sender failed to send to you" vs "the sender chose not to send to you" (either they blocked you or your device explicitly, or because they had the 'Never send encrypted messages to unverified devices from this device' setting (https://github.com/matrix-org/matrix-js-sdk/pull/336) checked).

Of course that would probably mean the sender sending a "you're blocked' notification to the recipient, but that wouldn't be hard.

richvdh commented 7 years ago

Added vector-im/element-web#3845 for the "blocked vs failure" sidebar

ubmarco commented 6 years ago

The workaround 'opening the contact details' did not work for us. I can read all messages from the mobile of my colleague but not from Linux Riot. We found a workaround: my colleague closed Riot, deleted the config folder ~/.config/Riot and started the app again; now his device showed up as unverified but I could read the messages again; after verification, we get the green lock and everything works as expected.

Hetti commented 6 years ago

I had problems with my Phone Key after restoring a TWRP Backup. I couln't read my phone on the same account with my web client on the desktop. With exactly this error: "Unable to decrypt: The sender's device has not sent us the keys for this message." After Bugrequest I got following Fault + Solution: Fault: "Looks like by restoring your phone from backup its crypto state got completely hosed and out of sync with the server."

Solution that worked: Export my room keys, logging out & in again and reimporting them on the telephone.

Phone will get new key after log out and new login.

Twitter Thread: https://twitter.com/Th3PeKo/status/988821611313758208

gerroon commented 5 years ago

For anyone reading this looking for a workaround, the best advice currently is that if you suddenly find yourself unable to decrypt messages from someone, ask them to open up your contact details on their client. This should force their Riot to resync its copy of your device list, increasing the chance that the next message sent will actually be encrypted for your current device.

If this doesn't work, you have no choice but create a new room (or worst case, export your session keys, logout, login and import your session keys), until the workaround in vector-im/element-web#3553 is implemented (or this meta-bug is finally closed up)

None of this works forme me:( Creating new room, deleting the user from device and resigning in, trusting all the users oif the room from scratch etc does not work. And I have this issue with only one user that I interact with. The other user is using Android (Fdroid) and Linux versions.

I am wondering if anyone else was completely able to get rid of this issue without deleting the user from Synapse.

dbkr commented 4 years ago

Opened https://github.com/vector-im/riot-web/issues/13473 as a specific type of UISI that seems to be cropping up

jlusPrivat commented 3 years ago

I had long troubles finding out, that the option "never send encrypted messages from this session to unverified sessions" was causing troubles... In hindsight it was pretty obvious, that it didn't work, nevertheless it cost me many hours thinking it was my HS config.

aaronraimist commented 3 years ago

@jlusPrivat What version of Element are you using? The current version should show a different message when that option is enabled.

Instead of saying "Unable to decrypt: The sender's device has not sent us the keys for this message." it should say "Unable to decrypt: The sender has disabled encrypting to unverified devices."

jlusPrivat commented 3 years ago

I use 1.7.16 (latest) for the windows as well as the browser variant (from element.io). The android version (1.0.13) showed "Waiting for this message. This could take a while" (spontaniously translated from german version). That would've been neat, if the message showed, that the sender disabled encryption to unverified devices.

aaronraimist commented 3 years ago

@jlusPrivat I don't know anything about Android but the web version definitely should show different messages. I thought it might be a translation issue where someone just used the same translation but those messages don't appear to be translated at all.

screenshot

Are you sure you didn't see these messages? Can you test again? They look very similar so you may have just stopped reading at "Unable to decrypt". If it doesn't say "Unable to decrypt: The sender has disabled encrypting to unverified devices" when that option is enabled then you should file a separate bug.

Mjk29 commented 3 years ago

Started getting this message a couple hours ago using the Android client, as well as the windows client. ** Unable to decrypt: The sender's device has not sent us the keys for this message. **

Saw the same error on 1/3/2021 after an update, resolved by verifying the other party.

jungletek commented 3 years ago

I had long troubles finding out, that the option "never send encrypted messages from this session to unverified sessions" was causing troubles... In hindsight it was pretty obvious, that it didn't work, nevertheless it cost me many hours thinking it was my HS config.

That option is terribly unintuitively named / described. Would love to see that changed, clarified. The way it is now, I and other well-meaning security-conscious new users read it as doing the opposite, FWIW.

stephen304 commented 3 years ago

In addition to having a better error message, I think it would be a good solution to warn the message sender the first time they send a message in a room if their setting will prevent their message from being read by people in the room. I'm in a large room and it's just pages and pages of UTD as new users come in and we have to figure out whether it's just lagging or whether that person didn't change the setting. The room admins set the room topic telling everyone to change the setting, which is a really annoying thing to have to do, especially since a good portion of people don't seem to notice it.

aaronraimist commented 3 years ago

@stephen304 what setting are you talking about? "Never send encrypted messages to unverified sessions from this session"? Why do they have to tell everyone to change it? It is not on by default.

stephen304 commented 3 years ago

@stephen304 what setting are you talking about? "Never send encrypted messages to unverified sessions from this session"? Why do they have to tell everyone to change it? It is not on by default.

Beats me. Last time I created an account was years ago, all I know is tons of people are joining because of recent events and it's a mess. If it's not on by default then maybe adding a feature that reminds people that half of the room can't read their messages would be a good idea. Because it seems like a ton of people are unaware.

weeman1337 commented 3 years ago

I am struggling with that, too at the moment.

I have a desktop Element client showing the message Unable to decrypt: The sender's device has not sent us the keys for this message.. Clicking re-request ends up in an incoming key request on my mobile client. But the request then is stuck in "pending" state. I don't know whether that is already covered by an issue listed in the description.

mattgphoto commented 2 years ago

Running into this on a new account, as well from Desktop Client, which is the only client I'm logged into and verified on.

masscream commented 2 years ago

I also experience this bug on my two synapse instances. Logs were sent from desktop and Android clients.

vmario89 commented 2 years ago

this bug happens for 2 years now :-( no idea to get rid

masscream commented 2 years ago

In the meantime, I upgraded both synapse instances to the latest version and forced to update the Element on the clients side. The new messages started coming OK, the "corrupted" ones, even sent for test purposes before the upgrade stayed hanging, probably forever. If it now stays working, I'm OK. Otherwise I would probably turn off the encryption and its enforcing preference from the new rooms...

syrakozz commented 2 years ago

its become very annoying with this core bug , i start thinking to move on from this library !!!!

the-j0k3r commented 2 years ago

Yep, I found this bug report just by googling the message. So this means its still a problem. and workarounds lets see if it fixes it, so far not much luck.

dkasak commented 2 years ago

To be clear, this is an umbrella issue with many possible different causes, not a narrow problem that's not being worked on. Most of the time when you're unable to decrypt a message due to a bug, you'll get such an error message displayed.

Therefore, your UISI might not have the same cause as the next one. Many causes of UISIs have already been solved and solving the remaining ones is currently a topic in active focus.

turt2live commented 2 years ago

We're still collecting rageshakes/bug reports for those cases, by the way. Best thing to see them fixed is to submit logs from Settings -> Help -> Submit Debug Logs from both the sender and receiver sides, referencing this issue.

the-j0k3r commented 2 years ago

Well this was a new experimental non offical room for a project, trying to find it if it was worth having an official room or not made from it.

Either they could see my messages and me not see theirs or vice-versa in the space of 24 hours the situation just decidedly against any encryption at all. No one cares to repeat that experience, so encrypted room is a flop for this end. Instead a plaintext room was created and lets hope people return.

I hold no hope the people involved will share their logs, some were frustrated and left. If I am successful will report back with logs..

However note that workaround posted about opening profiles for the affected people and forcing some resync has no effect.

Thanks.

the-j0k3r commented 2 years ago

2 of us already sent the logs with the GitHub issue to this URL

But consider that different people can login from different devices, and then if you need to repeat this debug log submission in each device, I.m just asking, because if that needs doing then logs.date is the least of your problems to solve this.

@turt2live thx

lachesis commented 2 years ago

My network is moving back from Element to Signal as a result of this bug. In particular, my messages sent from Element Desktop often enter this state for other users running Element Android. Nobody is having any issue Desktop -> Desktop or Android -> Android, or even Android -> Desktop. Sometimes messages exit this state, but I'd say only about 1/3rd ever become readable. Most remain in this state forever.

the-j0k3r commented 2 years ago

@lachesis I had a feeling that was also a factor, hence I mentioned,

It also happens to me when via DM to people on different servers as I found out just yesterday,. And to regular users vai existing DM windows after logging off and back on.

Session verification on login should be 100% one step not two two as is.

As a result I came to the conclusion that

A) Cannot force anyone to use just one platform or any particular client (shouldn't be an issue anyway) or force them to verify sessions. B) Because of A, Encryption is a pipe dream for on matrix.org and a quick swift kick to any privacy focused communities.

I have heard on help channels, other servers dont have such issues either.

uhoreg commented 2 years ago

Please note that this isn't a single bug. This issue can be caused by multiple different things. We're investigating and have fixed some issues, but there may be other problems.

As mentioned earlier, you can help us fix issues by submitting debug logs on both the sending and receiving sides when it happens.

the-j0k3r commented 2 years ago

@uhoreg

Submitted logs already, thanks, and Im pretty aware there are many vectors at play, its quite obvious that different platforms are not playing well together amongst other factors.

Cant do anything else, having submitted multiple logs for different issues linking to specific bug reports. we are at the mercy of your foo.

If you need any more specific info let us know, Ill try as best possible providing such.

However may I suggest that verification of the session is done in a single step for everyone on all applicable platforms I notice that the current setup allows non verified sessions and this is not good IMO.

Good luck.

ghost commented 2 years ago

I'm working in a build/code & development environment, logging in from 30 or more installations of my OS per day...

There should be a SIMPLE box to check "Yes all these logins are ME..." I'm verifying them ALL by my phone... my Samsung android device has no issues... but this incessant need to wait because of :

" Unable to decrypt: The sender's device has not sent us the keys for this message. Re-request encryption keys from your other sessions."

happening across my multiple installs on three different testing machines is more than an annoyance. I can't "re-request encryption keys... from installations that no longer exist...

disheartening to see this is an open & unresolved issue from 2017

uhoreg commented 2 years ago

If you are logging in, then it is expected that you will be unable to read old messages until your device has access to key backup or can receive the keys from another device. This is not a bug; it's just the way encryption works.

disheartening to see this is an open & unresolved issue from 2017

This is an umbrella issue. There are many different things that cause "Unable to decrypt" messages, some of which are not bugs but are expected behaviour. Several of the causes have been fixed over the years. Some don't have a clear fix.

foresto commented 2 years ago

At least some of us following this umbrella issue appreciate that it's not easy and will take time.

Nevertheless, one form or another of this error still pops up somewhat often when it should not, and has been doing so for years. It creates frustrating experiences that undermine confidence in Matrix, driving new users away, and makes it very difficult to get Matrix accepted by our communities. Given the impact of the so-called "network effect", perhaps it's time to prioritize sorting out the various causes of this issue, fix the ones that are bugs, and fix the UI around the expected cases?

For my part, I have spent many hours helping less technically inclined users troubleshoot things when they go wrong, and have submitted reports and logs when possible. I have been doing my best to manage expectations among my contacts, assuring them that Matrix and its client apps are improving, and that the rough edges are being filed down, and that bugs are not being ignored. After years of frustrations, though, people are (understandably) losing patience. If these bugs aren't fixed soon, I fear people are going to give up on Matrix. Nobody wants an unreliable messaging system.

The latest This Week in Matrix had an encouraging excerpt about chasing down E2EE bugs in the coming week. Are the bugs behind this issue to be included in that effort?

ghost commented 2 years ago

I just find it irritating... that the moment I finish verifying my login with my phone as a secondary device... all previous sessions, which were ALSO verified with my secondary device...

are now UNTRUSTED?

Nobody else but me is logging into this account...

All previous desktop sessions are no longer accessible... as they've been wiped away during my constant testing from within newly installed desktop instances...

Perhaps I am an unusual use case?

spence

Sent from ProtonMail mobile

-------- Original Message -------- On Dec 13, 2021, 2:14 PM, Forest wrote:

At least some of us following this umbrella issue appreciate that it's not easy and will take time.

Nevertheless, one form or another of this error still pops up somewhat often when it should not, and has been doing so for years. It creates frustrating experiences that undermine confidence in Matrix, driving new users away, and makes it very difficult to get Matrix accepted by our communities. Given the impact of the so-called "network effect", perhaps it's time to prioritize sorting out the various causes of this issue, fix the ones that are bugs, and fix the UI around the expected cases?

For my part, I have spent many hours helping less technically inclined users troubleshoot things when they go wrong, and have submitted reports and logs when possible. I have been doing my best to manage expectations among my contacts, assuring them that Matrix and its client apps are improving, and that the rough edges are being filed down, and that bugs are not being ignored. After years of frustrations, though, people are (understandably) losing patience. If these bugs aren't fixed soon, I fear people are going to give up on Matrix. Nobody wants an unreliable messaging system.

The latest This Week in Matrix had an encouraging excerpt about chasing down E2EE bugs in the coming week. Are the bugs behind this issue to be included in that effort?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

uhoreg commented 2 years ago

The latest This Week in Matrix had an encouraging excerpt about chasing down E2EE bugs in the coming week. Are the bugs behind this issue to be included in that effort?

My understanding of the community testing sessions is that it has been mainly focused (so far) on testing previously-reported issues to see if they are still problems. The bugs behind this issue tend to be difficult to reproduce, so I don't think that they will be part of the community testing sessions.

Some of the things that we have planned soon for this bug are:

For my part, I have spent many hours helping less technically inclined users troubleshoot things when they go wrong, and have submitted reports and logs when possible.

Thank you!

uhoreg commented 2 years ago

I just find it irritating... that the moment I finish verifying my login with my phone as a secondary device... all previous sessions, which were ALSO verified with my secondary device... are now UNTRUSTED?

If previously-verified devices are no longer verified, that's a different bug. Please file a new issue, and submit debugging logs from both devices that you're verifying.

the-j0k3r commented 2 years ago

Trying to get this to work across devices and verified sessions is like pulling teeth, Ive tried it and given up, its a lovely idea on paper but in practice its not even ready for alpha never mind beta or production.

And I'm fully aware of the million pieces of the jigsaw, its just not even ready and should come with a warning label, unless you dont are about UX. Which then its OK to alienate whomever you like otherwise.

Ive given up its already wasted enough of my time and everyone else's time who I dragged into this can of worms, even just for myself my own previous messages on same device, browser and same verified sessions is a big failure, when one cant read his own messages. Just log out and log back in and its not even consistent, one login maybe OK the next all gone to hell.

Good luck to you all on both ends of this stick, I will not be using any encryption here.

dkasak commented 2 years ago

@the-j0k3r: I understand the frustration, but please let's try keeping constructive here since this is essentially a bug tracker.

It's hard to tell what's happening on your end, but your case is definitely not typical. For instance, I keep fully verified sessions with a few friends and family and relatively frequently log in from new devices, and I'm just not seeing the problems you're describing. That's why the encryption doesn't come with big, fat warnings -- because it works much more often than not.

You must be encountering some pathological case, but to identify what's wrong we must first see what you see. I can appreciate you may be beyond the point where you're interested in engaging to supply this info, but I can assure you it's not the case that the team doesn't care about this.

ghost commented 2 years ago

The "unusual-ness" is that none of my prior sessions exist, save the one on my phone

... most exist for only a few hours, some only exist for a few minutes before I have installed a new version of my software over top the previous versions...

Therefore, there are no other sessions to request decryption nor seek keys from...

Once I verify a session with my phone, I see no reason for previous sessions to become once again unverified or untrusted..

-------- Original Message -------- On Dec 14, 2021, 4:18 PM, Denis Kasak wrote:

@.***(https://github.com/the-j0k3r): I understand the frustration, but please let's try keeping constructive here since this is essentially a bug tracker.

It's hard to tell what's happening on your end, but your case is definitely not typical. For instance, I keep fully verified sessions with a few friends and family and relatively frequently log in from new devices, and I'm just not seeing the problems you're describing. That's why the encryption doesn't come with big, fat warnings -- because it works much more often than not.

You must be encountering some pathological case, but to identify what's wrong we must first see what you see. I can appreciate you may be beyond the point where you're interested in engaging to supply this info, but I can assure you it's not the case that the team doesn't care about this.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

the-j0k3r commented 2 years ago

@the-j0k3r: I understand the frustration, but please let's try keeping constructive here since this is essentially a bug tracker.

It's hard to tell what's happening on your end, but your case is definitely not typical. For instance, I keep fully verified sessions with a few friends and family and relatively frequently log in from new devices, and I'm just not seeing the problems you're describing. That's why the encryption doesn't come with big, fat warnings -- because it works much more often than not.

You must be encountering some pathological case, but to identify what's wrong we must first see what you see. I can appreciate you may be beyond the point where you're interested in engaging to supply this info, but I can assure you it's not the case that the team doesn't care about this.

I have submitted debug logs ad nauseam linking this issue every time I encountered such issues, so your assessment is incorrect, what I will not do is keep trying to test the limits of my sanity and patience by trying the same thing over and over and expecting different results, when the results are consistently towards the negative. Im sorry you feel this is not constructive, but maybe its that frustration you talk about that made it sound, not constructive.

I never stated, I wouldn't share debug logs or information to help you guys, As a developer I understand without relevant actionable data fixing complex issues such as this, is theoretically unlikely. Ive state this several times, hence I shared all debug logs possible.

FYI Im using always Element web over Firefox with 100% verified sessions which accumulate in profile making endless large lists from same IP/device, always my Desktop, unlike others I dont use apps or android clients, which I never understood, unless different IP, but OK.

So with this in mind I can categorically and emphatically state my experience with my own sessions over encrypted channel has failed 90% of the time and where other participants are involved, equally failed to a very high percentage.

I'm quite happy to share info, if you need more specific anything just ask.

uhoreg commented 2 years ago

FYI Im using always Element web over Firefox with 100% verified sessions which accumulate in profile making endless large lists from same IP/device,

That sounds like you're re-logging-in frequently (e.g. his can happen if you use Element in an incognito tab, or with certain privacy settings enabled, as the browser will not retain the information needed to keep you logged in). If that's the case, then it's expected that encryption will not work for you. This will be fixed by dehydrated devices, which is currently behind a labs flag.

ShadowJonathan commented 2 years ago

Hold on, yeah @the-j0k3r, are you using Element in an incognito tab, and constantly have to log in again and again?

the-j0k3r commented 2 years ago

No incognito tabs. But yes I constantly have to login again and again. I presume its due to my cookie handling of no cross site. Capture

I also have [x] Clear history when Firefox closes enabled.

Firefox is the latest 95, but this has happened always since... 90.x which is when I joined Matrix.

That said, it worked 10% of the time, and oddly enough, [x] Never send encrypted messages to unverified sessions from this session

And people with unverified sessions on the channel where still reading the messages just fine, sometimes until two 3 days later, perhpas to low activity in channel...