Retriving stored backup ends up with error: OLM.BAD_MESSAGE_MAC

[AbdullahQureshi1080]

This error occurs when I delete my app or remove the app caches mainly the indexedDB data and after that when I try to restore the data this error occurs. With some docs and code this seems to happen when the signature has been invalided and in our case the backup signature. Not sure why this happens. Any sort of help would be appreciated. With this any new messages being received are not being decrypted.

My understanding is that the backup is not device specific but dependent on device Id .

Some common errors

• The sender's device has not sent us the keys for this message.
• The secure channel with the sender was corrupted. Trying to create a new secure channel and re-requesting the keys.

Sharing how I am setting the backup and restoring it.

Setup Backup

  async setAndEnableBackup() {
    const client = await this.getClient();
    const keypharse = 'secure';
    if (!client) {
      return alert('Client does not exist');
    }
    const backup = await this.getBackup();
    if (backup && backup.backupInfo) {
      this.restoreBackup(backup, client, keypharse);
    }
  }
}

Check & Create Backup

async getBackup() {
    const client = await this.getClient();
    let backup = await client.checkKeyBackup();
    console.log('KEY BACKUP - checkKeyBackup', backup);

    if (!backup.backupInfo) {
      console.log('KEY BACKUP - createSecureBackup');
      return this.createSecureBackup(client);
      // throw new Error('Backup broken or not there');
    }
    console.log('KEY BACKUP - retrivedBackup');
    return backup;
  }

Creating Backup

  async createSecureBackup() {
    const client = await this.getClient();
    const keypharse = 'secure';
    // const client = await this.getClient();
    console.log('KEY BACKUP - keypharse', keypharse);
    const prepareKeybackup = await client.prepareKeyBackupVersion(keypharse);
    console.log('KEY BACKUP - prepareKeyBackupVersion', prepareKeybackup);
    const newKeybackup = await client.createKeyBackupVersion(prepareKeybackup);
    console.log('KEY BACKUP - createKeyBackupVersion', newKeybackup);
    return newKeybackup;
  }

Restoring Backup

async restoreBackup(backup, client, passphrase) {
    if (
      backup.backupInfo.auth_data?.private_key_salt &&
      backup.backupInfo.auth_data?.private_key_iterations
    ) {
      const backupWithPassword = await client.restoreKeyBackupWithPassword(
        passphrase,
        undefined,
        undefined,
        backup.backupInfo,
        {},
      );
      console.log(
        'KEY BACKUP - restoreKeyBackupWithPassword',
        backupWithPassword,
      );
    } else {
      console.log('KEY BACKUP - enableKeybackup');
      try {
        await client.enableKeyBackup(backup.backupInfo);
        if (!backup.trustInfo.usable) {
          // this will also set trust usable and local trust true.
          const recoverInfo = await client.restoreKeyBackupWithSecretStorage(
            backup.backupInfo,
            undefined,
            undefined,
          );
          console.log('RECOVER INFO', recoverInfo);
        }
      } catch (e) {
        console.error('Error in restore backup', e);
      }
    }
  }

What have I done to make it work, but it's not the best case help in validating the backup would be awesome

To resolve the error, I create/setup a new backup after the invalided one, it seems to work fine, but what happens is that any new messages after the new backup seems to be the not decrypted on receive but after syncing the keys with the other members of the room it works just fine. So maybe if the backup is restored with a valid signature. this would not be required.

And this happens for some cases, but most times what happens is that the encrypted session gets deleted, while I do not see the warning on my own client, I can see a little shield on element specifying that this message has been sent by a deleted encrypted session. Plus element seems to decrypt the message but my client with the invalidated backup does not even after creating a new backup.

I get this log

When handling UISI from @rohan:localhost (sender key zz/mf0Yoo4mtWjiobhj+UOwrKPvZgW4yErPg+evR9W8): recent session problem with that sender

  async requestRoomKeys(roomId, event) {
    const client = await this.getClient();
    const room = client.getRoom(roomId);
    const userId = this.userId;
    const deviceId = this.deviceId;
    const sessionId = client.getSessionId();
    const senderkey = client.getDeviceCurve25519Key();
    const members = room.getMembers().map(user => ({
      userId: user.userId,
      deviceId: client.getStoredDevicesForUser(user.userId)[0].deviceId,
    }));
    const roomKeyRequest = {
      algorithm: 'm.megolm.v1.aes-sha2',
      room_id: roomId,
      session_id: sessionId,
      sender_key: senderkey,
      recipients: members,
    };

    console.log('Sender Key', roomKeyRequest);

    const res = await client.crypto.requestRoomKey(
      roomKeyRequest,
      members,
      true,
    );
    console.log('RES', res);
  } 

matrix-js-sdk: 23.1.1

[EnglanM]

Did you solve this one?

[AbdullahQureshi1080]

Did you solve this one?

@EnglanM Are you using the sdk in a mobile environment? React Native?

[EnglanM]

No, I'm using it on the web.

[AbdullahQureshi1080]

No, I'm using it on the web.

What exactly are you trying to do? What is the specific error. I had tried to make this work a long time ago. I don't really remember the exact steps at the moment.

If you can share specific details, maybe I can help out.

[EnglanM]

that would be very kind of you. I'm trying to make a group chat inside a 3d platform that my company is developing. The problem is that it takes way too long to do the encryption for each user, and after it has finished it may still not work. I'm sending some snippets of code too. Here I create a client: ` this.client = sdk.createClient({ baseUrl: config.matrixServer, accessToken: authData.access_token, userId: authData.user_id, deviceId: authData.device_id, cryptoStore: new IndexedDBCryptoStore(indexedDB, 'matrix-js-sdk:crypto'), store: new sdk.MemoryStore({ localStorage: window.localStorage }), }); await this.client.initCrypto(); await this.client.startClient({ initialSyncLimit: 1, });

this.client.once(sdk.ClientEvent.Sync, (state: string) => {
  if (state === SyncState.Prepared) {
    const cryptoApi = this.client.getCrypto();
    if (cryptoApi) {
      cryptoApi.globalBlacklistUnverifiedDevices = true;
    }
  }
});`   then i check if we have an decryption error, where we mostly  do: `this.client.on('Event.decrypted' as EmittedEvents, (event: MatrixEvent) => {
  if (event.isDecryptionFailure()) {
    console.log('Decryption failure: ', event);
    return;
  }
  console.log('Event.decrypted %o', event.getContent());
});`   and now just to make sure , i verify each device manually and make them known to the users: `ivate async changeRoom(matrixRoomId: string) {
this.roomId = matrixRoomId;
console.log(
  'getuserid',
  this.client.getUserId(),
  this.client.getDeviceId(),
  this.client.getRoom(matrixRoomId),
  this.client,
);
// await this.client.leave(this.roomId);
await this.client.joinRoom(this.roomId);
const userId = this.client.getUserId() || ''; // Ensure non-null string value
const deviceId = this.client.getDeviceId() || ''; // Ensure non-null string value
await this.client.setDeviceVerified(userId, deviceId, true);
await this.client.sendEvent(this.roomId, 'm.room.message', { body: 'Hello this is a test', msgtype: 'm.text' }, '');
// process.exit(0);
// (await this.client.joinRoom(this.roomId))
// .getMembers().forEach((member) => {
//   const deviced: string = member.userId.replace('@robot-', '').replace(':staging.zorabots.be', '');
//   const device = this.client.getStoredDevice(member.userId, deviced);
//   console.log('member&deviceBeforeVerification', member.userId, device, device?.isVerified());
//   if (!device) return;
//   if (!device.isVerified()) {
//     this.client
//       .setDeviceVerified(member.userId, deviced, true)
//       .then(() => {
//         console.log('setDeviceVerified', member.userId, deviced, true);
//       })
//       .catch(console.error);
//   }
//   if (!device.known) {
//     this.client
//       .setDeviceKnown(member.userId, deviced, true)
//       .then(() => {
//         console.log('setDeviceKnown', member.userId, deviced, true);
//       })
//       .catch(console.error);
//   }

// device.known = true;
// device.verified = 1;
// });`. but it still has the issue that i mentioned at the beginning of this comment. I test this by logging in to different accounts, entering the same chat room, and texting each other.  After the encryption process end (which takes a couple of minutes, and i don't have that many users) some of the accounts can read the messages and some not. The console of the accounts which work but cant read the messages of the account which do not work: `[!NOysHSHeqAZcwACUER:staging.zorabots.be decryption] When handling UISI from @robot-virtual-robot-867d99fa-b2c3-4a0c-9b02-fb3367f30ef7-1690877537:staging.zorabots.be (sender key v98BFU2Ic8HcCZE/ukcW2WSSSRMCIMnnpxkbb/lGeEQ): recent session problem with that sender: {
"deviceKey": "v98BFU2Ic8HcCZE/ukcW2WSSSRMCIMnnpxkbb/lGeEQ",
"type": "wedged",
"fixed": false,
"time": 1707308037289

}the console of the account who do not work:logger.js:49 Error decrypting event (id=$whVNk8mDPz7Ha_NRiSzG-5nks8NHvWRLIXdh04HJbLI type=m.room.encrypted sender=@robot-virtual-robot-4cfc0680-7301-47a4-aa6c-f36a9702e3ce-1700839857:staging.zorabots.be room=!NOysHSHeqAZcwACUER:staging.zorabots.be ts=2024-02-07T13:24:16.491Z): DecryptionError[msg: The sender's device has not sent us the keys for this message., session: bKinWZ+h+TJs3x+vKE7q1XsM3RLLaKtVhnGVrY77jy0|G8rc9wXpE5/2xQKACtHeeyH381kfVO1jyt6gI/GcSa4]`{ "decrypted": { "type": "m.room.message", "sender": "@robot-virtual-robot-4cfc0680-7301-47a4-aa6c-f36a9702e3ce-1700839857:staging.zorabots.be", "content": { "msgtype": "m.bad.encrypted", "body": "** Unable to decrypt: DecryptionError: The sender's device has not sent us the keys for this message. **" }, "origin_server_ts": 1707312256491, "unsigned": { "age": 67 }, "event_id": "$whVNk8mDPz7Ha_NRiSzG-5nks8NHvWRLIXdh04HJbLI", "room_id": "!NOysHSHeqAZcwACUER:staging.zorabots.be" }, "encrypted": { "type": "m.room.encrypted", "sender": "@robot-virtual-robot-4cfc0680-7301-47a4-aa6c-f36a9702e3ce-1700839857:staging.zorabots.be", "content": { "algorithm": "m.megolm.v1.aes-sha2", "sender_key": "bKinWZ+h+TJs3x+vKE7q1XsM3RLLaKtVhnGVrY77jy0", "ciphertext": "AwgGEoABm4FDNmn//AxCCc0qelair10sONIUeJ1Iiht1tJuQGN1vBKB5lHs9bddGLPHuJj8jXTHTcHCZhuQBDnuPyfsSUeLP+AlQiheQB6qGHygq59fBsh+qMR6goP8o9b2aDNAaCnVTi3dI32Czy1vcCwFnnGxIo7atLQqtoMAJnjGkdwZI4BTncg4A7f8xiJD5/zRTul0AsY0rDHymL7Tls8SVwkFN+7dNiUq7sIjrHkL00uuAQB50lkW9+gsPtUbhX/3iSAqLL8TxXgo", "session_id": "G8rc9wXpE5/2xQKACtHeeyH381kfVO1jyt6gI/GcSa4", "device_id": "virtual-robot-4cfc0680-7301-47a4-aa6c-f36a9702e3ce-1700839857" }, "origin_server_ts": 1707312256491, "unsigned": { "age": 67 }, "event_id": "$whVNk8mDPz7Ha_NRiSzG-5nks8NHvWRLIXdh04HJbLI", "room_id": "!NOysHSHeqAZcwACUER:staging.zorabots.be" } } also this one: ** Unable to decrypt: DecryptionError: The sender's device has not sent us the keys for this message. **

[EnglanM]

@AbdullahQureshi1080 I hope I was clear, it's no problem if you don't want to deal with this headache. I greatly appreciate it!

[AbdullahQureshi1080]

@AbdullahQureshi1080 I hope I was clear, it's no problem if you don't want to deal with this headache. I greatly appreciate it!

@EnglanM Ah man, this feels like going again in the matrix black hole. And it's been a year since I transitioned away from active development. So I am not sure if I can help you that much.

But here's a very detailed doc I created for my efforts on making e2e work on a mobile environment. Unfortunately, I couldn't and we found a work-around by creating our own encryption layer on top of the matrix base service.

But an insight here is that, I made it work on a debug environment of mobile environment. That emulated the things that we needed to make e2e work. Things such as indexedDB, local storage and crypto storage. So in the web it would be comparatively easier. Referred in the doc, I went to level 2.

Here's a very messy, but a complete service that I made.

This the crypto file, I use the verification methods to cross sign the devices for the user.

This is the complete project, you can run it and see how things work. React native environment setup required. Just connect it with the synapase server, local or hosted.

I hope all this helps. Best of luck on this!

And if you make it work, I'd love to know about it.

[EnglanM]

@AbdullahQureshi1080 Wow that is very very helpful, very kind of you man. Are you saying that instead of oml for encryption, you made your own systems? I'll take a look at all the docs you sent me and let you know if I or my team solve this. I am very grateful, may god bless you!

[AbdullahQureshi1080]

@AbdullahQureshi1080 Wow that is very very helpful, very kind of you man. Are you saying that instead of oml for encryption, you made your own systems? I'll take a look at all the docs you sent me and let you know if I or my team solve this. I am very grateful, may god bless you!

No worries, I too had this problem loom over my head for a while so I completely understand.

Yes we did, unfortunately that isn't in the repo, it was a work effort and part of the company project. I can't share that code. Though I can discuss how we made it work.

The above repo is purely based on how the matrix intends e2e to work. And only includes the implementation based on how they've shared.

[EnglanM]

I understand, thank you and I'll let you know if I get anywhere with this.

[EnglanM]

@AbdullahQureshi1080 I hope you are doing well man! I think I found the problem, I see some cases where e2e encryption worked and some when it did not, and in the cases where e2e encryption worked, inside the indexedDB, in the inbound_group_sessions, there were the senderCurve25519Key and sessionId of both devices, which are used from message decryption later. In the cases where the encryption didn't work, at least one of the users didn't have the senderCurve25519Key and sessionId of the other users for whatever reason. This has to be an internal problem of the library since it sometimes works and sometimes does not. Not I know the problem but how to solve it, I can't find a method that resends sessionId's to all users or something similar. DO you have any suggestions since your experience is much much greater than mine? If not, no problem I just wanna hear your opinion on this.

[AbdullahQureshi1080]

@EnglanM Not sure, how I can help you here. Without the code, it would be really difficult.

As far as I remember, the SDK under the hood, requests for missing keys. You can listen to those events and set up keys for messages that weren't decrypted. It's a flow that needs to be completed. And it does. But for the current message, we need to run decryption again. You can do that by syncing the room message again.

You can look at these from their detailed spec sheet. It also explains the API sequence as well.

m.room_key_request m.key.verification.request

Also, can you share how you initialize the client and the crypto layer?

And what do you mean by "some cases" is it in the same session or different sessions? Same-user devices or multiple-user different devices. Are you crossing signing the same user devices?

Is your backup created and stored successfully and is validated when you retrieve it?

[EnglanM]

@AbdullahQureshi1080 bro was the version for the web that you had done fully working, im implementing forwarded_room_key event the same way as you but i doesnt seem to work

[AbdullahQureshi1080]

@EnglanM I am not sure now. I think it was either v21.3.0 or v23.1.1

[EnglanM]

@AbdullahQureshi1080 sorry for not posting lately but i just dont want to give false/wrong hints on where the problem can be. Untill now ive come into conclusion that its the fault of the library, because the encryption/creation of one time keys is not done correctly,as different users have different one-time-keys for the same device, where its supposed to have the same one time key. At least this is the problem that we are facing in our case. Ecerything else such as key requests and key forwards come consequently because the sending/receiving device is not verified.Anyway Ill let you know if we finish this and get it to work. Wish you alll the best my friend!

[AbdullahQureshi1080]

@AbdullahQureshi1080 sorry for not posting lately but i just dont want to give false/wrong hints on where the problem can be. Untill now ive come into conclusion that its the fault of the library, because the encryption/creation of one time keys is not done correctly,as different users have different one-time-keys for the same device, where its supposed to have the same one time key. At least this is the problem that we are facing in our case. Ecerything else such as key requests and key forwards come consequently because the sending/receiving device is not verified.Anyway Ill let you know if we finish this and get it to work. Wish you alll the best my friend!

I understand that, it's all good. I think you meant the same user has different cross signing keys for the different devices?

I think you might have to look at creating backups with the cross signing keys with specifying device ids. You can look at the doc below how verification for the same user but different devices works. I hope that helps.

https://matrix.org/docs/older/e2ee-cross-signing/