How to enable encryption?

[select]

I am trying to enable encryption in my client and already found out that the example the readme provides is not complete. What i did was the following: In package.json add the following package

"olm": "https://matrix.org/packages/npm/olm/olm-2.3.0.tgz",

In webpack in the module section I disabled the parsing so I would not get the errors about the missing node modules

module: {
        noParse: [/olm[\\\/](javascript[\\\/])?olm\.js$/],
                ...
}

To start the server with encryption I have the following code

import 'olm/olm';
import Matrix from 'matrix-js-sdk';
const webStorageSessionStore = new Matrix.WebStorageSessionStore(window.localStorage);
const client = Matrix.createClient({
  ...credentials,
  sessionStore: webStorageSessionStore,
  baseUrl: 'https://matrix.org',
  timelineSupport: true,
});

client.initCrypto();
client.startClient({ initialSyncLimit: 4 });

But when I try to send a message to an encrypted room I get an error

Error sending event Error: Room was previously configured to use encryption, but is no longer. Perhaps the homeserver is hiding the configuration event.

In addition I do not recieve messages and see the following console output

already have key request outstanding for !PYxxEmesBhVjIulDoL:matrix.org / I+aEaIxxS8mgy5xRHeAgM0T0+fkGhNKfmImm+/AwGSY: not sending another

What do I need to do next? I have the feeling I have to find out how to listen to key request and then confirm some keys?

[jaller94]

I have similar problems, so let me try to help you and text back, if you got it working.

1. For the most recent version of the SDK, update to olm-3.0.0.tgz.
2. Before importing the SDK you should set global.Olm to the output of require('olm'). I am not sure how to do this with import.
3. initCrypto() returns a Promise and you should most likely wait for it, before calling startClient.
4. The event m.room.encryption might have gotten redacted. I am not sure that the SDK can deal with this yet.

[select]

Great news, I was waiting for some help for a long time, thanks already so much, I will try it on the weekend!

[t3chguy]

@jaller94

I am not sure how to do this with import.

import olm
global.Olm = olm

depending on your build system you may need import * as olm from olm or similar instead

[select]

So I have tried it again and the encryption seems to be initialized ok but the messages arrive unencryped. Here is what I did. Anybody up for a helping hand?

#!/usr/bin/env node
const sdk = require('matrix-js-sdk');
global.Olm = require('olm');
const { LocalStorage } = require('node-localstorage');
const localStorage = new LocalStorage('./scratch');

try {
  const homeserver = 'matrix.org';
  const channel = '!XXXXXXXXXXX:matrix.org';
  const token = 'XXXXXXXXXXXXX';
  const message = 'hello world e2e';
  const messagetype = 'm.text';
  const webStorageSessionStore = new sdk.WebStorageSessionStore(localStorage);
  const client = sdk.createClient({
    baseUrl: `https://${homeserver}`,
    accessToken: token,
    sessionStore: webStorageSessionStore,
    userId: '@XXXXXXX:matrix.org',
    deviceId: 'XXXXXXX',
  });
  client
    .initCrypto()
    .then(() => client.joinRoom(channel))
    .then(() =>
      client.sendEvent(
        channel,
        'm.room.message',
        {
          msgtype: messagetype,
          format: 'org.matrix.custom.html',
          body: message,
          formatted_body: message,
        },
        ''
      )
    )
    .then(() => {
      console.log('message send!');
    })
    .catch((err) => {
      console.error('err', err);
    });
} catch (error) {
  console.error('error', error);
}

[t3chguy]

If you don't call & await startClient (and the first sync) then the client will have no way of knowing that the room (channel) is even encrypted as it comes down /sync

[select]

Thanks @t3chguy I tried the following but the message does not arrive

#!/usr/bin/env node
const sdk = require('matrix-js-sdk');
global.Olm = require('olm');
const { LocalStorage } = require('node-localstorage');
const localStorage = new LocalStorage('./scratch');

try {
  const homeserver = 'matrix.org';
  const channel = '!XXXXXXXXXXX:matrix.org';
  const token = 'XXXXXXXXXXXXX';
  const message = 'hello world e2e';
  const messagetype = 'm.text';
  const webStorageSessionStore = new sdk.WebStorageSessionStore(localStorage);
  const client = sdk.createClient({
    baseUrl: `https://${homeserver}`,
    accessToken: token,
    sessionStore: webStorageSessionStore,
    userId: '@XXXXXXXXXX:matrix.org',
    deviceId: 'XXXXXXXXX',
  });
  client
    .initCrypto()
    .then(() => client.startClient({ initialSyncLimit: 1 }))
    .then(() => client.joinRoom(channel))
    .then(() => {
      console.log('joined room!');
    })
    .catch((err) => {
      console.error('err', err);
    });
  client.on('sync', function (state, prevState, res) {
    console.log('state', state);
    if (state === 'PREPARED') {
      client.sendEvent(
        channel,
        'm.room.message',
        {
          msgtype: messagetype,
          format: 'org.matrix.custom.html',
          body: message+' '+state,
          formatted_body: message+' '+state,
        },
        ''
      ).then(() => {
        console.log('message was send')
      });
    } else {
      console.log(state);
      process.exit(1);
    }
  });
} catch (error) {
  console.error('error', error);
}

I get the following messages on the shell

sendEvent of type m.room.message in !XXXXXXXXX:matrix.org with txnId m1608044861132.0
setting pendingEvent status to encrypting in !XXXXXXXXX:matrix.org event ID ~!XXXXXXXXX:matrix.org:m1608044861132.0 -> undefined

[select]

Ok I tried the following, still no success

#!/usr/bin/env node
const sdk = require('matrix-js-sdk');
global.Olm = require('olm');
const { LocalStorage } = require('node-localstorage');
const localStorage = new LocalStorage('./scratch');
const {
  LocalStorageCryptoStore,
} = require('matrix-js-sdk/lib/crypto/store/localStorage-crypto-store');

try {
  const homeserver = 'matrix.org';
  const channel = '!XXXXXXXXXXXX:matrix.org';
  const token = 'XXXXXXXXXXXX'; 
  const deviceId = 'XXXXXXXXXXXX';
  const message = 'hello world e2eeeee'; 
  const messagetype = 'm.text'; 
  const webStorageSessionStore = new sdk.WebStorageSessionStore(localStorage);
  const cstore = new LocalStorageCryptoStore(localStorage);
  sdk.setCryptoStoreFactory(() => cstore);
  const client = sdk.createClient({
    baseUrl: `https://${homeserver}`,
    accessToken: token,
    sessionStore: webStorageSessionStore,
    cryptoStore: cstore,
    userId: '@XXXXXXXXXXXX:matrix.org',
    deviceId,
  });
  client.on('sync', function (state, prevState, res) {
    console.log('###########state', state);
    if (state === 'PREPARED') {
      client
        .joinRoom(channel)
        .then(() => client.setRoomEncryption(channel, { algorithm: 'm.megolm.v1.aes-sha2' }))
        .then(() => {
          console.log('seeeeend now');
          return client.sendEvent(
            channel,
            'm.room.message',
            {
              msgtype: messagetype,
              format: 'org.matrix.custom.html',
              body: message,
              formatted_body: message,
            },
            ''
          );
        })
        .then(() => {
          console.log('message was send');
        })
        .catch((err) => {});
    } else {
      console.log('exit ', state);
      process.exit(1);
    }
  });
  client
    .initCrypto()
    .then(() => client.startClient({ initialSyncLimit: 1 }))
    .catch((err) => {
      console.error('err', err);
    });
} catch (error) {
  console.error('error', error);
}

[select]

I think i'm a bit closer now. The problem was process.exit(1); on sync events which quit the process to early. The next problem I'm facing is

Error sending event UnknownDeviceError: This room contains unknown devices which have not been verified. We strongly recommend you verify them before continuing.

Is there a way to tell the client to ignore that

[select]

YESSS I finally figured it out. This is a working minimal example of how to send an encrypted message with the js-sdk. It runs on node and you will need to install the following dependencies (choose your olm version here):

 npm i matrix-js-sdk https://packages.matrix.org/npm/olm/olm-X.X.X.tgz node-localstorage 

#!/usr/bin/env node
const sdk = require('matrix-js-sdk');
global.Olm = require('olm');
const { LocalStorage } = require('node-localstorage');
const localStorage = new LocalStorage('./scratch');
const {
  LocalStorageCryptoStore,
} = require('matrix-js-sdk/lib/crypto/store/localStorage-crypto-store');

const channel = '!XXXXXXXXXXXX:matrix.org';
const message = 'hello world e2e encrypted';
const client = sdk.createClient({
  baseUrl: `https://matrix.org`,
  accessToken: 'XXXXXXXXXXXX',
  sessionStore: new sdk.WebStorageSessionStore(localStorage),
  cryptoStore: new LocalStorageCryptoStore(localStorage),
  userId: '@XXXXXXXXXXXX:matrix.org',
  deviceId: 'XXXXXXXXXXXX',
});

client.on('sync', function (state) {
  if (state !== 'PREPARED') return;
  client.setGlobalErrorOnUnknownDevices(false);
  client
    .joinRoom(channel)
    .then(() =>
      client.sendEvent(
        channel,
        'm.room.message',
        {
          msgtype: 'm.text',
          format: 'org.matrix.custom.html',
          body: message,
          formatted_body: message,
        },
        ''
      )
    )
    .then(() => {
      process.exit(0);
    })
    .catch((err) => {
      console.error('err', err);
    });
});
client
  .initCrypto()
  .then(() => client.startClient({ initialSyncLimit: 1 }))
  .catch((err) => {
    console.error('err', err);
  });

... only took me 2 years :P

[select]

ES5 version

#!/usr/bin/env node
const sdk = require('matrix-js-sdk');
global.Olm = require('olm');
const { LocalStorage } = require('node-localstorage');
const localStorage = new LocalStorage('./scratch');
const {
  LocalStorageCryptoStore,
} = require('matrix-js-sdk/lib/crypto/store/localStorage-crypto-store');

const channel = '!XXXXXXXXXXXX:matrix.org';
const message = 'hello world e2e encrypted';

const client = sdk.createClient({
  baseUrl: `https://matrix.org`,
  accessToken: 'XXXXXXXXXXXX',
  sessionStore: new sdk.WebStorageSessionStore(localStorage),
  cryptoStore: new LocalStorageCryptoStore(localStorage),
  userId: '@XXXXXXXXXXXX:matrix.org',
  deviceId: 'XXXXXXXXXXXX',
});

client.on('sync', async function (state, prevState, res) {
  if (state !== 'PREPARED') return;
  client.setGlobalErrorOnUnknownDevices(false);
  await client.joinRoom(channel);
  await client.sendEvent(
    channel,
    'm.room.message',
    {
      msgtype: messagetype,
      format: 'org.matrix.custom.html',
      body: message,
      formatted_body: message,
    },
    ''
  );
  process.exit(0);
});
async function run() {
  await client.initCrypto();
  await client.startClient({ initialSyncLimit: 1 });
}

run().catch((error) => console.error(error));

[select]

To read encrypted messages I found this to be working

client.on('Room.timeline', async (message, room) => {
    if (!isReady) return;
    let body = '';
    try {
        if (message.event.type === 'm.room.encrypted') {
            const event = await client._crypto.decryptEvent(message);
            ({ body } = event.clearEvent.content);
        } else {
            ({ body } = message.event.content);
        }
        if (body) {
            // do something
        }
    } catch (error) {
        console.error('#### ', error);
    }
});

[FlyveHest]

Hi select, and thank you for posting your findings, strange that this is the only really useful resource on getting encryption to work in the Matrix SDK that i've been able to find.

A quick question, i've done what you suggested, but I am getting this error in an encrypted room, did this happen to you, and if so, how did you handle it?

DecryptionError[msg: The sender's device has not sent us the keys for this message. ...

It would seem like I need to do something more than just enabling encryption, to actually be able to decrypt messages.

[vulet]

I recently encountered this useful issue(#731) while looking for how to implement OLM in an application for a user living on a homeserver that requires e2ee. The logging becomes really detailed after initCrypto(), so I'll still see that DecryptionError on startup, but it sorts itself out.

There's a particular line in one of the code blocks above, which I found really useful while testing:

client.setGlobalErrorOnUnknownDevices(false)

I think this might also be what the Element client uses, basically w/o it, OLM will throw if there's unverified sessions in the room.

[select]

@FlyveHest

DecryptionError[msg: The sender's device has not sent us the keys for this message. ...

did you wait until the client is ready? if (state == 'PREPARED') isReady = true;?

[FlyveHest]

@select Yes, communication in non-encrypted rooms work fine.

I've been trying for a couple hours today, and have simply not been able to get anything to work (using sdk v9.7.0), so i've put encryption to the side for a bit, and I think that I will use pantalaimon for encryption if the need arises in the future.

To me, it doesn't seem like the SDK is really ready to handle encryption fully just yet.

[select]

@FlyveHest have a look here, I have just published a working version of a bot that can read and send e2e messages https://gitlab.com/s3lect/taavi-bot It uses the sdk v9.7.0

[FlyveHest]

@select Interesting, the part of your code that implements the Matrix connectivity is more or less the exact same I had used (not a big surprise, as I used the code you posted above :), but, you have included the olm.js as a file directly in your project, and I added it as an NPM package.

I'll try and clone your repo and see if I can get it to work on my server, it might not be related to the code, but maybe that I have to import / identify keys for my BOT account.

Edit: Tried it, and it works, sort of, it can't verify a session when I log into Element with the same account, but encrypting the channel does work and it can read the messages posted after the channel has been encrypted.

Thanks a lot for the link, i'll try and take a look and see where what I did differ from your implementation.

[select]

@FlyveHest what I found was that olm is not distributed with npm anymore, I could not find a reason though.

• https://www.npmjs.com/package/olm This package has been deprecated
• https://packages.matrix.org/npm/olm/

[zhaytee]

@FlyveHest I was also able to (eventually) get e2e working, although there definitely were some peculiarities and hoops to jump through. Feel free to take a look at my implementation here: https://github.com/zhaytee/matrix-rpc-bot

[FlyveHest]

@zhaytee @select Thanks you both, I got e2e working in my BOT implementation as well, the key point I was missing to begin with was that I did not save the device ID from the initial login, apparantly, when I did that, everything seemed to click.

Its hard to say whats best practice here, but I like the way @zhaytee handles the decryption a bit better than manually decrypting it, but that process is not described anywhere in the docs (at least, none that I could find), so kudos for the digging work there, it helped a lot. (And, on a sidenote, what I am aiming at is not that different from your project, its just not using gRPC)

[Myzel394]

@FlyveHest Can you share your implementation please? I'm facing the same issue. Honestly, I don't know what device ID is. Do you know what it is? Thanks

[select]

@Myzel394 you get a device id when you create the login credentials.

getCredentialsWithPassword(username, password) {
        return new Promise(resolve => {
            Matrix.createClient('https://matrix.org')
                .loginWithPassword(username, password)
                .then(credentials => {
                    resolve({
                        accessToken: credentials.access_token,
                        userId: credentials.user_id,
                        deviceId: credentials.device_id,
                    });
                });
        });
    },

[skylord123]

How do I decrypt a received file? I got the message decrypted and the mxc url converted but when the file is downloaded it is clearly encypted. I see the content.file.key contains key information which I assume is for decrypting the contents.

[skylord123]

I wanted to come back and leave this for anyone else that wants to encrypt/decrypt files. There is a repo that has an example on how to do just this: https://github.com/matrix-org/browser-encrypt-attachment/blob/master/index.js

I had to modify it a little bit to get it to work in my project but it pointed me in the right direction.

[trancee]

@zhaytee @select Thanks you both, I got e2e working in my BOT implementation as well, the key point I was missing to begin with was that I did not save the device ID from the initial login, apparantly, when I did that, everything seemed to click.

Its hard to say whats best practice here, but I like the way @zhaytee handles the decryption a bit better than manually decrypting it, but that process is not described anywhere in the docs (at least, none that I could find), so kudos for the digging work there, it helped a lot. (And, on a sidenote, what I am aiming at is not that different from your project, its just not using gRPC)

@zhaytee @FlyveHest any chance to share the code of how it should be done the right way? unfortunately, @zhaytee's repository does no longer exist and I would like to see how he has done it... any help would be appreciated!

[Electrofenster]

@trancee

Instead of doing:

matrixClient.crypto.decryptEvent(ev).then(encryptedEvent => {
  const { type, content } = encryptedEvent.clearEvent;

  console.error('----->', { type, content });
});

it should be done this way:

await matricClient.decryptEventIfNeeded(ev);
console.error('--->', ev.getContent());

[AbdullahQureshi1080]

@trancee Hey, how's it going? Did you manage to get how to send and receive encrypted files?

[Electrofenster]

@AbdullahQureshi1080 I got sending and receiving encrypted files working in encrypted rooms. Backups, keysharing, multi device verification and all other works too.

[AbdullahQureshi1080]

@Electrofenster That's awesome. I have been working on making the whole encryption system work with the js sdk using react native clients.

I have had some success but there are issues with backups.

Currently working on the sending/receiving encrypted attachments.

Is there a example project that you can share?

Also would be awesome if you can help with these issues.

https://github.com/matrix-org/matrix-js-sdk/issues/3140

https://github.com/matrix-org/matrix-js-sdk/issues/3157

https://github.com/matrix-org/matrix-js-sdk/issues/2506

[Electrofenster]

@AbdullahQureshi1080 do you have a SessionStorage? If no, you may need to implement an own SessionStorage to handle the required actions (see matrix-js-sdk). I recommended react-native-mmkv for storing data, it's very fast and stores the data betweeen restarts which is required. With a right setup I think you can fix your most problems.

[AbdullahQureshi1080]

@Electrofenster

Well I use the session store that matrix creates itself and it works fine up until I delete the stores or delete the app and then when I try to restore the backups it does get the backup but invalidates the signature. Not sure what happens over there.

[Electrofenster]

@AbdullahQureshi1080 well, that's a problem. As I remember, the store from matrix uses for crypto store and session store the indexeddb-backend which is not available in react-native. The session store and crypto store are required for correct key storing/key restoring and checking/verificating the signatures. So if it tries to use indexeddb it could not finish the actions so the data for keys are not correct saved in matrix.

[AbdullahQureshi1080]

@AbdullahQureshi1080 well, that's a problem. As I remember, the store from matrix uses for crypto store and session store the indexeddb-backend which is not available in react-native. The session store and crypto store are required for correct key storing/key restoring and checking/verificating the signatures. So if it tries to use indexeddb it could not finish the actions so the data for keys are not correct saved in matrix.

Hmm, I see. But how to setup the crypto and session store? I tried to using a async crypto store that used react-native-async-storage but it does not work.

Can you help me out in setting both the stores in client initialization?

As you suggested if you can let me know how to initialize the stores properly with the right methods that exists I can use react-native-mmkv.

[Electrofenster]

@AbdullahQureshi1080 as the react-native-async-storage does not work properly as crpyto-, and session store you'll get a lot of errors when using it. Since there is nothing documented it's very hard to use the right one.

I can't provide some code but you can create one like this:

import { SessionStore } from 'matrix-js-sdk/src/client';

class MySessionStore implements SessionStore {}

now the IDE should suggest you the missing functions you could create. Now you can log in each function the parameter. Just look the indexeddb-storages to understand the logic which you need to implement like your own.

The same with the more important crypto store:

import { CryptoStore } from 'matrix-js-sdk/src/crypto/store/base';

class MyCryptoStore implements CryptoStore {
    public storeEndToEndRoom(roomId: string, roomInfo: IRoomEncryption, txn: any): void {
        console.log({ roomId, roomInfo, txn })
    }
}

[AbdullahQureshi1080]

@AbdullahQureshi1080 as the react-native-async-storage does not work properly as crpyto-, and session store you'll get a lot of errors when using it. Since there is nothing documented it's very hard to use the right one.

I can't provide some code but you can create one like this:

import { SessionStore } from 'matrix-js-sdk/src/client';

class MySessionStore implements SessionStore {}

now the IDE should suggest you the missing functions you could create. Now you can log in each function the parameter. Just look the indexeddb-storages to understand the logic which you need to implement like your own.

Will have a go at it.

Same with the crypto store? Yes?

[Electrofenster]

@AbdullahQureshi1080 see my edited answer. Same with crypto store. You'll also need a custom transaction class for crypto store to work. Which "emulates" IDBTransaction. It's not as easy as I say. I needed a lot of time to make it working.

[AbdullahQureshi1080]

@AbdullahQureshi1080 see my edited answer. Same with crypto store. You'll also need a custom transaction class for crypto store to work. Which "emulates" IDBTransaction. It's not as easy as I say. I needed a lot of time to make it working.

I can understand, been working with matrix for a while now and e2e has a lot of kicks to it that can take time to make things work.

Thanks for the pointers 👍🏻

[AbdullahQureshi1080]

@Electrofenster Hey, I looked into the stores. The session store does not exists in the latest SDK version v23.3.0 instead there is a memory-store/local-storage that extends for both storage store and crypto store.

I am using this Async Crypto Store as base and added missing methods, updated store is this.

cryptoStore: new AsyncCryptoStore(AsyncStorage),

There were still issues with the backup that did not arise earlier, particularly with the backup manager after this new initialization.

The backup manager is not being able to upload pending keys for a few sessions. Further debugging led that there is session data missing in a lot of sessions in the crypto stores.

[Doumor]

YESSS I finally figured it out. This is a working minimal example of how to send an encrypted message with the js-sdk. It runs on node and you will need to install the following dependencies (choose your olm version here):

 npm i matrix-js-sdk https://packages.matrix.org/npm/olm/olm-X.X.X.tgz node-localstorage 

#!/usr/bin/env node
const sdk = require('matrix-js-sdk');
global.Olm = require('olm');
const { LocalStorage } = require('node-localstorage');
const localStorage = new LocalStorage('./scratch');
const {
  LocalStorageCryptoStore,
} = require('matrix-js-sdk/lib/crypto/store/localStorage-crypto-store');

const channel = '!XXXXXXXXXXXX:matrix.org';
const message = 'hello world e2e encrypted';
const client = sdk.createClient({
  baseUrl: `https://matrix.org`,
  accessToken: 'XXXXXXXXXXXX',
  sessionStore: new sdk.WebStorageSessionStore(localStorage),
  cryptoStore: new LocalStorageCryptoStore(localStorage),
  userId: '@XXXXXXXXXXXX:matrix.org',
  deviceId: 'XXXXXXXXXXXX',
});

client.on('sync', function (state) {
  if (state !== 'PREPARED') return;
  client.setGlobalErrorOnUnknownDevices(false);
  client
    .joinRoom(channel)
    .then(() =>
      client.sendEvent(
        channel,
        'm.room.message',
        {
          msgtype: 'm.text',
          format: 'org.matrix.custom.html',
          body: message,
          formatted_body: message,
        },
        ''
      )
    )
    .then(() => {
      process.exit(0);
    })
    .catch((err) => {
      console.error('err', err);
    });
});
client
  .initCrypto()
  .then(() => client.startClient({ initialSyncLimit: 1 }))
  .catch((err) => {
    console.error('err', err);
  });

... only took me 2 years :P

And... And it doesn't work now.

TypeError: sdk.WebStorageSessionStore is not a constructor

If delete sessionStore:

Waiting for saved sync before starting sync processing...
/sync error DOMException [AbortError]: This operation was aborted
    at Object.fetch (node:internal/deps/undici/undici:11457:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MatrixHttpApi.requestOtherUrl (/root/sclp-mtx/node_modules/matrix-js-sdk/lib/http-api/fetch.js:223:13)
Number of consecutive failed sync requests: 1