Matrix store and olm account is not loaded.

[Setom29]

Hello! I have a problem related to e2ee. I log in using an access token. I followed the instructions and edited the config.json and compose.yaml. I initially ran the Docker image using sudo docker compose without any problems, but after I tried to add e2ee, I found the following error message:

matrix-stt-bot  | 2023-12-04 09:34:41,890 - INFO - Bot started!
matrix-stt-bot  | 2023-12-04 09:34:43,052 - INFO - Login via access_token
matrix-stt-bot  | 2023-12-04 09:34:43,052 - INFO - start import_keys process, this may take a while...
matrix-stt-bot  | Traceback (most recent call last):
matrix-stt-bot  |   File "/app/src/bot.py", line 632, in <module>
matrix-stt-bot  |     asyncio.run(main())
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
matrix-stt-bot  |     return runner.run(main)
matrix-stt-bot  |            ^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
matrix-stt-bot  |     return self._loop.run_until_complete(task)
matrix-stt-bot  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
matrix-stt-bot  |     return future.result()
matrix-stt-bot  |            ^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/app/src/bot.py", line 613, in main
matrix-stt-bot  |     await bot.import_keys()
matrix-stt-bot  |   File "/app/src/bot.py", line 537, in import_keys
matrix-stt-bot  |     resp = await self.client.import_keys(
matrix-stt-bot  |                  ^^^^^^^^^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/site-packages/nio/client/base_client.py", line 115, in inner
matrix-stt-bot  |     raise LocalProtocolError("Matrix store and olm account is not loaded.")
matrix-stt-bot  | nio.exceptions.LocalProtocolError: Matrix store and olm account is not loaded.

[hibobmaster]

Use user_id and password to login instead of access_token.

[Setom29]

Thanks for the quick reply. I tried to log in with user_id and password, but it was unsuccessful because of the server configuration. There is OIDC authorization and i can't use user_id and password.

[hibobmaster]

Create a matrix account from matrix.org or other providers.

[Setom29]

I can't create such accounts because I have self-hosted server with disabled federation. Maybe you can suggest what I can change to make e2ee work using access_token?

[hibobmaster]

Sorry, i use matrix-nio library to develop bot. It has poorly document and i can't find a way to achive that.

[Setom29]

Okay, thank you. I'll continue my search, and I'll let you know if I find the answer.

[hibobmaster]

@Setom29 Please use the latest version and follow the instructions below to make access_token work in E2EE room.

1. Open a private browser window and visit your element web url, then login to your bot account via OIDC.
2. Get current session id and access_token from element web.
3. Close private browser window and open a new one, revisit your element web url and login bot account via OIDC.
4. Send some text in E2EE room from your normal account, if you can read the decrypted content from private browser window(bot account), then export E2EE room keys from element web(bot account).
5. Edit your config.json (without the <>), then docker compose up

   {
   "homeserver": "https://xxxx.xxxxx",
   "user_id": "@xxxxx:xxxx.xxxxx",
   "access_token": "<access_token>",
   "device_id": "<session id>",
   "model_size": "base",
   "room_id": "<some non exist room id>",
   "import_keys_path": "element-keys.txt",
   "import_keys_password": "<E2EE room keys passphrase>"
   }

   Some errors can be ignored.

Wait some seconds to let the bot do some syncing, then Ctrl+C stop the container.

6. Remove import_keys_path and import_keys_password options, and set a correct room_id or remove it if you hope the bot to work in the rooms it is in.

7. Finally, launch the container in detach mode

   docker compose up -d

[alexander-potemkin]

@hibobmaster , thank you. Session id is the thing that changes over time, as I conclude from it's name, which means that my bot is not sustainable without man's intervention, which makes it looks more like a script, not a bot...

As I can see in the docs it is not, indeed:

Megolm sessions may not be reused indefinitely. The parameters which define how often a session should be rotated are defined in the m.room.encryption state event of a room.

[hibobmaster]

@alexander-potemkin You can understand in this way. From element web, you can see how many sessions are there. All of them work unless you manually logout it. (Each session_id has a correspond access_token)

I can confirm the bot is sustainable without man's intervention after you properly set it up. Can you try https://github.com/hibobmaster/matrix-stt-bot/issues/4#issuecomment-1840398688 ?

[alexander-potemkin]

Thank you and I'm sorry to say that it still doesn't work for me... :/ I followed all of steps down to the dot in them, but the only room it works at - it's unencrypted initial room, that is also happened to be specified in the config.

To make sure it's not some room keys issue, I created a new encrypted room, added my bot there, here is what I have on the console:

2023-12-07 21:16:18,569 - INFO - Joined !ycxAopEWqQVtJssZhx:mydomain.com
2023-12-07 21:16:18,714 - INFO - Joined !ycxAopEWqQVtJssZhx:mydomain.com
Olm event doesn't contain ciphertext for our key
Received a undecryptable Megolm event from a device with no Olm sessions: @alex:mydomain.com XFNMXWGKIM
Error decrypting megolm event, no session found with session id tzb+slR8NejmneCYjp29CQKGMeqpyg4CJK5MA5gU9QA for room !ycxAopEWqQVtJssZhx:mydomain.com
2023-12-07 21:16:32,243 - decryption_failure - ERROR - Failed to decrypt message: $Jki6KordOd2Int-cgeVWJYilGn7N0rw9ywep2WedlBE from @alex:mydomain.com             in !ycxAopEWqQVtJssZhx:mydomain.com
Please make sure the bot current session is verified
2023-12-07 21:16:32,243 - ERROR - Failed to decrypt message: $Jki6KordOd2Int-cgeVWJYilGn7N0rw9ywep2WedlBE from @alex:mydomain.com             in !ycxAopEWqQVtJssZhx:mydomain.com
Please make sure the bot current session is verified

[alexander-potemkin]

And yes - bot's session(s) are all verified. Triple checked that from the browser (logged in as a bot) and from 'regular user' browser.

[hibobmaster]

Can you confirm

"Send some text in E2EE room from your normal account, if you can read the decrypted content from private browser window(bot account), then export E2EE room keys from element web(bot account)"

You need two sessions.

A: used for matrix-stt-bot (close the browser window after getting its session_id and access_token)

B: login bot account from element web(do not close the browser before making session A work in E2EE room)

Use your personal account sending some text in E2EE room. Make sure session B in element web can read what you send. If it is ok, you can go on, otherwise logout B session, repeat this step.

Then, export E2EE room keys from element web(session B).

Finally, import E2EE room keys in session A using following config, then spin up container.

{
    "homeserver": "https://xxxx.xxxxx",
    "user_id": "@xxxxx:xxxx.xxxxx",
    "access_token": "<access_token>",
    "device_id": "<session id>",
    "model_size": "base",
    "room_id": "<some non exist room id>",
+  "import_keys_path": "element-keys.txt",
+  "import_keys_password": "<E2EE room keys passphrase>"
}

You may still see some decryption_failure errors for previous messages but newly create message will decrypt success, just send voice message to verify. Then you can close the browser window session B used at this time.

After that, even bot is invited to other rooms, it can work too without man's intervention. (Just make sure session A is the only online session, other sessions are offline)

[alexander-potemkin]

Thank you very much. I followed all of the steps you provided, but I'm still getting the error:

Received a undecryptable Megolm event from a device with no Olm sessions: @alex:mydomain.com XFNMXWGKIM
Error decrypting megolm event, no session found with session id Awrm7TMzGvFPRIqmB1Sx1a1RPW9ENEF9YXdqgYpOArw for room !GnyRxyvAEIRTVnFpJJ:mydomain.com
2023-12-09 12:21:34,712 - decryption_failure - ERROR - Failed to decrypt message: $gTT1hMxBLbmHG3yKD7R-DF28K6yRvA2nqxTM4oPh4vg from @alex:mydomain.com             in !GnyRxyvAEIRTVnFpJJ:mydomain.com
Please make sure the bot current session is verified
2023-12-09 12:21:34,712 - ERROR - Failed to decrypt message: $gTT1hMxBLbmHG3yKD7R-DF28K6yRvA2nqxTM4oPh4vg from @alex:mydomain.com             in !GnyRxyvAEIRTVnFpJJ:mydomain.com
Please make sure the bot current session is verified

I did a few times, resetting the keys, device ids, etc and the issue still persist. Double checked - all my bot sessions are verified.

Is there probably some ways to understand what exactly is missing? What is the session id that is referred in the error message: Awrm7TMzGvFPRIqmB1Sx1a1RPW9ENEF9YXdqgYpOArw?

Or any other help! Would be much appreciated!

[hibobmaster]

@alexander-potemkin Are you using password to login? Don't use password, use access token instead if you follow the instructions above.

As you see:

[alexander-potemkin]

I use login & password for Element client and token for the bot. Guess I can’t login with token on Element, unless I’m missing something?

[hibobmaster]

I mean matrix-stt-bot. You can use either password or access token in config.json but not both.

If you use password, use a device_id different from element web session list and follow the instructions in readme.

If you use access token, use the instructions above.

Notice that some errors can be ignored, make sure always send a voice to test.

[alexander-potemkin]

I use access token, follow instructions and only sending voice messages… 🙁

[hibobmaster]

Can you share me your config.json and compose.yaml with redacted info.

[alexander-potemkin]

I'm running bot.py directly from IDE, to catch errors / exceptions and follow the logic (with all files created just next to the script in src folder).

Please, find information attached: https://gist.github.com/alexander-potemkin/17d9c86778e54372714c88086a96bc18

Please, let me know once you access / download it, so that I can remove it.

[hibobmaster]

I get the same error Olm event doesn't contain ciphertext for our key which is not occured in my personal server or matrix.org server, weird~

[hibobmaster]

After a lot of tests, I got it work finally. Maybe there is a bug or something else. What i do is import_keys that the keys exported from hibobmaster account(Not bot account, i don't know why it's not work here).

Make it briefly,

1. Login bot account in element web, verify it with backup key(if you forget it, reset and create a new one)
2. Get bot account session_id and access_token from element web, then close the browser window
3. Export E2EE room keys from your normal account
4. Edit config.json accordingly and launch the bot and let it import the keys you just exported.

[alexander-potemkin]

Oh, wow! Thanks a lot! I shall test it all very thoroughly and get back to you in a little while!

[alexander-potemkin]

Unfortunately, that didn't work... =( I've tried it on another self-hosted server, but same thing.

One thing - sessions keys dump is getting bigger with every new session / chat in a new room and from the description I can see that they gives access to the past messages, not to the future messages. And Element client generates those sessions keys as required, which makes me think that something is wrong about the way bot handles that data.

I understand that nio documentation is almost absent and Matrix protocol is not so easy to dig into, but it feels like bot shall handle device verification and sessions keys on his own, otherwise something get lost somewhere.

btw, doesn't it feels weird, that with session keys old messages can't be decrypted, despite the fact that those keys are created to enable old messages decrypt?

[hibobmaster]

As for matrix bot, we don't care old messages. What i see is when the bot join the new rooms without man's intervention, it works well at my own server(I forgot to test in your server).

If you want the bot handle device verification, you can use emoji verification. The screenshot use https://mozilla.modular.im as well as my homeserver.

Unfortunately, that didn't work

Can you try logout your current normal account, then relogin, export its E2EE room keys and let bot import again?

The weird thing is i got Olm event doesn't contain ciphertext for our key following my previos instructions in your server but that thing not happens in my synapse matrix server or official matrix server or mozilla matrix server. The problem got rid of after i let bot import E2EE room keys from normal user account(hibobmaster) you provided.

[alexander-potemkin]

@hibobmaster , I appreciate your support and prompt responses - that was really of importance for me.

During the digging into the things I found myself rewriting the functionality with simple-matrix-bot-lib and matrix-commander and I made it finally work.

During my journey I was facing similar issues, until I dropped Element client completely and started using matrix-commander to obtain token - then it was back to normal.

Once again - thank you a lot for your product, time and efforts - I borrowed a few things from there and your support was of really great help!