matrix-org / matrix-rust-sdk

Matrix Client-Server SDK for Rust
Apache License 2.0
1.19k stars 232 forks source link

Race condition when sharing secrets to a new session #2667

Open BillCarsonFr opened 11 months ago

BillCarsonFr commented 11 months ago

I can reproduce this scenario consistently on my local dev setup.

Some how related to this panic detected on web https://github.com/vector-im/element-web/issues/26301

After a verification, the existing session will try to send the secrets to the new one:

This what happens in the racy scenario

There is no olm session to encrypt the secret, so it's added to the wait_queue (would be retried on next key claim)

INFO /Users/valere/.cargo/git/checkouts/matrix-rust-sdk-1f4927f82a3d27bb/bcbb7c6/crates/matrix-sdk-crypto/src/gossiping/machine.rs:295 Secret request is missing an Olm session, putting the request in the wait queue
secret_name = "m.cross_signing.master"

The key/claim is actually made right after (in the middle of the secret sharing process)

rageshake.ts:74 rust-crypto: successfully made HTTP request: POST /_matrix/client/v3/keys/claim
rageshake.ts:74 INFO /Users/valere/.cargo/git/checkouts/matrix-rust-sdk-1f4927f82a3d27bb/bcbb7c6/crates/matrix-sdk-crypto/src/session_manager/sessions.rs:418 Established new Olm sessions

Then after that we can see that the next secret is shared twice:

INFO /Users/valere/.cargo/git/checkouts/matrix-rust-sdk-1f4927f82a3d27bb/bcbb7c6/crates/matrix-sdk-crypto/src/gossiping/machine.rs:285 Sharing a secret with a device
user_id = "@backup_test:localhost:8480"
device_id = "OKYPXQIAZM"
secret_name = "m.megolm_backup.v1"

Then again just after

INFO /Users/valere/.cargo/git/checkouts/matrix-rust-sdk-1f4927f82a3d27bb/bcbb7c6/crates/matrix-sdk-crypto/src/gossiping/machine.rs:285 Sharing a secret with a device
user_id = "@backup_test:localhost:8480"
device_id = "OKYPXQIAZM"
secret_name = "m.megolm_backup.v1"

We have again the log for msk twice:

 INFO /Users/valere/.cargo/git/checkouts/matrix-rust-sdk-1f4927f82a3d27bb/bcbb7c6/crates/matrix-sdk-crypto/src/gossiping/machine.rs:285 Sharing a secret with a device
user_id = "@backup_test:localhost:8480"
device_id = "OKYPXQIAZM"
secret_name = "m.cross_signing.master"

INFO /Users/valere/.cargo/git/checkouts/matrix-rust-sdk-1f4927f82a3d27bb/bcbb7c6/crates/matrix-sdk-crypto/src/gossiping/machine.rs:285 Sharing a secret with a device
user_id = "@backup_test:localhost:8480"
device_id = "OKYPXQIAZM"
secret_name = "m.cross_signing.master"
BillCarsonFr commented 11 months ago

Full logs https://rageshakes.element.io/api/listing/2023-10-04/124959-YQCFXJMA/logs-0000.log.gz