crypto: events can be sent without an up-to-date member list causing UTDs

[kegsay]

I've seen two bug reports from the same user on EXA which shows the following:

• Get invited to a DM over federation
• Accept the invite
• Send some messages
• Those messages are UTD.
• ~45s later send more messages, they are not UTD.

The SDK is supposed to hit /members prior to sending encrypted events to ensure they have an up-to-date member list. This is done in https://github.com/matrix-org/matrix-rust-sdk/blob/0c97c85aea952887b3d7677932b96af492808a42/crates/matrix-sdk/src/room/futures.rs#L166-L168

I see no evidence of a /members hit when sending the event.

However, this is complicated by the several confounding factors:

• the app itself also asks for /members and then immediately cancels the job, causing kotlinx.coroutines.JobCancellationException: Job was cancelled; job=SupervisorJobImpl{Cancelling}@6f96c4a - this is normal behaviour when you back out of a room quickly before it can fully load.
• sync_members will not actually sync the members if the user is invited and history visibility is HistoryVisibility::WorldReadable | HistoryVisibility::Invited
• the SS request doesn't return the member event of the inviter in this scenario because it's before the join event and the client doesn't required_state: [[m.room.member, *]].
• the client is stuck unable to upload OTKs due to https://github.com/matrix-org/matrix-rust-sdk/issues/1415 which is causing /keys/upload to HTTP 400.

These factors mean there's several things which could be going wrong here:

• Can calling Room.members() break the state machine such that it doesn't call it again when you send an event?
• Is there a race condition where you can send an event after you join but before the rust SDK layer believes you are joined, so sync_members doesn't actually sync members?
• The SDK does eventually learn that the inviter is in the room, but how? SS isn't telling the SDK directly, so what mechanism is being used here? /messages? That doesn't make sense as it's historical state.
• Does the inability to upload OTKs cause the state machine to terminate early or something?

There's a lot of moving parts here but I've done due diligence:

• haproxy logs on the server show that /members is only hit one time, just after the /join (this was triggered by EXA)
• haproxy logs show the response size is about right for 2 members to be returned in the JSON, indicating that the server likely did include the inviter in the response.

[poljar]

Is there a race condition where you can send an event after you join but before the rust SDK layer believes you are joined, so sync_members doesn't actually sync members?

I don't think so, due to this check happening at the start of the send: https://github.com/matrix-org/matrix-rust-sdk/blob/abda95979691c4d51d6b7fc06a9777d9e32a795d/crates/matrix-sdk/src/room/futures.rs#L169

Which does this check:

https://github.com/matrix-org/matrix-rust-sdk/blob/abda95979691c4d51d6b7fc06a9777d9e32a795d/crates/matrix-sdk/src/room/mod.rs#L2524-L2529

Later on, the same piece of data is used to skip the /members request:

https://github.com/matrix-org/matrix-rust-sdk/blob/abda95979691c4d51d6b7fc06a9777d9e32a795d/crates/matrix-sdk/src/room/mod.rs#L517-L522

In conclusion, if the SDK doesn't think you are joined, it doesn't let you send a message.

Does the inability to upload OTKs cause the state machine to terminate early or something?

I think this is an unrelated issue, failing to upload OTKs would produce UTDs on the EX side, not on the EW side.

[poljar]

the app itself also asks for /members and then immediately cancels the job, causing kotlinx.coroutines.JobCancellationException: Job was cancelled; job=SupervisorJobImpl{Cancelling}@6f96c4a - this is normal behaviour when you back out of a room quickly before it can fully load.

I think this might be the problem, though the cancellation shouldn't set the members_synced flag to true, so next time you try to send a message it should retry the /members request.

To further expand, we have a request deduplication handler:

https://github.com/matrix-org/matrix-rust-sdk/blob/abda95979691c4d51d6b7fc06a9777d9e32a795d/crates/matrix-sdk/src/deduplicating_handler.rs#L29-L38

The handler allows the first caller to acquire a lock and the lock is held for the duration of the call. The second caller then tries to acquire the same lock, though it will do so only after the first caller is done.

The first caller gets the result of the operation and shares it through the lock with the rest of them. So a failure would be propagated correctly to the rest of the callers, but a cancellation, like EX does here, might not be a failure due to this critical line:

https://github.com/matrix-org/matrix-rust-sdk/blob/abda95979691c4d51d6b7fc06a9777d9e32a795d/crates/matrix-sdk/src/deduplicating_handler.rs#L65-L67

[bnjbvr]

the app itself also asks for /members and then immediately cancels the job, causing kotlinx.coroutines.JobCancellationException: Job was cancelled; job=SupervisorJobImpl{Cancelling}@6f96c4a - this is normal behaviour when you back out of a room quickly before it can fully load.

We also have a deduplicating mechanism for some queries, including /members: if there's already another caller doing this query, then the second query is deduplicated.

Now, I didn't think about cancellability when I implemented that deduplication (one extra proof that making it possible to cancel any future makes reasoning really hard). So what could happen:

• the app calls /members
• we try to send an event, which requires calling /members because we don't know who's in the room
• since there's already a /members query, the one called when sending in the event waits for the former (caused by the app) to finish
• the app cancels the first query
• ???

Ideally, the SDK would "transmit" ownership of the deduplicated query to one of the waiters on the result, but I'm not sure that's what's happening here...

[poljar]

Ideally, the SDK would "transmit" ownership of the deduplicated query to one of the waiters on the result, but I'm not sure that's what's happening here...

I think we assume a successful result, look at my previous reply.

[bnjbvr]

Indeed, I hadn't seen your answer yet while typing mine. Good catch!

[kegsay]

So the failure scenario would be:

• Call room.members() to trigger a /members request
• Whilst this is ongoing, send a message / do something to trigger the member sync logic?
• Cancel the room.members() call - How is this done at an FFI level? There is no cancel function or anything.
• This causes the member sync code to return a success when it shouldn't, which propagates to the send message code?

[poljar]

Cancel the room.members() call - How is this done at an FFI level? There is no cancel function or anything.

Not sure how it's done in this case, as I haven't checked what EXA is doing, but we provide a general cancellation primitive:

https://github.com/matrix-org/matrix-rust-sdk/blob/abda95979691c4d51d6b7fc06a9777d9e32a795d/bindings/matrix-sdk-ffi/src/task_handle.rs#L4-L39

This causes the member sync code to return a success when it shouldn't, which propagates to the send message code?

Yes, correct. The previous steps are correct as well.

[bnjbvr]

Fwiw, I'm working on adding a test showing the deduplicating handler failure (in this case), and making it more robust against this particular cancelling race.

This doesn't mean it will solve this particular issue entirely (since it's unclear whether that was the root cause or not), but at least it'll make the whole deduplication handling more robust.

[poljar]

Fwiw, I'm working on adding a test showing the deduplicating handler failure (in this case), and making it more robust against this particular cancelling race.

Ah nice, thanks.

[kegsay]

Manually confirmed that https://github.com/matrix-org/matrix-rust-sdk/pull/3533 fixes the issue. Whilst EXA still produces kotlinx.coroutines.JobCancellationException: Job was cancelled; the SDK does now retry /members requests.