Calling the deactivation API without rate limiting causes room inconsistencies

ThoreKr commented 1 year ago

Description

When bulk removing users in the IdM (keycloak), at the same time multiple requests will be sent to the /_synapse/admin/v1/deactivate/<user_id> endpoint to deactivate (erase=true) the concerning accounts in synapse.

When not rate limiting these delete requests, the erasure can have the following side effect for a subset of the deactivated users:

user unset their display name
user left the room
user joined the room

These hulls of deactivated accounts then sit in the room as regular members. Manually calling the deactivate endpoint again will remove them from the room again, but they shouldn't be rejoining in the first place.

Steps to reproduce

Have a room with about 1k users (authenticated via OIDC) all auto-joining into a shared public, non-encrypted room
In a for loop without wait times call the deactivate endpoint for 20-30 users
Watch them unsetting their display name and leaving
Some will shortly after rejoin the room

Homeserver

selfhosted

Synapse Version

1.89.0

Installation Method

Docker (matrixdotorg/synapse)

Database

postgresql, single instance

Workers

Single process

Platform

x86 (64 bit) VM

Configuration

Presence is enabled

Message retention is enabled

purge_jobs:
- longest_max_lifetime: 2d
  interval: 1h
- shortest_max_lifetime: 2d
  interval: 1d

Relevant log output

Only the api call to the deactivate endpoint is logged.

Anything else that would be useful to know?

No response

reivilibre commented 1 year ago

possibly related: if we deduplicated deactivations, it might sort out the issues seen here — #16055

erikjohnston commented 1 year ago

I think we can also probably just skip setting peoples display names in rooms when doing the deactivation? That would fix the race and reduce the cost of deactivation.

I think that would be a case of only calling the following on deactivation?

https://github.com/matrix-org/synapse/blob/2763c49eca483dbb848b70b951891afd57016f17/synapse/handlers/profile.py#L190

clokep commented 1 year ago

I think we can also probably just skip setting peoples display names in rooms when doing the deactivation?

Or could we do it as a background task?

matrix-org / synapse