matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

Yet another case of state reset in State Resolution v2 #16629

Open wj25czxj47bu6q opened 11 months ago

wj25czxj47bu6q commented 11 months ago

Description

I have encountered yet another room state reset.

I suspect that the reset of m.room.join_rules was the root cause of the 51 m.room.member resets, as the latter would now be rejected events. There is no obvious explanation for the m.room.join_rules reset.

No state events were sent or redacted in the time immediately preceding the state reset. The last state event before the reset was sent approximately 2 hours prior.

Steps to reproduce

  1. Use the room as normal.
  2. Observe 51 members spontaneously dropped from the room.

Homeserver

arcticfoxes.net

Synapse Version

1.96.0rc1

Installation Method

Debian packages from packages.matrix.org

Database

PostgreSQL 15.4, single server, not ported, restored a very long time ago

Workers

Single process

Platform

Configuration

Relevant log output

SELECT * FROM current_state_delta_stream WHERE room_id='!frWBWOPyriepLexnsA:arcticfoxes.net' AND stream_id=25794806; ``` stream_id | room_id | type | state_key | event_id | prev_event_id | instance_name -----------+-------------------------------------+-------------------+-------------------------------------+----------------------------------------------+----------------------------------------------+--------------- 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @aldorithms:matrix.org | | $74_XqgpHw_D18w0Xq0Lrc7_nmBvzDx1UajEn9yKc3Qc | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @jhs177992f:arcticfoxes.net | | $rFYpIiFmMMFHYqY_835sCFOEpJVUyLpEDM5Xoqr4uPc | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ariyaro:matrix.org | | $JqoI5tf1YrjoifsPEdxT67IeLJW6BCAdhIsFgxIJQ9Y | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @andreus:matrix.org | | $CUryaK8LXBZxMsDlVohrahm3LaiBC-uCoHHOot0UWS4 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @heliopolis:nitro.chat | | $utSCExwz0EzMAD5eIpCZ9BeEcinoslMFKTjgLO-H6rU | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @backpack:matrix.org | | $cCS3Ry7EFHCXQHTyMrJLratjz0GYRGsyCqU5AkDcnQ4 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @tfox42:matrix.org | | $93-srvPW852lSXze_r-y5VsDsi32Jf6H0GbXxrI8zFo | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @houndvoyager:matrix.org | | $5XMMvwgqdHpX6eyBzamDKpTADycXXvdUhMAIKhtPbTY | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @mercifulrage:converser.eu | | $EDFgLxKC6dz2cG7BYk7gO5xY2NNIKKRcMr7bM9XrPyA | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @crypticangel:pragma-messenger.ch | | $QG5tHgkzolxZAS0E3auilaPTjZqssp5fFr3NPgiwex0 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @coalminecanary:arcticfoxes.net | | $SG_cN3tHXhg9_7_mRHe5Qt4wFlQO7o8hvoV1XPepkTA | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @vovafreeman:matrix.org | | $r9YGaq9q9dbye19fLlO09TO-IC9k6bfOWl-unkJRNNw | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @qoijjj:arcticfoxes.net | | $XbGI2ynAn3WC5hGRrmsBmRbW6f3l5mzzyXoDsqp_C00 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @mebious:tchncs.de | | $zYy_smuirJ0TDeYdC5vgbyQQRxHqxTTF9OTdQ1-cjhs | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @windystone:matrix.org | | $Qa-UYRrnev1cof2voq4VrjlmnDmvEn7cF6QgIYomxFA | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @user0761:matrix.org | | $4w8S40ugloh8ixEufM0tj-_QvnR3oaiXXqdBIPicZDg | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @matrixruinedmylife:arcticfoxes.net | | $Q0mg0Le1GBwXh_H2tFLvwCM97Kf-njC0tW5GcqyFVPU | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @suicidalprincess:nitro.chat | | $Jv4B2FBIDgxkMR93mc9CNE88aTJqG8NrB79tMeVqhJY | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @hat:arcticfoxes.net | | $IDqAum_6ytZX_u0tz-8JglUBjwD39edl329bmExXrzo | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @iustitia:4d2.org | | $86lLB5SWm9xB93vZroSDU4hfjGQ-ZYrJr9fbnE2DZRE | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @beyiyaf450:matrix.org | | $DgRnGVumPcveQGtNWGLKrMzBwowHRJwtrRggmA_ndFo | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @fpmksoba:matrix.org | | $2GtIIACc89vvMrYb1AKJeF0ky4IgDtsZpsFCy_trBTM | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @sneber:nope.chat | | $9mpOqALVoUK1SQ6BbV4H0wKaiooTXbbHnmAIyDjXlvw | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @austin:tchncs.de | | $zg1FQYQk3pTix5UOY7PRxSCmE6Q-94lbXnpgMbByoys | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @bhydden:matrix.org | | $tp1MSrvb4KOEPtxqtJNQ6bKRMdzOVs7puBihkkI6d_4 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @masterrank:matrix.org | | $GJ4SZsaZuJg0NOS0GauO9DcNX3Qa1ESTSP-C84YbSYQ | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @w14:matrix.org | | $i-ZGHSTUJuRLjw6y7IZVSxAbxYZFi42AgYK_xkzEeOA | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @cyri:matrix.org | | $hxptXTV6_m2O4ZoP0w69HOIc1FQ22GC-UnJUz6d0HwQ | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ilove9999:matrix.org | | $k3C1PKERqlnX3ulqHe5qzpzZKfTVWQrknX1z5avD2Dc | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @quietlyjaded:matrix.org | | $NRY6fXAazYFaVa0kLpYWIhy2gfqJW8tVCFyZxVul01E | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @green_sheep:matrix.org | | $cGwRLAiQxivw7d3h9uMca5egIkDOhRVgZnD2Gl_zWVk | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @tisktisk:monero.social | | $lZ01gtBU50UhZ8sacAtpp6Ia5h6Gd54jON7o6Rxu9mc | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @loliloli:arcticfoxes.net | | $iPSlyT37ecv2ZqGH4yOAhkJ6_C51_J-muduqoeSf7PU | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @chili1803:matrix.org | | $jUwmF9DuNmdb56KzAP_D364QM7aocojUY-O95j6hx0c | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @sclanchy:matrix.org | | $27hLXs_qKe4WIgGqy6GigIV6TuknsBmzDSgBHn4c3TI | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @faxex:matrix.org | | $tz3qjuYJnJTHQBaUotnXXVQOEeoUAyGIQTd-1sdhKXc | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ghr89sb:matrix.org | | $UgsO3HsBuDnYVpxzpda-7WUH5vh9AujnN7dEcXX7L5A | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @poisonivy:arcticfoxes.net | | $pVH8QOPHfNfuv1cILOOSELlOuLAPKbliYYbflMkOg-g | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @johan942:matrix.org | | $jjKcZyb7w2DcbYvaKWf5c_052nutBDxcMbM3FJI5V5o | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @carlfriedrich:tchncs.de | | $CeMBICUBs6UPB1hpKooo7YjiTsbevuzC7IRjxihiOxU | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @2nami:matrix.org | | $MEz7DkYNjmX1dXW4aJoiGN8kkNnQ1zjmGekNh3fQijU | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @un32:matrix.org | | $JAAIV6FTLNgANKHyi1dhxskos-RngW_0tr44roqbRL4 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ancloadi:matrix.org | | $Dw7r7MLjd3JqLzzVdHKuXe0eVsO4wXQzNpOUvoCGtAQ | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @losintos:matrix.org | | $Viw3Yw_2fyjF6Jnk4-ozYMOq6JWrt2TasbEhagy7jyE | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @xyhhx:arcticfoxes.net | | $F6xBiS5WgeyYB7-R4B8eOMKWOD9LA0yNjGj_mDDJEX0 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.join_rules | | $AxE3gEzTLPT_hy7ML1MJKsX7DI6XkqujWP50LTDBODM | $P2FwR8kpzW1LDIZU2lKRc_wmghbXlIQZpwvE5D82d5Q | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @dahustlah:matrix.org | $8GW8hJvAHAGPTQIsPysF_mePLpd-GtZyqYIb4VzfoiU | $aGradtN72do5IOaG1dvRExtW4PAwFfjdlTNZVyOP6sQ | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @gvl7bcq6dn2xrbl:matrix.org | $_s6-w6-v0tE3-tYkWYQ1bbukUs844bn_9bUkttYCXF0 | $iNV97F-d1DnI0dGqj7L23a2pTtRmvSCNPpfDfGbGbg4 | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @masayuki:wired.rehab | $yMVmIFtT_hpeMh-9GLSdY84NrNYSRhn8rUX2Z2AD5_g | $egJIEy0PhxdUfdDCAPAVJDSrIYTLYtdAIIWCQ3zvsNA | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @matchboxbananasynergy:matrix.org | $A4R_VVl2TaN53EWiLBoipPyigkEZodgWOiceeieshaQ | $s-1rvX3ldLsL5Azxj7mSjkhWxboqaEuQiJlMCglMIYk | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @skull_gap:archoslinux.cz | $MX2EhPV0TYzMxegHKJX1U1KKvqTrAp8vwS5DwqqDlNw | $_0SGO3HGR2RgYWYAxczIkwXzCl7e3PLuZgrO_HTgj6Y | master 25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @slate-module:matrix.org | $xM6Y-mUyvW9yOZf_lODJOdqLI0Ao1nYuaYQYonysi84 | $2IUW5LONVOx_k1I5nxRv9AuF8BFAp5NL8I_KxFa16Zk | master (52 rows) ```

Anything else that would be useful to know?

In addition to arcticfoxes.net documented above, this state reset was confirmed via the C‑S API on matrix.org (Synapse 1.96.0rc1) and a third server (Synapse 1.95.1). I do not know the exact details of their deployments.

thestinger commented 11 months ago

This keeps happening for the GrapheneOS rooms, which were among the largest Matrix communities until this happened. It's always the same symptom of join rules, power levels, etc. resetting and then all users who joined since the join rules it resets to being dropped out of the room with the room bricked in Element where they can't send, receive or leave without clearing cache.