search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

Performance issue, 100% CPU for some time related to _process_new_pulled_events_with_failed_pull_attempts #16635

Closed n-peugnet closed 11 months ago

n-peugnet commented 11 months ago

Description

Since the upgrade to Synapse 1.95 and 1.95.1 we some time have around 20minutes of unresponsiveness, due to the CPU usage going up to 100%. We have a small instance no workers and have enabled retention policy.

In the logs we have a LOT of these messages when this happens (thousands):

2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$GeBn6_jqQH8qAhq_P9E3Jvzhox4Ie1kyNJUE4yit3Og - Dropping event <FrozenEventV3 event_id=$GeBn6_jqQH8qAhq_P9E3Jvzhox4Ie1kyNJUE4yit3Og, type=m.room.member, state_key=@mathfold:matrix.org, outlier=False>, which relies on auth_event $YbYGoEa3I6SZ7WZayCHNIWGp3uswaxiyQz3kx84Mt5g, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$GrvuQ56mbhx09wUZCZleoZjztU2V0wfD8Aq_asOmjag - Dropping event <FrozenEventV3 event_id=$GrvuQ56mbhx09wUZCZleoZjztU2V0wfD8Aq_asOmjag, type=m.room.member, state_key=@j.668:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$JqriNbJVo0uwcRYquBgafjwzX4T40NZCbJcgoCrgKn0 - Dropping event <FrozenEventV3 event_id=$JqriNbJVo0uwcRYquBgafjwzX4T40NZCbJcgoCrgKn0, type=m.room.member, state_key=@zxc4564:matrix.org, outlier=False>, which relies on auth_event $kAoHJeSnwt2un1Z252CHlEtuYsQmxAb9Wb2uVJq_bZM, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$Wlm_UrSeJbZ4l6LuZbLqU2fNE9tVKSUqpcXgy7voSgw - Dropping event <FrozenEventV3 event_id=$Wlm_UrSeJbZ4l6LuZbLqU2fNE9tVKSUqpcXgy7voSgw, type=m.room.member, state_key=@barbatos360:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$sAfUEQK_telhJjWJKnwmkyvIg0uBYNFZHHKXWGTGBcU - Dropping event <FrozenEventV3 event_id=$sAfUEQK_telhJjWJKnwmkyvIg0uBYNFZHHKXWGTGBcU, type=m.room.member, state_key=@lucifer.s:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$t55AqmMtWWqFDwIGyxs31NHMJbvLt1zwKuc5Yz5sID8 - Dropping event <FrozenEventV3 event_id=$t55AqmMtWWqFDwIGyxs31NHMJbvLt1zwKuc5Yz5sID8, type=m.room.member, state_key=@tagg181:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$ROPuyhKEdUGgvd4DmAaGZcO7Y5NmHYpapYUj6A1FP_o - Dropping event <FrozenEventV3 event_id=$ROPuyhKEdUGgvd4DmAaGZcO7Y5NmHYpapYUj6A1FP_o, type=m.room.member, state_key=@zadiant:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$tISd0yIoHacQ7mOFao8nCvdW5-1NJb8W1Gx2RL5Aw2Y - Dropping event <FrozenEventV3 event_id=$tISd0yIoHacQ7mOFao8nCvdW5-1NJb8W1Gx2RL5Aw2Y, type=m.room.member, state_key=@eastanzaa:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$I65o5eCsiBYuajF7FTZcwSLbZcXfKfaGldaf4aXo5P4 - Dropping event <FrozenEventV3 event_id=$I65o5eCsiBYuajF7FTZcwSLbZcXfKfaGldaf4aXo5P4, type=m.room.member, state_key=@1dyablo:matrix.org, outlier=False>, which relies on auth_event $bxhFWiYB_-w5cTnmZJbsw0vrfjt5YHHfY345RK8A_5E, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$ZufRiBlJ8fwr1V4lHB3Sa8wqrQ9tYppWEBBR-WUvZ2U - Dropping event <FrozenEventV3 event_id=$ZufRiBlJ8fwr1V4lHB3Sa8wqrQ9tYppWEBBR-WUvZ2U, type=m.room.member, state_key=@kharma:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$F7OFHNwhXDpflYjHPUfbOq6x99ZnX9AJ9cuIM_-WL5M - Dropping event <FrozenEventV3 event_id=$F7OFHNwhXDpflYjHPUfbOq6x99ZnX9AJ9cuIM_-WL5M, type=m.room.member, state_key=@omxian:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found

Not only from matrix.org, but from a lot of different servers.

Steps to reproduce

Sorry I really don't know what to add in my case.

Homeserver

club1.fr

Synapse Version

1.95.1

Installation Method

Debian packages from packages.matrix.org

Database

PostgreSQL single server

Workers

Single process

Platform

Configuration

I just disabled presence but it was enabled until now.

Retention policy is ON:

retention:
  # The message retention policies feature is disabled by default. Uncomment the
  # following line to enable it.
  #
  enabled: true

  # Default retention policy. If set, Synapse will apply it to rooms that lack the
  # 'm.room.retention' state event. Currently, the value of 'min_lifetime' doesn't
  # matter much because Synapse doesn't take it into account yet.
  #
  default_policy:
    min_lifetime: 1d
    max_lifetime: 3y

media_retention:
    local_media_lifetime: 2y
    remote_media_lifetime: 90d

Relevant log output

2023-11-14 21:21:21,272 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$N7u0LNHLSR0NRhL3OljVVQZBAS74PD5osEuc9e8t4jo - Dropping event <FrozenEventV3 event_id=$N7u0LNHLSR0NRhL3OljVVQZBAS74PD5osEuc9e8t4jo, type=m.room.member, state_key=@glittervappy:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$hp0HuMZaTkHkc49PYVcu0lqy_NL1EJA5dSsDtd2xN_c - Dropping event <FrozenEventV3 event_id=$hp0HuMZaTkHkc49PYVcu0lqy_NL1EJA5dSsDtd2xN_c, type=m.room.member, state_key=@hodgoblin:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$-NukKwrCyezB2_TcKVapghXY3mNgsil2yqKw-1TCPog - Dropping event <FrozenEventV3 event_id=$-NukKwrCyezB2_TcKVapghXY3mNgsil2yqKw-1TCPog, type=m.room.member, state_key=@kostyaekaterina:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$qloNFYsuACHHn3K0dKTFsYTy6ZvWAj1vxQ7dJ9ypxrw - Dropping event <FrozenEventV3 event_id=$qloNFYsuACHHn3K0dKTFsYTy6ZvWAj1vxQ7dJ9ypxrw, type=m.room.member, state_key=@matrx0th:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$3zjjLhGqKV7F-PFWKMphnc7FO3B-fzZB1vEDDuL48ys - Dropping event <FrozenEventV3 event_id=$3zjjLhGqKV7F-PFWKMphnc7FO3B-fzZB1vEDDuL48ys, type=m.room.member, state_key=@underronin:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,275 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$8KYJRHrJcZRh4q1Y2TvGhU7Sbpa7w12arxe1UOu1OL4 - Dropping event <FrozenEventV3 event_id=$8KYJRHrJcZRh4q1Y2TvGhU7Sbpa7w12arxe1UOu1OL4, type=m.room.member, state_key=@tttttttt:waifuhunter.club, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,275 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$a7yDjHoWNoREUpLmH3YEzfCSsXZHhrLlge_97fGykjg - Dropping event <FrozenEventV3 event_id=$a7yDjHoWNoREUpLmH3YEzfCSsXZHhrLlge_97fGykjg, type=m.room.member, state_key=@fun_adminx:matrix.juggler.jp, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$rjhdB4i0PxUu4PvSUtyN8lhgmgHmV2i_idxc81CPw9g - Dropping event <FrozenEventV3 event_id=$rjhdB4i0PxUu4PvSUtyN8lhgmgHmV2i_idxc81CPw9g, type=m.room.member, state_key=@jagasaki:koneko.chat, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$OQphZ7H2jMD5HIje9JXyRx9j7-hpcgN1M_tKLK_BV6A - Dropping event <FrozenEventV3 event_id=$OQphZ7H2jMD5HIje9JXyRx9j7-hpcgN1M_tKLK_BV6A, type=m.room.member, state_key=@kemuri:koneko.chat, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$J2VDMGJYfFitzsIX6dBomyk3vSX20rJ04RERJEaicPw - Dropping event <FrozenEventV3 event_id=$J2VDMGJYfFitzsIX6dBomyk3vSX20rJ04RERJEaicPw, type=m.room.member, state_key=@presloli:koneko.chat, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$fn10pv4DQPFlnhv9-sCdz4S3x8oln7_y09KJZkgb_c4 - Dropping event <FrozenEventV3 event_id=$fn10pv4DQPFlnhv9-sCdz4S3x8oln7_y09KJZkgb_c4, type=m.room.member, state_key=@greexxx:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,277 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$9Fkw2-BdSO_fG_Ua9afnzYdFn7VJ3_9UUWOf6jq_DzI - Dropping event <FrozenEventV3 event_id=$9Fkw2-BdSO_fG_Ua9afnzYdFn7VJ3_9UUWOf6jq_DzI, type=m.room.member, state_key=@gunuk:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,277 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$zoHjtSPUDAXPLA3x3JGplPwiZAgObX6JxHzy4TLVye0 - Dropping event <FrozenEventV3 event_id=$zoHjtSPUDAXPLA3x3JGplPwiZAgObX6JxHzy4TLVye0, type=m.room.member, state_key=@krestomi:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,277 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$FRVNQUfvTSfPdUiWZ0s1DbQHlOUAkH4b5kOxj7z29fI - Dropping event <FrozenEventV3 event_id=$FRVNQUfvTSfPdUiWZ0s1DbQHlOUAkH4b5kOxj7z29fI, type=m.room.member, state_key=@septemberrain:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found

Anything else that would be useful to know?

2023-11-14-222341_663x623_scrot 2023-11-14-224649_723x387_scrot (the purple one is synapse- _process_new_pulled_events_with_failed_pull_attempts)

clokep commented 11 months ago

Since the upgrade to Synapse 1.95 and 1.95.1

What version of Synapse did you upgrade from?

That particular error comes while backfilling events and comes from:

https://github.com/matrix-org/synapse/blob/43d1aa75e8cbf9d522b425d51d5ac1a742b59ffb/synapse/handlers/federation_event.py#L1736-L1748

It looks like all those events are from the room !VXimQkuRxeRUFhboju:anontier.nl. I'm slightly suspicious that you have the retention policies feature enabled as that has been known to be buggy in the past. Have you had it enabled for a long time?

n-peugnet commented 11 months ago

Since the upgrade to Synapse 1.95 and 1.95.1

What version of Synapse did you upgrade from?

I upgraded from 1.92.3 to 1.95.0 then later to 1.95.1

It looks like all those events are from the room !VXimQkuRxeRUFhboju:anontier.nl.

How did you know which room these events are related to? After a quick look, I won't be sad if I have to purge this room from the server.

I'm slightly suspicious that you have the retention policies feature enabled as that has been known to be buggy in the past. Have you had it enabled for a long time?

I have it enabled for a long time yes, since Tue Aug 10 18:00:55 2021 exactly. And I was also afraid that it might be related.

clokep commented 11 months ago

It looks like all those events are from the room !VXimQkuRxeRUFhboju:anontier.nl.

How did you know which room these events are related to? After a quick look, I won't be sad if I have to purge this room from the server.

Event IDs are globally unique, so I did:

SELECT room_id FROM events WHERE event_id = '...';

I'm slightly suspicious that you have the retention policies feature enabled as that has been known to be buggy in the past. Have you had it enabled for a long time?

I have it enabled for a long time yes, since Tue Aug 10 18:00:55 2021 exactly. And I was also afraid that it might be related.

There were known bugs in the feature until Synapse ~1.94, so I think this might be the cause. You could purge the room entirely and then rejoin it. I know that's not an ideal solution though.

n-peugnet commented 11 months ago

How did you know which room these events are related to? After a quick look, I won't be sad if I have to purge this room from the server.

Event IDs are globally unique, so I did:

SELECT room_id FROM events WHERE event_id = '...';

But if the event has been dropped as the log suggests, can I still try this query on my own homeserver?

I have it enabled for a long time yes, since Tue Aug 10 18:00:55 2021 exactly. And I was also afraid that it might be related.

There were known bugs in the feature until Synapse ~1.94, so I think this might be the cause.

That's what I saw by following a little bit the project on GitHub.

You could purge the room entirely and then rejoin it. I know that's not an ideal solution though.

Thank you for your advice, I just purged the room. We only had a single user from club1.fr in this room so that's not a big loss. I will keep monitoring the server and the logs to see if this problem arise again, but until then I will close this issue.

clokep commented 11 months ago

Feel free to shout if you see it come back!