Closed ara4n closed 4 years ago
I seem to be getting these extremities still, I cleared them out about a month ago and since then these new forward extremities have accumulated:
room_id | c
------------------------------------------+----
!vUXcVeSXkoAGbIREoj:matrix.org | 99
!vJPDJLWMJbPnKlwyzE:matrix.org | 66
!jvKJuwwWRGDzBhLurm:matrix.org | 42
!ifEXJOtimDZnbaYObc:matrix.org | 26
!xGmquSnSozzjgUDBIB:matrix.org | 16
!BfPzMJuQMaOmBzFOdD:matrix.org | 11
!awLQkSooHfMgAfXwCj:matrix.org | 10
!bzQWZFKZWVJEvEdVqE:matrix.org | 10
!YaQDIfXXPIjxTtkKZv:matrix.org | 10
!lxLFDpQJHZtqePeIUK:matrix.org | 10
!DBSOFOSAeAPWhqLARZ:matrix.org | 10
!bVLqshyvYOtNWfudcg:matrix.org | 10
!Blade_Sparxx_Pats_King_Anubis:ponies.im | 10
!dseyIKgGxDaWVXpeHf:matrix.org | 10
!MFogdGJfnZLrDmgkBN:matrix.org | 10
!jlAFXBRrJBvyWsnEsc:matrix.org | 10
!KIwsdPeEnTTmPnMpMv:fam-ribbers.com | 10
!kfXkmFVLeQDBFexcew:matrix.org | 10
!geNjbnNYhhankxHbDh:matrix.org | 10
!YJPnqWJENzArNZWzIz:mozilla.org | 10
My Synapse version: 1.23.0
fwiw, a query to help identify the room names and such of those rooms is:
select rss.room_id, rss.name, (select count(*) from event_forward_extremities as efe where efe.room_id = rss.room_id) as extremities, rss.join_rules, rsc.current_state_events, rsc.joined_members, rsc.invited_members, rsc.left_members, rsc.local_users_in_room from room_stats_state as rss join room_stats_current as rsc on rsc.room_id = rss.room_id where rss.room_id in (select room_id from event_forward_extremities group by room_id order by count(*) desc limit 20);
It's not pretty, but it does at least give a bit more context if one is trying to determine if a particular room even needs to be on the server.
There is a related PR #9062
PR is merged, cool! But how to reveal rooms, that have too much extremities, via admin API call, instead of old-school raw SQL query?
TLDR: To determine if you are affected by this problem, run the following query:
Any rows showing a count of more than a handful (say 10) are cause for concern. You can probably gain some respite by running the query at https://github.com/matrix-org/synapse/issues/1760#issuecomment-379183539.
Whilst investigating the cause of heap usage spikes in synapse, correlating jumps in RSZ with logs showed that 'resolving state for !curbaf with 49 groups' loglines took ages to execute and would temporarily take loads of heap (resulting in a permenant hike in RSZ, as python is bad at reclaiming heap).
On looking at the groups being resolved, it turns out that these were the extremities of the current room, and whenever the synapse queries the current room state, it has to merge these all together, whose implementation is currently very slow. To clear the extremities, one has to talk in the room (each message 'heals' 10 extremities, as max prev-events for a message is 10).
Problems here are: