Forward extremities accumulate and lead to poor performance

ara4n commented 7 years ago

TLDR: To determine if you are affected by this problem, run the following query:

select room_id, count(*) c from event_forward_extremities group by room_id order by c desc limit 20;

Any rows showing a count of more than a handful (say 10) are cause for concern. You can probably gain some respite by running the query at https://github.com/matrix-org/synapse/issues/1760#issuecomment-379183539.

Whilst investigating the cause of heap usage spikes in synapse, correlating jumps in RSZ with logs showed that 'resolving state for !curbaf with 49 groups' loglines took ages to execute and would temporarily take loads of heap (resulting in a permenant hike in RSZ, as python is bad at reclaiming heap).

On looking at the groups being resolved, it turns out that these were the extremities of the current room, and whenever the synapse queries the current room state, it has to merge these all together, whose implementation is currently very slow. To clear the extremities, one has to talk in the room (each message 'heals' 10 extremities, as max prev-events for a message is 10).

Problems here are:

[ ] Why are we accumulating so many extremities? I assume it's whenever there's some downtime the graph breaks, leaving a dangling node.
[ ] Is there a way to stop them accumulating by healing or discarding them on launch (e.g. by sending a null healing event into the room)?
[ ] Why is state resolution so incredibly heavy? There should hardly be any conflicting state here, unless the bifurcation has been going on for months. Is it because to auth potential conflicts we have to load all auth events, which include every m.room.member
[ ] Logs of a state resolution happening from arasphere at DEBUG show lots of thrashing on the rejections table too.
[ ] We're also seeing ominous pauses in the logging of requests which resolve state, as if there's some lock we're contending for. (This might be the same as #1774)
[ ] Can we just insert dummy nodes in our local copy of the DAG after doing a successful state resolution, to avoid having to constantly re-calculate it or rely on naive caching?

PC-Admin commented 3 years ago

I seem to be getting these extremities still, I cleared them out about a month ago and since then these new forward extremities have accumulated:

                 room_id                  | c  
------------------------------------------+----
 !vUXcVeSXkoAGbIREoj:matrix.org           | 99
 !vJPDJLWMJbPnKlwyzE:matrix.org           | 66
 !jvKJuwwWRGDzBhLurm:matrix.org           | 42
 !ifEXJOtimDZnbaYObc:matrix.org           | 26
 !xGmquSnSozzjgUDBIB:matrix.org           | 16
 !BfPzMJuQMaOmBzFOdD:matrix.org           | 11
 !awLQkSooHfMgAfXwCj:matrix.org           | 10
 !bzQWZFKZWVJEvEdVqE:matrix.org           | 10
 !YaQDIfXXPIjxTtkKZv:matrix.org           | 10
 !lxLFDpQJHZtqePeIUK:matrix.org           | 10
 !DBSOFOSAeAPWhqLARZ:matrix.org           | 10
 !bVLqshyvYOtNWfudcg:matrix.org           | 10
 !Blade_Sparxx_Pats_King_Anubis:ponies.im | 10
 !dseyIKgGxDaWVXpeHf:matrix.org           | 10
 !MFogdGJfnZLrDmgkBN:matrix.org           | 10
 !jlAFXBRrJBvyWsnEsc:matrix.org           | 10
 !KIwsdPeEnTTmPnMpMv:fam-ribbers.com      | 10
 !kfXkmFVLeQDBFexcew:matrix.org           | 10
 !geNjbnNYhhankxHbDh:matrix.org           | 10
 !YJPnqWJENzArNZWzIz:mozilla.org          | 10

My Synapse version: 1.23.0

turt2live commented 3 years ago

fwiw, a query to help identify the room names and such of those rooms is:

select rss.room_id, rss.name, (select count(*) from event_forward_extremities as efe where efe.room_id = rss.room_id) as extremities, rss.join_rules, rsc.current_state_events, rsc.joined_members, rsc.invited_members, rsc.left_members, rsc.local_users_in_room from room_stats_state as rss join room_stats_current as rsc on rsc.room_id = rss.room_id where rss.room_id in (select room_id from event_forward_extremities group by room_id order by count(*) desc limit 20);

It's not pretty, but it does at least give a bit more context if one is trying to determine if a particular room even needs to be on the server.

dklimpel commented 3 years ago

There is a related PR #9062

MurzNN commented 3 years ago

PR is merged, cool! But how to reveal rooms, that have too much extremities, via admin API call, instead of old-school raw SQL query?

matrix-org / synapse

Forward extremities accumulate and lead to poor performance #1760