Timeline reordering when run with multiple Synapse slow synchrotrons

[kegsay]

This is a hypothetical problem and has not yet been seen in the wild.

The proxy relies on all pollers seeing the same set of events in the same order, and no poller can have seen event N without having seen N-1 first. This is not always true in reality: see #198. There is an edge case which can cause events to be inserted into the wrong place in the timeline. Consider:

• Poller A is on a slow synchrotron and is N-52 events behind the latest event in the room.
• User B joins the room. Poller B sees the latest N-50..N events and a state block. It can snapshot the state at N-50 and roll forward to N fine.
• At this point, the proxy has not seen N-51.
• Poller A then sees N-51, inserting it after N as it looks to be brand new from the proxy's perspective.

All of the following need to happen for this to occur:

• a synchrotron to be 50+N events behind another synchrotron in a single room.
• someone joins the room, or a poller comes back from the dead (when we stop polling for awol clients).
• the slower synchrotron delivers <N events to the poller.

If the synchrotron is <50 events behind, then joining a room or repolling will fill in the timeline gap (due to the proxy asking for the max timeline limit of 50). If the slower synchrotron then delivers in a single sync response any one of those 50 events (e.g events [50+2, 50+1, 50] then we would ignore events [50+2, 50+1] due to taking the timeline positions into account (since #198).

The ramifications are unfortunately quite large, as the incorrectly positioned event could also affect the room state, not just the timeline.

[kegsay]

https://github.com/matrix-org/matrix-spec-proposals/pull/4033 would fix this as we can see event N-51 is older than N-50+.