Handling state_ids responses loads huge amounts of events into memory at once, causing OOMs

[richvdh]

For example: the response to matrix://matrix.org/_matrix/federation/v1/state_ids/%21jWRVQAlVGjigKCRGwS%3Amatrix.org?event_id=%241577162865947686RYsuB%3Amatrix.org includes an auth chain with 22327 events. surely there can't be that many events in $1577162865947686RYsuB:matrix.org's auth chain?

[richvdh]

I wonder if this has been caused (or exacerbated) by us deploying #6556

[richvdh]

right, so the problem here is that synapse is returning the auth_events for each of the events in the state of the room at the given event_id. This is consistent with the spec; however it seems to be utterly pointless. For a given event in the state, either:

• we already have it, and will therefore know its auth_events
• we don't have it, so will have to request it via /event which will tell us its auth events.

[richvdh]

we don't have it, so will have to request it via /event which will tell us its auth events.

unfortunately, as of #7817, synapse won't recursively fetch auth events in this way :/

[richvdh]

On reflection, it's correct that we return all these auth events. The problem is that we load them all into memory at once.

[richvdh]

this might be a bit better since #9601, but it's still problematic

[DMRobertson]

To summarise:

• /state_ids gives you two lists of strings: auth event ids and state event ids
• We think that is completely sensible.
• But loading all of these event ids into memory in order to provide that list to someone who is requesting /state_ids can be expensive.
• If we had some way to stream in event ids from the DB and stream them out as part of the response, we could smooth out the spike in memory usage
  • Do we have an example of this? IIRC ijson is just for parsing?

Relevant bits of source:

https://github.com/matrix-org/synapse/blob/da2c93d4b69200c1ea9fb94ec3c951fd4b424864/synapse/federation/transport/server/federation.py#L171-L185

https://github.com/matrix-org/synapse/blob/1eed795fc56d95df3968e37f3a4db92f24513e15/synapse/federation/federation_server.py#L562-L581

https://github.com/matrix-org/synapse/blob/1eed795fc56d95df3968e37f3a4db92f24513e15/synapse/federation/federation_server.py#L583-L590