Performance problems with bridges with ephemeral events enabled

[Half-Shot]

A lot of these problems seem to happen at scale. This is what we saw when we restarted matrix.org with ephemeral events enabled for the OFTC bridge:

• The memory usage of the appservice worker has increased ~3.5x, presumably to handle caching of members in rooms ahead of time.
• CPU will churn during startup handling notify_interested_services_ephemeral
• The database will be hammered for get_users_in_room and set_type_stream_id_for_appservice

[Half-Shot]

There is also 2020-12-09 10:55:13,448 - synapse.metrics.background_process_metrics - 217 - ERROR - notify_interested_services_ephemeral-19 - Background process 'notify_interested_services_ephemeral' threw an exception aplenty in the logs

as well as 2020-12-09 11:45:32,310 - synapse.storage.txn - 517 - WARNING - notify_interested_services_ephemeral-135952 - [TXN OPERROR] {set_type_stream_id_for_appservice-61d89} could not serialize access due to concurrent update

[Half-Shot]

Thought about this some more and we could save a lot of CPU by just adding a linearizer to notify_interested_services_ephemeral. I think I'll have a go at that.

[Half-Shot]

I've stuck some tags on this that I think are appropriate.

[kevinrademan]

When I have this feature enable my app service worker logs get spammed with the following error "could not serialize access due to concurrent update"

It also results in several minutes delay before events are delivered to my app services.

[anoadragon453]

@kevinrademan Note that that indicates a higher increase in time that Synapse is waiting for database access. Thus in its current state this feature potentially has a high impact on database resources.

[anoadragon453]

See also https://github.com/matrix-org/synapse/issues/10836.