Open janonym1 opened 1 year ago
janonym1
commented
1 year ago some more before/after metrics: you can see the increased/spiking CPU usage and HTTP requests starting when joining said room in the beginning of the graphs (at about 10/3):
I also noticed, that when the CPU usage of a federation_worker explodes, it causes a lot of DB txn:
janonym1
commented
1 year ago The initial huge CPU/DB txn spikes (in the first week) didnt occur again but the server CPU usage is still increased by about +5% permanently and the outgoing HTTP requests (~10/s) are still persisting. At this point, I can live with the +5% CPU usage for now but the all the resync warnings are flooding my logs with synapse.handlers.device - 1014 - WARNING - _maybe_retry_device_resync-9855 - Failed to handle device list update for X warnings which makes it hard to see other relevant warnings. I am still not seeing the room listed on my homeserver (synapse-admin)
Any advice? I dont think, manually clearing some DB tables is a solution because this can just happen again, if any user on my homeserver is joining said room.
Maybe I can manually add the failed resync servers to a banlist? I am also wondering If I should just blanket ban the biggest rooms on matrix.org (the mainHQ and BlackArch at least) if any user joining them can trigger said problems
yingziwu
commented
1 year ago I meet the same problem with you.
After joining the Matrix HQ room, my synapse is constantly under heavy load.
The federation_transaction_transmission_loop is found to cause a sustained high load on the CPU through grafana graph.
janonym1
commented
1 year ago @yingziwu joining a huge room will take a lot of resources and can take some time, depending on the system you are using (maybe even days?). I think this behavior is expected but the CPU usage should die down some time after joining.
What I am specifically interested in, is the outgoing HTTP requests, which seem to fluctuate on your server and is normal in the beginning. After joining (and leaving) the room, are there still a lot of outgoing HTTP requests? On my system I have a "noise floor" of about 5-6 requests per second, which correlates to my log showing a failed connection to homeservers like: synapse.handlers.device - 1191 - WARNING - _maybe_retry_device_resync-70 - Failed to handle device list update for @zmvibdrwb0:abxy.fr: 403
It seems, this doesnt happen on your system, since you sometimes seem to have 0 outgoing HTTP requests.
btw: the average CPU usage from all the churning maybe_retry_device_resyncs has decreased to about 2-3% now (using synapse 1.74) but the log is still getting spammed with the said warnings
Description
We have a big Synapse-Server with a lot of workers (for generic, federation, etc) running with open federation and it has been working fine so far. However, I recently joined the BlackArch Room (on the matrix.org server) to test how fast it can join a really big room with more than 30k+ users.
After the usual initial high CPU usage (for about 15-20m) it worked and I joined the room as expected but afterwards, the server permanently consumed about 1/3th of the CPU power available (we have 12 cores and 20GB Ram), even days later and when everyone was idle (e.g. over weekends). I didn't have any issues with "smaller" rooms before, (like the ArchLinux or RaspberryPi rooms, and they still have about 10k users) so I investigated some more. I noticed the following things:
After joining the BlackArch room, one of my federation worker permanently used about ~1/3 of my CPUs:
After restarting said federation_worker, the CPU usage dropped and stayed low for a few days, after which it picked up ~1/3 of our CPU usage again. I restarted it again and it stayed low for a few days after the same thing happens again. I am not sure, what exactly triggers the high CPU usage in the federation_worker.
The overall CPU usage of the master process (specifcally: master_0_maybe_retry_device_sync) increased by about 5%, which isnt a huge deal but still bad for just joining (and leaving) a room:
I think, the overall increase of the CPU usage is caused by the outgoing HTTP request rate (master-0 GET(federation)), which increased massively to about 10 req/s, and that value is way above our average. It also is permanently staying at 10req/s permanently since then, regardless of time, date or usage.
I am not sure how to fix this exactly, it doesnt matter if I rejoin or leave the room. After leaving it, I also tried deleting the room (with the synapse-admin webGUI) but to no avail. I read somewhere about deleting some live DB tables but that sounds risky too me, especially in a live prod environment.
I assume, this is happening because some homeservers/accounts in those big rooms are not existing anymore and our synapse server keeps trying to reconnect to them (see logs). I cannot tell, why this has become relevant only now, since we have a lot federated rooms joined (a few with 10k+ users) for a long time now. This massive increase in sustained CPU usage however happened, after joining the BlackArch room. Since then, the outoing http req/s and CPU usage is off the charts
What can I do against that? Since we are openly federated (and want to stay that way) I can only try to ban certain problematic rooms (like the mainHQ) and hope, others wont become problematic soon. However, since the damage is already done, what can I do now? I already restarted everything and have my global_cache_factor increased to about 5.0. Deleting said room from my HS doesnt seem to help, since the maybe_retry_device_sync keeps churning away regardless. For now, I manually restart the federation_worker once it falls back into the superhigh CPU usage, which is happening about 2x/week.
Is there a error_backoff_timer or something similar, I can set? Can I ban/ignore the hosts, causing the federation_transaction_transmission_loop somewhere? Isnt this also a possible DDOS scenario? Maybe deleting the DB tables (see https://github.com/matrix-org/synapse/issues/11807) will help but the problem will probably pop up again (also, I didnt suddenly start whitelisting or changing some configs)
Steps to reproduce
Homeserver
another synapse homeserver
Synapse Version
1.68
Installation Method
Other (please mention below)
Platform
Server: Ubuntu 20.04 LTS 12 cores, 20GB RAM, 1GBit/s Network, dedicated postgreSQL DB cluster presence disabled, cache_factor 5.0 installed with Slavis ansible playbook running on a VM it worked fine so far and is still usable for everything
Relevant log output
some matrix-synapse main process logs: