r1bnc
commented
2 years ago I also seem to get a lot of these messages and the synapse times out (goes down):
matrix-synapse[613379]: 2022-08-05 05:03:42,560 - synapse.http.server - 795 - WARNING - PUT-20973 - Not sending response to request <XForwardedForRequest at 0x7f47ad9cb430 method='PUT' uri='/_matrix/federation/v1/send/1659445109176' clientproto='HTTP/1.0' site='8048'>, already disconnected.
...
matrix-synapse[613379]: 2022-08-05 05:03:42,608 - synapse.http.server - 795 - WARNING - PUT-20975 - Not sending response to request <XForwardedForRequest at 0x7f47e2aa0d90 method='PUT' uri='/_matrix/federation/v1/send/1659578924069' clientproto='HTTP/1.0' site='8048'>, already disconnected.
...
scrubbedI am using the ansible docker image for matrix.
UPDATE: Fixed mine by deleting a large room on synapse admin.
UPDATE: After updating matrix and postgre, the issue is back.
anoadragon453
richvdh
hubermat
reivilibre
squahtx
H-Shay
Description
Hi,
Our own matrix bridge runs sometimes into long running join requests (> 30 sec.), which triggers a timeout on the bridge side. This is what the bridge executes:
return bridge.getIntent()._ensureJoined(roomAlias, false, undefined, true);Typically this takes 0.03 sec.. But sometimes it takes a loooong time!
Here is the result from homeserver.log (filtered only for POST-9744812):
As you can see, connection from the bridge is closed after 30 seconds (timeout), the POST request in total takes 70 seconds.
What we notice, is that these long running join-requests sometimes occur in parallel with other similar join-request.
Here for example are three join requests from the bridge bot for three different rooms, all taking a long time and finishing at the same time! The above request POST-9744812 is one of them:
Looks like there is something blocking in synapse (or the database, PostgreSQL in our case), then when the lock is released, all requests finish at the same time. Runtimes of the three requests above: 40, 70 and 100 seconds.
Here the POST join requests in the bigger context:
We have activated prometheus metrics on our synapse server, but could not find anything abnormal when these delays occured.
Steps to reproduce
bridge.getIntent()._ensureJoined(roomAlias, false, undefined, true);Homeserver
https://chat.church.tools/
Synapse Version
{"server_version":"1.63.1","python_version":"3.8.10"}
Installation Method
Debian packages from packages.matrix.org
Platform
Ubuntu 18.04
PostgreSQL Version
postgres (PostgreSQL) 10.20 (Ubuntu 10.20-0ubuntu0.18.04.1)
Relevant log output
Anything else that would be useful to know?
No response