Closed DeezyTheFox closed 1 year ago
Tricky to advise, we'd need to investigate and it sounds like it would be involved.
Can you see a pattern between the problem and the rate of incoming media requests? E.g. are you getting lots of small requests? Maybe a few requests that upload huge files? Is there any logging on S3's side which you can correlate with the timeouts?
Can you provide more logs from Synapse? Ideally I'd like to see 5 minutes of logs before and after a request, along with the request URL and the request timestamp.
Can you provide more logs from Synapse? Ideally I'd like to see 5 minutes of logs before and after a request, along with the request URL and the request timestamp.
My synapse has been working suspiciously stable for the last week.
It's a bit difficult to collect logs for irregular problem with ~1400 users on homeserver. I'll back with detailed logs when the problem shows up.
I am going to close this for now as we cannot action this without logs, if the problem arises again and logs are available we can re-open.
Hello! We have a repeatable but unregular bug with connection to s3. Sometimes media stopped to upload to matrix, clients got timeouts by reserse-proxy. Restart of main synapse process helps for some time. We can't cath this and can't reproduce. Logs spammed by few messages.
Spams everytime, regardless of the problem:
urllib3.connectionpool - 305 - WARNING - GET-61775200 - Connection pool is full, discarding connection: [s3 address]. Connection pool size: 10
Cathed sometimes, spammed when there is a problem:
Cathed only when problem exists:
Synapse installation: