POST https://matrix-client.matrix.org/_matrix/client/r0/join/%23libera%3Alibera.chat errors with 502

progval commented 1 year ago

Description

It is impossible to join #libera:libera.chat from matrix.org

POST requests to https://matrix-client.matrix.org/_matrix/client/r0/join/%23libera%3Alibera.chat error with Cloudflare's standard 502

I don't know how long this has been going on, but people have been complaining about it in #irc:matrix.org and #libera-matrix:libera.chat for about a month.

Steps to reproduce

Try to join #libera:libera.chat from matrix.org

Homeserver

matrix.org

Synapse Version

1.73.0rc2 (b=matrix-org-hotfixes,34fa1276a4)

Installation Method

I don't know

Database

postgresql

Workers

I don't know

Platform

n/a

Configuration

No response

Relevant log output

n/a

Anything else that would be useful to know?

No response

clokep commented 1 year ago

Probably a duplicate of #14462?

DMRobertson commented 1 year ago

Or https://github.com/matrix-org/synapse/issues/14480

DMRobertson commented 1 year ago

It is impossible to join #libera:libera.chat from matrix.org

Are you able to join this room from another HS?

progval commented 1 year ago

I just tried on envs.net. First, Element showed a spinner forever, so I reloaded the page. Now I get {"errcode":"M_UNKNOWN","error":"Internal server error"}.

iakat commented 1 year ago

Getting the same error from my HS, which has no issue federating.

Diablo-D3 commented 1 year ago

This is also affecting ##tea

apos0 commented 1 year ago

Can't join #hardware #libera #bash from neither data.haus nor matrix.org but somehow #ffmpeg works ok

realtyem commented 1 year ago

I'm wondering if this is related to the 502 errors that have been seen around #14103 lately?

apos0 commented 1 year ago

just for reference i was able to join the channels. They just appeared in my list after many hours. Very weird. Maybe libera servers are overloading?

schickling commented 1 year ago

Also affects #metabrainz

holdenger commented 1 year ago

Probably affects #fedora-cs too.

reivilibre commented 1 year ago

I had a look at the logs for matrix.org and libera.chat. Shay made an attempt to join on the 12th and this is what happened:

matrix.org

19:41:36,978: the /join request was routed to event_creator_users2 (POST-77368a818d0e645c-SJC)
- the room alias was successfully looked up on libera.chat in about 1 second (so not exactly #14462)
- it sent a request to the master in order to start a remote join
19:41:37,126: the remote join request arrived at the master process (POST-6133673)
- a /make_join completes within 600ms against libera.chat
- a /send_join request is started
- ... but then times out at 19:42:37,730
- (there are then 3 retries; the final one times out at 19:45:41,293)

libera.chat

19:41:40,592: first sign of something happening with the send_join: Received response to POST http://event-persister-repl:9092/_synapse/replication/fed_send_events/oaOXaVacsy: 200 (PUT-5881, federation reader 0)
- this is only ~4 seconds after the /join request was made by the person on matrix.org, so what went wrong? ...
- 19:42:37,739: Connection from client lost before response was sent
- the request finally completes at 20:01:27,582: Processed request: 1189.458sec/-1129.843sec (17.792sec, 0.652sec) (0.071sec/95.973sec/17) 0B 200! ... [20 dbevts]
  - 95 s of DB time is not great, 18 s of CPU time is not great ... but it's not obvious where the 1189 s of wall clock time is going??

(Note: all the retries are a little bit faster (likely caching); but still the fastest was 400 sec — not fast enough to respond in time.) (Another note: one of the retries says PUT-5944- Ratelimiter(matrix.org): queueing request (queue now 9 items) — I wonder if we're harming response times because of rate limiting? From the logs it's not clear how long the request is delayed for, though.)

Conclusion though: the Libera server isn't responding in time. With that said: the CPU and DB time graphs for federation reader 0 on Libera look very quiet — struggling to believe this is being overloaded. Similarly, the event persister looks quiet at that point in time — I don't expect that the issue is waiting for the join to be persisted.

summary:

The Libera server isn't responding in time.
I can't see why that is — the Synapse workers look pretty quiet at the time in question — if the issue was the resources needed to process the request, I'd expect to be able to see something being overloaded.

edit: Rich suspects the ratelimiter isn't working correctly.

DMRobertson commented 1 year ago

For xrefs: we suspect https://github.com/matrix-org/synapse/issues/14480 is related, if not the cause

erikjohnston commented 1 year ago

Le'ts re-investigate once libera.chat is on 1.75.0?

quite commented 1 year ago

Is this related to the failed bridging setup that happens towards various libera-channels? Like in: https://github.com/matrix-org/matrix-appservice-irc/issues/1652

DMRobertson commented 1 year ago

Le'ts re-investigate once libera.chat is on 1.75.0?

This should be done now---need to investigate if these problems are still occurring.

ht990332 commented 1 year ago

https://app.element.io/?updated=1.11.19#/room/#libera:libera.chat is still unable to join #libera.

quite commented 1 year ago

Is this related to the failed bridging setup that happens towards various libera-channels? Like in: matrix-org/matrix-appservice-irc#1652

No change for this issue. That issue is still with Scalar it seems.

Diablo-D3 commented 1 year ago

tea is now free from this bug.

erikjohnston commented 1 year ago

If people still see this, can you open a new issue and include logs. Thanks!

ara4n commented 1 year ago

https://github.com/matrix-org/synapse/issues/15115 is closely related to this. It wasn’t libera timing out tho in that instance

JeanPaulLucien commented 1 year ago

Maybe these issues are related too. https://github.com/vector-im/element-web/issues/24617 -> https://github.com/matrix-org/synapse/issues/15145 https://github.com/matrix-org/synapse/issues/15142

JeanPaulLucien commented 1 year ago

It is impossible to join #libera:libera.chat

It's possible to join: https://github.com/vector-im/element-web/issues/24482 I've collected the various errors from libera.chat. Seems it's random.

matrix-org / synapse