jitsi / jitsi-meet

Jitsi Meet - Secure, Simple and Scalable Video Conferences that you use as a standalone app or embed in your web application.
https://jitsi.org/meet
Apache License 2.0
23.2k stars 6.74k forks source link

Delayed reply to bosh requests #12212

Closed wfleischer closed 1 year ago

wfleischer commented 2 years ago

Description:

Replies to bosh requests are deferred to the arrival of the subsequent request.

Steps to reproduce:

  1. Local deploy the latest docker-jitsi-meet code (for convenience you can enforce bosh using these settings).
  2. Browse to https://localhost:8443/1
  3. Watch Network in the developer tools of your browser

Expected behavior:

Bosh requests should take milliseconds to be handled.

Actual behavior:

The reply to the request arrives on the next bosh request.

This is the log of the web container (values rt=10.010 uct="0.002" uht="10.010" urt="10.010" are as described here):

172.21.0.1 - - [21/Sep/2022:11:04:54 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 549 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.002 uct="0.000" uht="0.002" urt="0.002"
172.21.0.1 - - [21/Sep/2022:11:04:54 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 199 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.002 uct="0.000" uht="0.001" urt="0.001"
172.21.0.1 - - [21/Sep/2022:11:04:54 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 599 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.001 uct="0.000" uht="0.000" urt="0.000"
172.21.0.1 - - [21/Sep/2022:11:04:55 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 324 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.004 uct="0.000" uht="0.003" urt="0.003"
172.21.0.1 - - [21/Sep/2022:11:04:55 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 264 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.010 uct="0.000" uht="0.009" urt="0.009"
172.21.0.1 - - [21/Sep/2022:11:04:55 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 3425 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.002 uct="0.000" uht="0.001" urt="0.001"
172.21.0.1 - - [21/Sep/2022:11:04:55 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 652 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.009 uct="0.001" uht="0.009" urt="0.009"
172.21.0.1 - - [21/Sep/2022:11:05:05 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 310 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=9.786 uct="0.001" uht="9.786" urt="9.786"
172.21.0.1 - - [21/Sep/2022:11:05:05 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 148 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.200 uct="0.000" uht="0.200" urt="0.200"
172.21.0.1 - - [21/Sep/2022:11:05:05 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 519 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.068 uct="0.001" uht="0.068" urt="0.068"
172.21.0.1 - - [21/Sep/2022:11:05:05 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 2303 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.003 uct="0.000" uht="0.002" urt="0.002"
172.21.0.1 - - [21/Sep/2022:11:05:05 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 2790 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=0.003 uct="0.000" uht="0.002" urt="0.002"
172.21.0.1 - - [21/Sep/2022:11:05:25 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 310 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=19.480 uct="0.000" uht="19.475" urt="19.475"
172.21.0.1 - - [21/Sep/2022:11:05:35 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 310 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=10.003 uct="0.001" uht="10.003" urt="10.003"
172.21.0.1 - - [21/Sep/2022:11:05:45 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 310 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=9.998 uct="0.001" uht="9.999" urt="9.999"
172.21.0.1 - - [21/Sep/2022:11:05:55 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 310 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=10.019 uct="0.001" uht="10.019" urt="10.019"
172.21.0.1 - - [21/Sep/2022:11:06:05 +0000] "POST /http-bind?room=1 HTTP/2.0" 200 310 "https://localhost:8443/1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" rt=10.010 uct="0.002" uht="10.010" urt="10.010"

As one can see, every request takes as long as the time passed before the next request.

This is not an error in logging, as one can see on the client:

Screenshot 2022-09-21 at 13 19 19

Screenshot 2022-09-21 at 13 22 27

Additional Information

I came across this bug, when I was debugging why our monitoring system constantly reports responses times of our proxies close to 10s. The response time is somewhat limited to 10s because of the bosh ping being sent every 10s. That's probably why this problem does not break the functionality of bosh clients. Nevertheless, having bosh working properly will certainly improve the functionality.

wfleischer commented 2 years ago

The following prosody debug log might be helpful (especially the part Have nothing to say, so leaving request unanswered for now):

connSbtHdzxBLdZp                                             debug      New connection FD 16 (172.22.0.3, 56656, 172.22.0.2, 5280) on server FD 8 (*, 5280)
connSbtHdzxBLdZp                                             debug      Connected (FD 16 (172.22.0.3, 56656, 172.22.0.2, 5280))
runnerrKBvvaMbi7Gi                                           debug      creating new coroutine
http.server                                                  debug      Firing event: POST /http-bind
http.server                                                  debug      Firing event: POST meet.jitsi/http-bind
mod_bosh                                                     debug      Handling new request table: 0xaaab0b6a7a10: <body rid="1026384703" sid="310ffe4b-9891-4689-b972-0dad2b215ece" xmlns="http://jabber.org/protocol/httpbind"><iq id="53fc0cc8-8885-42a1-a256-ecd05f5d5ed7:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq></body>
----------
mod_bosh                                                     debug      BOSH body open (sid: 310ffe4b-9891-4689-b972-0dad2b215ece)
bosh310ffe4b-9891-4689-b972-0dad2b215ece                     debug      rid: 1026384703, sess: 1026384702, diff: 1
mod_bosh                                                     debug      BOSH stanza received: <iq to='meet.jitsi' xml:lang='en' id='53fc0cc8-8885-42a1-a256-ecd05f5d5ed7:sendIQ' type='get'>

bosh310ffe4b-9891-4689-b972-0dad2b215ece                     debug      Received[c2s]: <iq to='meet.jitsi' xml:lang='en' id='53fc0cc8-8885-42a1-a256-ecd05f5d5ed7:sendIQ' type='get'>
mod_bosh                                                     debug      We have an open request, so sending on that
mod_bosh                                                     debug      Request destroyed: table: 0xaaab0b680aa0
connXiBK0qcoIflH                                             debug      Close after writing remaining buffered data
mod_bosh                                                     debug      Session 310ffe4b-9891-4689-b972-0dad2b215ece has 1 out of 1 requests open
mod_bosh                                                     debug      and there are 0 things in the send_buffer:
mod_bosh                                                     debug      Have nothing to say, so leaving request unanswered for now
runnerrKBvvaMbi7Gi                                           debug      changed state from ready to waiting
connXiBK0qcoIflH                                             debug      Closing now
saghul commented 2 years ago

What's the response payload in the first request?

wfleischer commented 2 years ago

Hi @saghul. Here are the first few requests followed by their replies:

<body content="text/xml; charset=utf-8" hold="1" rid="761208308" to="meet.jitsi" ver="1.6" wait="60" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:version="1.0"/>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmpp:version='1.0' polling='5' inactivity='60' xmlns:stream='http://etherx.jabber.org/streams' requests='2' authid='74557782-1a69-4739-b020-59a4c2e9b03e' hold='1' wait='60' ver='1.6' secure='true' xmlns='http://jabber.org/protocol/httpbind' from='meet.jitsi' xmlns:xmpp='urn:xmpp:xbosh'><stream:features xmlns='jabber:client'><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mechanism>ANONYMOUS</mechanism></mechanisms><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/></stream:features></body>

<body rid="761208309" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><auth mechanism="ANONYMOUS" xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><success xmlns='urn:ietf:params:xml:ns:xmpp-sasl'/></body>

<body rid="761208310" sid="74557782-1a69-4739-b020-59a4c2e9b03e" to="meet.jitsi" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:restart="true"/>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><stream:features xmlns='jabber:client'><ver xmlns='urn:xmpp:features:rosterver'/><c hash='sha-1' xmlns='http://jabber.org/protocol/caps' node='http://prosody.im' ver='CStwOyzbh2KKhrh/v4d6qAH6+Cs='/><sub xmlns='urn:xmpp:features:pre-approval'/><bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'><required/></bind><session xmlns='urn:ietf:params:xml:ns:xmpp-session'><optional/></session><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/></stream:features></body>

<body rid="761208311" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><iq id="_bind_auth_2" type="set" xmlns="jabber:client"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/></iq></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><iq xmlns='jabber:client' id='_bind_auth_2' type='result'><bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'><jid>qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R</jid></bind></iq></body>

<body rid="761208312" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><iq id="_session_auth_2" type="set" xmlns="jabber:client"><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></iq></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><iq to='qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R' xmlns='jabber:client' id='_session_auth_2' type='result'/></body>

<body rid="761208313" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><iq id="e3de0937-6171-4b4c-81c5-377948fd602f:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><services xmlns="urn:xmpp:extdisco:2"/></iq><iq from="qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R" id="3ec1bb1d-f078-40af-83de-520c87c745d1:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><query xmlns="http://jabber.org/protocol/disco#info"/></iq></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><iq id='e3de0937-6171-4b4c-81c5-377948fd602f:sendIQ' type='error' xmlns='jabber:client' from='meet.jitsi' to='qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R'><error type='cancel'><service-unavailable xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></error></iq><iq id='3ec1bb1d-f078-40af-83de-520c87c745d1:sendIQ' type='result' xmlns='jabber:client' from='meet.jitsi' to='qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R'><query xmlns='http://jabber.org/protocol/disco#info'><identity name='lobby.meet.jitsi' category='component' type='lobbyrooms'/><identity name='breakout.meet.jitsi' category='component' type='breakout_rooms'/><identity name='endconference.&lt;no value&gt;' category='component' type='end_conference'/><identity name='conferenceduration.meet.jitsi' category='component' type='conference_duration'/><identity name='speakerstats.meet.jitsi' category='component' type='speakerstats'/><identity name='Prosody PubSub Service' category='pubsub' type='service'/><identity name='avmoderation.meet.jitsi' category='component' type='av_moderation'/><identity name='Prosody' category='server' type='im'/><feature var='vcard-temp'/><feature var='msgoffline'/><feature var='jabber:iq:register'/><feature var='urn:xmpp:ping'/><feature var='jabber:iq:version'/><feature var='urn:xmpp:time'/><feature var='jabber:iq:time'/><feature var='http://jabber.org/protocol/pubsub'/><feature var='http://jabber.org/protocol/pubsub#outcast-affiliation'/><feature var='http://jabber.org/protocol/pubsub#create-nodes'/><feature var='http://jabber.org/protocol/pubsub#persistent-items'/><feature var='http://jabber.org/protocol/pubsub#meta-data'/><feature var='http://jabber.org/protocol/pubsub#create-and-configure'/><feature var='http://jabber.org/protocol/pubsub#retrieve-items'/><feature var='http://jabber.org/protocol/pubsub#subscription-options'/><feature var='http://jabber.org/protocol/pubsub#publisher-affiliation'/><feature var='http://jabber.org/protocol/pubsub#modify-affiliations'/><feature var='http://jabber.org/protocol/pubsub#subscribe'/><feature var='http://jabber.org/protocol/pubsub#item-ids'/><feature var='http://jabber.org/protocol/pubsub#access-open'/><feature var='http://jabber.org/protocol/pubsub#retrieve-default'/><feature var='http://jabber.org/protocol/pubsub#publish-options'/><feature var='http://jabber.org/protocol/pubsub#multi-items'/><feature var='http://jabber.org/protocol/pubsub#delete-nodes'/><feature var='http://jabber.org/protocol/pubsub#retract-items'/><feature var='http://jabber.org/protocol/pubsub#delete-items'/><feature var='http://jabber.org/protocol/pubsub#publish'/><feature var='http://jabber.org/protocol/pubsub#instant-nodes'/><feature var='http://jabber.org/protocol/pubsub#member-affiliation'/><feature var='http://jabber.org/protocol/pubsub#config-node-max'/><feature var='http://jabber.org/protocol/pubsub#purge-nodes'/><feature var='http://jabber.org/protocol/pubsub#retrieve-subscriptions'/><feature var='http://jabber.org/protocol/pubsub#config-node'/><feature var='http://jabber.org/protocol/commands'/><feature var='jabber:iq:private'/><feature var='jabber:iq:last'/><feature var='jabber:iq:roster'/><feature var='http://jabber.org/protocol/disco#info'/><feature var='http://jabber.org/protocol/disco#items'/></query></iq></body>
Mupati commented 2 years ago

I don't know whether BOSH is a hard requirement in your case but when I switched from BOSH to Websocket, I stopped experiencing delays.

wfleischer commented 2 years ago

I don't know whether BOSH is a hard request in your case but when I switched from BOSH to Websocket, I no stopped experiencing delays.

Thanks, @Mupati, for your comment. That is indeed recommended. However, the mobile clients still use bosh.

saghul commented 2 years ago

I haven't had enough time to look into this yet @wfleischer but I quick observation: I noticed we don't set the protocol when proxying the request to Prosody. Maybe it's defaulting to HTTP 1.0 and no pipelining is happening? Can you try to force the version for BOSH like we do for WS?

wfleischer commented 2 years ago

I haven't had enough time to look into this yet @wfleischer but I quick observation: I noticed we don't set the protocol when proxying the request to Prosody. Maybe it's defaulting to HTTP 1.0 and no pipelining is happening? Can you try to force the version for BOSH like we do for WS?

Thanks @saghul. If adding "proxy_http_version 1.1;" to the /http-bind location is what you are thinking of, unfortunately that does not help.

(The Have nothing to say, so leaving request unanswered for now entry in the Prosody log to me rather indicates that something is wrong with the request handling on the Prosody side.)

damencho commented 2 years ago

https://github.com/bjc/prosody/blob/f19f1088b757c41c2c63b328f1d3faca8fe9a857/plugins/mod_bosh.lua#L184

I'm thinking that the prosody guys may help here ...

I think maybe @saghul meant to use https://boshurl vs //boshurl in config.js ...

saghul commented 2 years ago

Thanks @saghul. If adding "proxy_http_version 1.1;" to the /http-bind location is what you are thinking of, unfortunately that does not help.

Dang, that's unfortunate...

wfleischer commented 2 years ago

Maybe it does not really wait here: https://github.com/bjc/prosody/blob/f19f1088b757c41c2c63b328f1d3faca8fe9a857/plugins/mod_bosh.lua#L137 until the stream_callbacks finish and therefore the send_buffer is empty here: https://github.com/bjc/prosody/blob/f19f1088b757c41c2c63b328f1d3faca8fe9a857/plugins/mod_bosh.lua#L177, while it will be ready and sent with the next request.

saghul commented 2 years ago

Ping @mwild1 in case he catches this...

wfleischer commented 1 year ago

Here are the first bosh requests received from a web client:

<body content="text/xml; charset=utf-8" hold="1" rid="814448281" to="guest.meet.jitsi" ver="1.6" wait="60" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:version="1.0"/>
<body rid="814448282" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><auth mechanism="ANONYMOUS" xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/></body>
<body rid="814448283" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" to="guest.meet.jitsi" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:restart="true"/>
<body rid="814448284" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="_bind_auth_2" type="set" xmlns="jabber:client"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/></iq></body>
<body rid="814448285" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="_session_auth_2" type="set" xmlns="jabber:client"><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></iq></body>
<body rid="814448286" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="cfdd89a8-b9b0-4cf8-b1ee-eb45ad021e59:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><services xmlns="urn:xmpp:extdisco:2"/></iq><iq from="ok68hhmajpjruqym8uqi_u2_@guest.meet.jitsi/te9EAUyBZj9G" id="48eed14d-bbe3-4929-a64e-c09d2850cb46:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><query xmlns="http://jabber.org/protocol/disco#info"/></iq></body>
<body rid="814448287" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="ac008514-40fb-40eb-824f-4917a4ec3ef1:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><services xmlns="urn:xmpp:extdisco:1"/></iq><iq from="ok68hhmajpjruqym8uqi_u2_@guest.meet.jitsi/te9EAUyBZj9G" id="ca5b04c7-042e-4a35-8fcf-bc1813a8bddf:sendIQ" to="lobby.meet.jitsi" type="get" xmlns="jabber:client"><query node="lobbyrooms" xmlns="http://jabber.org/protocol/disco#info"/></iq></body>
<body rid="814448288" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"/>
<body rid="814448289" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="2b276da0-e002-447c-9974-3a982498bec7:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq></body>
<body rid="814448290" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="890cd43e-5228-407b-a6b9-2de9b3bd5cf4:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq></body>

Things become messy after this empty request is received: <body rid="814448288" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"/>. The request is not replied to nor is it destroyed. From there on every reply is sent on the previous request.

Do we have an idea why this request is sent to the server? Of course Prosody should correctly handle even such a request by destroying it, even if it has nothing to answer.

wfleischer commented 1 year ago

I could duplicate the behavior using this plain strophe example against alpha.jitsi.net. So at least we can tell, that it is not related to the Jitsi client. It must be a bug in strophe.js, Prosody or the Jitsi Prosody deployment.

wfleischer commented 1 year ago

So here Strophe is sending the empty request: https://github.com/strophe/strophejs/blob/master/src/bosh.js#L510

If there is no request within a 100 ms period, Strophe sends an empty request. Prosody does not handle that request well. It does not reply and does not remove the request either. @saghul: any idea whether Strophe or Prosody is violating https://xmpp.org/extensions/xep-0124.html?

wfleischer commented 1 year ago

The empty request is a standard means of keep alive: https://xmpp.org/extensions/xep-0124.html#technique

paweldomas commented 1 year ago

Yes, that empty request is the actual long polling part. That empty request is supposed to be used by the server to place a new message as soon as it becomes available.

wfleischer commented 1 year ago

To sum things up: Both services behave as they should. The reply to a request is not delayed, it is just sent as a response to the previous request. The new request is kept open and will be replied to as soon as there will be a response to the next request.

So replies are sent immediately as they are ready. It is just polluting the proxy (and browser) logs as from their point of view the response to a request took up to 10 seconds. In our logs haproxy reports an average response time of 5 seconds. Normally we would trigger an alert on such high response times of our backends. This means that BOSH now hides real issues with the backend response time. But this is not caused by the service – it is rather a weakness of the BOSH protocol. So we have to live with it :smile:

saghul commented 1 year ago

Do you have a reason for not using WS instead of BOSH?

wfleischer commented 1 year ago

My understanding of your posts in the community and https://github.com/jitsi/jitsi-meet/issues/8543#issuecomment-983700654 was, that it is not yet recommended for mobile clients. Did that change?

saghul commented 1 year ago

We are close to enabling it. You can use it today, the o lot downside is it may take a bit longer to detect disconnects, but connecting (specially to large meetings) is faster.

wfleischer commented 1 year ago

Thanks, @saghul, for your feedback! Good to know that it is stable enough. We will consider configuring a canary test.