Closed wfleischer closed 1 year ago
The following prosody debug log might be helpful (especially the part Have nothing to say, so leaving request unanswered for now
):
connSbtHdzxBLdZp debug New connection FD 16 (172.22.0.3, 56656, 172.22.0.2, 5280) on server FD 8 (*, 5280)
connSbtHdzxBLdZp debug Connected (FD 16 (172.22.0.3, 56656, 172.22.0.2, 5280))
runnerrKBvvaMbi7Gi debug creating new coroutine
http.server debug Firing event: POST /http-bind
http.server debug Firing event: POST meet.jitsi/http-bind
mod_bosh debug Handling new request table: 0xaaab0b6a7a10: <body rid="1026384703" sid="310ffe4b-9891-4689-b972-0dad2b215ece" xmlns="http://jabber.org/protocol/httpbind"><iq id="53fc0cc8-8885-42a1-a256-ecd05f5d5ed7:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq></body>
----------
mod_bosh debug BOSH body open (sid: 310ffe4b-9891-4689-b972-0dad2b215ece)
bosh310ffe4b-9891-4689-b972-0dad2b215ece debug rid: 1026384703, sess: 1026384702, diff: 1
mod_bosh debug BOSH stanza received: <iq to='meet.jitsi' xml:lang='en' id='53fc0cc8-8885-42a1-a256-ecd05f5d5ed7:sendIQ' type='get'>
bosh310ffe4b-9891-4689-b972-0dad2b215ece debug Received[c2s]: <iq to='meet.jitsi' xml:lang='en' id='53fc0cc8-8885-42a1-a256-ecd05f5d5ed7:sendIQ' type='get'>
mod_bosh debug We have an open request, so sending on that
mod_bosh debug Request destroyed: table: 0xaaab0b680aa0
connXiBK0qcoIflH debug Close after writing remaining buffered data
mod_bosh debug Session 310ffe4b-9891-4689-b972-0dad2b215ece has 1 out of 1 requests open
mod_bosh debug and there are 0 things in the send_buffer:
mod_bosh debug Have nothing to say, so leaving request unanswered for now
runnerrKBvvaMbi7Gi debug changed state from ready to waiting
connXiBK0qcoIflH debug Closing now
What's the response payload in the first request?
Hi @saghul. Here are the first few requests followed by their replies:
<body content="text/xml; charset=utf-8" hold="1" rid="761208308" to="meet.jitsi" ver="1.6" wait="60" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:version="1.0"/>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmpp:version='1.0' polling='5' inactivity='60' xmlns:stream='http://etherx.jabber.org/streams' requests='2' authid='74557782-1a69-4739-b020-59a4c2e9b03e' hold='1' wait='60' ver='1.6' secure='true' xmlns='http://jabber.org/protocol/httpbind' from='meet.jitsi' xmlns:xmpp='urn:xmpp:xbosh'><stream:features xmlns='jabber:client'><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mechanism>ANONYMOUS</mechanism></mechanisms><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/></stream:features></body>
<body rid="761208309" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><auth mechanism="ANONYMOUS" xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><success xmlns='urn:ietf:params:xml:ns:xmpp-sasl'/></body>
<body rid="761208310" sid="74557782-1a69-4739-b020-59a4c2e9b03e" to="meet.jitsi" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:restart="true"/>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><stream:features xmlns='jabber:client'><ver xmlns='urn:xmpp:features:rosterver'/><c hash='sha-1' xmlns='http://jabber.org/protocol/caps' node='http://prosody.im' ver='CStwOyzbh2KKhrh/v4d6qAH6+Cs='/><sub xmlns='urn:xmpp:features:pre-approval'/><bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'><required/></bind><session xmlns='urn:ietf:params:xml:ns:xmpp-session'><optional/></session><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/></stream:features></body>
<body rid="761208311" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><iq id="_bind_auth_2" type="set" xmlns="jabber:client"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/></iq></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><iq xmlns='jabber:client' id='_bind_auth_2' type='result'><bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'><jid>qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R</jid></bind></iq></body>
<body rid="761208312" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><iq id="_session_auth_2" type="set" xmlns="jabber:client"><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></iq></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><iq to='qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R' xmlns='jabber:client' id='_session_auth_2' type='result'/></body>
<body rid="761208313" sid="74557782-1a69-4739-b020-59a4c2e9b03e" xmlns="http://jabber.org/protocol/httpbind"><iq id="e3de0937-6171-4b4c-81c5-377948fd602f:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><services xmlns="urn:xmpp:extdisco:2"/></iq><iq from="qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R" id="3ec1bb1d-f078-40af-83de-520c87c745d1:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><query xmlns="http://jabber.org/protocol/disco#info"/></iq></body>
<body sid='74557782-1a69-4739-b020-59a4c2e9b03e' xmlns='http://jabber.org/protocol/httpbind' xmlns:stream='http://etherx.jabber.org/streams'><iq id='e3de0937-6171-4b4c-81c5-377948fd602f:sendIQ' type='error' xmlns='jabber:client' from='meet.jitsi' to='qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R'><error type='cancel'><service-unavailable xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></error></iq><iq id='3ec1bb1d-f078-40af-83de-520c87c745d1:sendIQ' type='result' xmlns='jabber:client' from='meet.jitsi' to='qearfttlkdkivzszu1ilmsrx@meet.jitsi/vdBK4aWTj20R'><query xmlns='http://jabber.org/protocol/disco#info'><identity name='lobby.meet.jitsi' category='component' type='lobbyrooms'/><identity name='breakout.meet.jitsi' category='component' type='breakout_rooms'/><identity name='endconference.<no value>' category='component' type='end_conference'/><identity name='conferenceduration.meet.jitsi' category='component' type='conference_duration'/><identity name='speakerstats.meet.jitsi' category='component' type='speakerstats'/><identity name='Prosody PubSub Service' category='pubsub' type='service'/><identity name='avmoderation.meet.jitsi' category='component' type='av_moderation'/><identity name='Prosody' category='server' type='im'/><feature var='vcard-temp'/><feature var='msgoffline'/><feature var='jabber:iq:register'/><feature var='urn:xmpp:ping'/><feature var='jabber:iq:version'/><feature var='urn:xmpp:time'/><feature var='jabber:iq:time'/><feature var='http://jabber.org/protocol/pubsub'/><feature var='http://jabber.org/protocol/pubsub#outcast-affiliation'/><feature var='http://jabber.org/protocol/pubsub#create-nodes'/><feature var='http://jabber.org/protocol/pubsub#persistent-items'/><feature var='http://jabber.org/protocol/pubsub#meta-data'/><feature var='http://jabber.org/protocol/pubsub#create-and-configure'/><feature var='http://jabber.org/protocol/pubsub#retrieve-items'/><feature var='http://jabber.org/protocol/pubsub#subscription-options'/><feature var='http://jabber.org/protocol/pubsub#publisher-affiliation'/><feature var='http://jabber.org/protocol/pubsub#modify-affiliations'/><feature var='http://jabber.org/protocol/pubsub#subscribe'/><feature var='http://jabber.org/protocol/pubsub#item-ids'/><feature var='http://jabber.org/protocol/pubsub#access-open'/><feature var='http://jabber.org/protocol/pubsub#retrieve-default'/><feature var='http://jabber.org/protocol/pubsub#publish-options'/><feature var='http://jabber.org/protocol/pubsub#multi-items'/><feature var='http://jabber.org/protocol/pubsub#delete-nodes'/><feature var='http://jabber.org/protocol/pubsub#retract-items'/><feature var='http://jabber.org/protocol/pubsub#delete-items'/><feature var='http://jabber.org/protocol/pubsub#publish'/><feature var='http://jabber.org/protocol/pubsub#instant-nodes'/><feature var='http://jabber.org/protocol/pubsub#member-affiliation'/><feature var='http://jabber.org/protocol/pubsub#config-node-max'/><feature var='http://jabber.org/protocol/pubsub#purge-nodes'/><feature var='http://jabber.org/protocol/pubsub#retrieve-subscriptions'/><feature var='http://jabber.org/protocol/pubsub#config-node'/><feature var='http://jabber.org/protocol/commands'/><feature var='jabber:iq:private'/><feature var='jabber:iq:last'/><feature var='jabber:iq:roster'/><feature var='http://jabber.org/protocol/disco#info'/><feature var='http://jabber.org/protocol/disco#items'/></query></iq></body>
I don't know whether BOSH is a hard requirement in your case but when I switched from BOSH to Websocket, I stopped experiencing delays.
I don't know whether BOSH is a hard request in your case but when I switched from BOSH to Websocket, I no stopped experiencing delays.
Thanks, @Mupati, for your comment. That is indeed recommended. However, the mobile clients still use bosh.
I haven't had enough time to look into this yet @wfleischer but I quick observation: I noticed we don't set the protocol when proxying the request to Prosody. Maybe it's defaulting to HTTP 1.0 and no pipelining is happening? Can you try to force the version for BOSH like we do for WS?
I haven't had enough time to look into this yet @wfleischer but I quick observation: I noticed we don't set the protocol when proxying the request to Prosody. Maybe it's defaulting to HTTP 1.0 and no pipelining is happening? Can you try to force the version for BOSH like we do for WS?
Thanks @saghul. If adding "proxy_http_version 1.1;" to the /http-bind location is what you are thinking of, unfortunately that does not help.
(The Have nothing to say, so leaving request unanswered for now
entry in the Prosody log to me rather indicates that something is wrong with the request handling on the Prosody side.)
I'm thinking that the prosody guys may help here ...
I think maybe @saghul meant to use https://boshurl vs //boshurl in config.js ...
Thanks @saghul. If adding "proxy_http_version 1.1;" to the /http-bind location is what you are thinking of, unfortunately that does not help.
Dang, that's unfortunate...
Maybe it does not really wait here: https://github.com/bjc/prosody/blob/f19f1088b757c41c2c63b328f1d3faca8fe9a857/plugins/mod_bosh.lua#L137 until the stream_callbacks finish and therefore the send_buffer is empty here: https://github.com/bjc/prosody/blob/f19f1088b757c41c2c63b328f1d3faca8fe9a857/plugins/mod_bosh.lua#L177, while it will be ready and sent with the next request.
Ping @mwild1 in case he catches this...
Here are the first bosh requests received from a web client:
<body content="text/xml; charset=utf-8" hold="1" rid="814448281" to="guest.meet.jitsi" ver="1.6" wait="60" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:version="1.0"/>
<body rid="814448282" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><auth mechanism="ANONYMOUS" xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/></body>
<body rid="814448283" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" to="guest.meet.jitsi" xml:lang="en" xmlns="http://jabber.org/protocol/httpbind" xmlns:xmpp="urn:xmpp:xbosh" xmpp:restart="true"/>
<body rid="814448284" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="_bind_auth_2" type="set" xmlns="jabber:client"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/></iq></body>
<body rid="814448285" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="_session_auth_2" type="set" xmlns="jabber:client"><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></iq></body>
<body rid="814448286" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="cfdd89a8-b9b0-4cf8-b1ee-eb45ad021e59:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><services xmlns="urn:xmpp:extdisco:2"/></iq><iq from="ok68hhmajpjruqym8uqi_u2_@guest.meet.jitsi/te9EAUyBZj9G" id="48eed14d-bbe3-4929-a64e-c09d2850cb46:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><query xmlns="http://jabber.org/protocol/disco#info"/></iq></body>
<body rid="814448287" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="ac008514-40fb-40eb-824f-4917a4ec3ef1:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><services xmlns="urn:xmpp:extdisco:1"/></iq><iq from="ok68hhmajpjruqym8uqi_u2_@guest.meet.jitsi/te9EAUyBZj9G" id="ca5b04c7-042e-4a35-8fcf-bc1813a8bddf:sendIQ" to="lobby.meet.jitsi" type="get" xmlns="jabber:client"><query node="lobbyrooms" xmlns="http://jabber.org/protocol/disco#info"/></iq></body>
<body rid="814448288" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"/>
<body rid="814448289" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="2b276da0-e002-447c-9974-3a982498bec7:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq></body>
<body rid="814448290" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"><iq id="890cd43e-5228-407b-a6b9-2de9b3bd5cf4:sendIQ" to="meet.jitsi" type="get" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq></body>
Things become messy after this empty request is received: <body rid="814448288" sid="da1b4a38-6b9d-47f4-9754-235c8c4999a8" xmlns="http://jabber.org/protocol/httpbind"/>
. The request is not replied to nor is it destroyed. From there on every reply is sent on the previous request.
Do we have an idea why this request is sent to the server? Of course Prosody should correctly handle even such a request by destroying it, even if it has nothing to answer.
I could duplicate the behavior using this plain strophe example against alpha.jitsi.net. So at least we can tell, that it is not related to the Jitsi client. It must be a bug in strophe.js, Prosody or the Jitsi Prosody deployment.
So here Strophe is sending the empty request: https://github.com/strophe/strophejs/blob/master/src/bosh.js#L510
If there is no request within a 100 ms period, Strophe sends an empty request. Prosody does not handle that request well. It does not reply and does not remove the request either. @saghul: any idea whether Strophe or Prosody is violating https://xmpp.org/extensions/xep-0124.html?
The empty request is a standard means of keep alive: https://xmpp.org/extensions/xep-0124.html#technique
Yes, that empty request is the actual long polling part. That empty request is supposed to be used by the server to place a new message as soon as it becomes available.
To sum things up: Both services behave as they should. The reply to a request is not delayed, it is just sent as a response to the previous request. The new request is kept open and will be replied to as soon as there will be a response to the next request.
So replies are sent immediately as they are ready. It is just polluting the proxy (and browser) logs as from their point of view the response to a request took up to 10 seconds. In our logs haproxy reports an average response time of 5 seconds. Normally we would trigger an alert on such high response times of our backends. This means that BOSH now hides real issues with the backend response time. But this is not caused by the service – it is rather a weakness of the BOSH protocol. So we have to live with it :smile:
Do you have a reason for not using WS instead of BOSH?
My understanding of your posts in the community and https://github.com/jitsi/jitsi-meet/issues/8543#issuecomment-983700654 was, that it is not yet recommended for mobile clients. Did that change?
We are close to enabling it. You can use it today, the o lot downside is it may take a bit longer to detect disconnects, but connecting (specially to large meetings) is faster.
Thanks, @saghul, for your feedback! Good to know that it is stable enough. We will consider configuring a canary test.
Description:
Replies to bosh requests are deferred to the arrival of the subsequent request.
Steps to reproduce:
Expected behavior:
Bosh requests should take milliseconds to be handled.
Actual behavior:
The reply to the request arrives on the next bosh request.
This is the log of the web container (values rt=10.010 uct="0.002" uht="10.010" urt="10.010" are as described here):
As one can see, every request takes as long as the time passed before the next request.
This is not an error in logging, as one can see on the client:
Additional Information
I came across this bug, when I was debugging why our monitoring system constantly reports responses times of our proxies close to 10s. The response time is somewhat limited to 10s because of the bosh ping being sent every 10s. That's probably why this problem does not break the functionality of bosh clients. Nevertheless, having bosh working properly will certainly improve the functionality.