search

jitsi / jitsi-meet

Jitsi Meet - Secure, Simple and Scalable Video Conferences that you use as a standalone app or embed in your web application.
https://jitsi.org/meet
Apache License 2.0
23.32k stars 6.76k forks source link

ICE connectivity check failures #3738

Closed ghost closed 5 years ago

ghost commented 5 years ago

Description

In a two-server environment, one host is running jitsi-meet (from a recently jitsi nightly deb package), with some small configuration modifications detailed below, and another host is running webrtc-streamer (with a small fix applied as per #688).

The streamer client appears to join the room and the browser client expands a full black screen to commence video output, but connectivity from the streamer client drops out, reproducibly.

There are some existing reports (both in the Jitsi community forum, and under the https://github.com/jitsi repo issue trackers) of similar issues and most of them point towards issues with NAT and ICE configuration - in this case the server is configured with near-default settings. The modifications from default are:

No firewalling is present on the jitsi-meet server. Attempts to use each of the following configuration keys (often in combination) in /etc/jitsi/videobridge/sip-communicator.properties have been attempted but without any success so far:

The following similar reading references have been found:

Current behavior

connectivity-issues

Expected Behavior

Possible Solution

This seems potentially related to ICE and client connectivity checks. The following warnings appear in the videobridge logs:

JVB 2018-12-26 01:05:46.475 INFO: [266] org.ice4j.ice.ConnectivityCheckClient.log() Error response for pair: 192.168.2.126:10000/udp/host -> 192.168.2.126:41487/udp/prflx (stream.RTP), failing.  Code = 401(class=4; number=1)

The error code 401 from these logs appears to be retrieved in ice4 and documented in RFC 3489.

In the client, which builds upon the Google webrtc project, warnings of the following format appear:

[042:060] [12565] (port.cc:614): Port[e807f890:video:1:0:local:Net[wlan0:192.168.2.x/24:Unknown:id=3]]: Received STUN request with bad local username gZhD from 192.168.2.x:10000

Steps to reproduce

NB: This same behavior can be observed by running webrtc-streamer locally against the cloud-hosted https://meet.jit.si instance - however server logs are not available in that case.

Environment details

ghost commented 5 years ago

Attaching jvb.log

ghost commented 5 years ago

https://community.jitsi.org/t/jvb-how-to-listen-to-port-443/15017 also looks relevant, but seems to suggest that setting up two separate machines is required - one for JVB and one for the jitsi-meet webserver.

Is there a convenient way to have everything work 'out of the box' on a single host, or with minor configuration changes? (it'd seem beneficial if possible - allowing new customers to onboard more easily and saving support overhead)

Until discovering the root cause I'm not entirely clear why single-host seems challenging (or if that indeed is the problem I'm hitting here)

damencho commented 5 years ago

Is your jvb running behind NAT?

rasos commented 5 years ago

The standard jitsi meet set-up on one machine does not allow to harvest TCP 443, it falls back to 4443 as 443 is being used by nginx. This may not affect your usage scenario. Making use of TCP 443 for media transport is supposed to be a "last resort" in case a user is sitting behind a very restricted firewall.

For making use of TCP 443 for media transport you either split signalling (=jicofo+prosody+nginx) and videobridge on two servers, so that the latter also can bind to a 443 port OR you try to replace nginx with jetty web server - however we have not been able to make that work https://git.fairkom.net/hosting/fairmeeting/issues/17

ghost commented 5 years ago

Thanks @damencho @rasos for your responses.

In this case there is no NAT involved - the jitsi-meet server is listening with apache2 on 443 (using the localhost.conf bundled with the jitsi-meet deb package, with an extra line to enable CORS requests from the client).

In the jvb.log attached earlier I'm trying org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT=443 and org.jitsi.videobridge.TCP_HARVESTER_PORT=4443 although I haven't changed any configuration elsewhere (i.e. no firewall rules or apache config changes), so I'm not sure if that's correct.

I'd like to get this working on a single host if possible, partly because I'd like to understand what's required for that, but thank you for the ticket reference @rasos, I'll try split signalling if I can't make further progress.

damencho commented 5 years ago

I don't get why you need to touch these: org.jitsi.videobridge.TCP_HARVESTER_PORT org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS

What is your idea?

So NAT_HARVESTER_LOCAL_ADDRESS and NAT_HARVESTER_PUBLIC_ADDRESS, need to be set if jvb does not have locally configured its public address, cause jvb needs to add it in signaling. TCP_HARVESTER_PORT is to change the port used to listen for TCP connections, which by default is either 443 or 4443 depends on the webserver used. TCP_HARVESTER_MAPPED_PORT is used to change the TCP port announced in the signaling, but the local used port for listening is not changed, this is normally used with some NAP mapping or iptables forwarding.

ghost commented 5 years ago

@damencho I'm fairly confident that the NAT settings should not be required; in this issue I'm reproducing everything within a local network, and JVB starts up and listens on an IP and port which the client can access:

# ps faux|grep jvb|more
root      8777  0.0  0.0   9384   784 pts/3    S+   08:18   0:00              |             
      \_ tail -F /var/log/jitsi/jicofo.log /var/log/jitsi/jvb.log /var/log/prosody/prosody.l
og
root      9162  0.0  0.0  16308  1096 pts/4    S+   08:19   0:00                            
      \_ grep --color=auto jvb
jvb       8608  6.2  3.3 6801840 128608 ?      Ssl  08:18   0:05 java -Xmx3072m -XX:+HeapDum
pOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi-videobridge/l
ib/native/linux-64 -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.si
p.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=
/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties -cp
 /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/agafua-
syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/annotations-15.0.jar:/usr/share/jitsi-videob
ridge/lib/bccontrib-1.0.jar:/usr/share/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/sh
are/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats

root@point:~# netstat -lnp|grep 8608
tcp6       0      0 192.168.2.126:443       :::*                    LISTEN      8608/java           
udp6       0      0 192.168.2.126:10000     :::*                                8608/java           
udp6       0      0 :::55942                :::*                                8608/java

My plan / thinking with the overriden TCP harvesting settings was that perhaps the client wasn't aware of the 443 -> 4443 fallback and that a mapping could help. On second thoughts I don't think that should be required. Attached here is a jvb.log where I've removed those settings.

jvb.log

Next I will try the NAT_HARVESTER_* settings, although again - all of this is network-local on a single host, so I don't believe any NAT tricks should be required here.

ghost commented 5 years ago

@damencho My mistake; the network paste above was from a previous execution (where I had moved apache2 to 4443; this didn't work either). Here it is again from the configuration matching https://github.com/jitsi/jitsi-meet/files/2712878/jvb.log

# ps faux|grep jvb|more
root     12201  0.0  0.0  16308  1044 pts/3    S+   08:31   0:00              |                   \_ grep --color=auto jvb
jvb      11798  6.7  3.4 6801840 131108 ?      Ssl  08:30   0:05 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi
-videobridge/lib/native/linux-64 -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.commu
nicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties -cp /usr/share/jitsi-videobridge/jitsi-videobridge.ja
r:/usr/share/jitsi-videobridge/lib/agafua-syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/annotations-15.0.jar:/usr/share/jitsi-videobridge/lib/bccontrib-1.0.jar:/usr/sha
re/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats-java-sdk-5.0.0.jar:/u
sr/share/jitsi-videobridge/lib/cglib-nodep-2.2.2.jar:/usr/share/jitsi-videobridge/lib/commons-codec-1.6.jar:/usr/share/jitsi-videobridge/lib/commons-lang3-3.1.jar:/usr/sh
are/jitsi-videobridge/lib/commons-logging-1.2.jar:/usr/share/jitsi-videobridge/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jitsi-videobridge/lib/core-2.0.1.ja
r:/usr/share/jitsi-videobridge/lib/dnsjava-2.1.7.jar:/usr/share/jitsi-videobridge/lib/dom4j-1.6.1.jar:/usr/share/jitsi-videobridge/lib/fmj-1.0-20161207.221530-23.jar:/usr
/share/jitsi-videobridge/lib/gson-2.3.1.jar:/usr/share/jitsi-videobridge/lib/guava-15.0.jar:/usr/share/jitsi-videobridge/lib/httpclient-4.4.jar:/usr/share/jitsi-videobrid
ge/lib/httpcore-4.4.jar:/usr/share/jitsi-videobridge/lib/ice4j-2.0.0-SNAPSHOT.jar:/usr/share/jitsi-videobridge/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/sha
re/jitsi-videobridge/lib/java-dogstatsd-client-2.5.jar:/usr/share/jitsi-videobridge/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jitsi-videobridge/lib/javax.servlet-api-3.
1.0.jar:/usr/share/jitsi-videobridge/lib/jbosh-0.9.0.jar:/usr/share/jitsi-videobridge/lib/jcip-annotations-1.0.jar:/usr/share/jitsi-videobridge/lib/jcl-core-2.8.jar:/usr/
share/jitsi-videobridge/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-continuation-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/je
tty-http-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-io-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/
jitsi-videobridge/lib/jetty-rewrite-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-security-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-serve
r-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlet-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlets-9.2.10.v20150310.jar:/usr/share
/jitsi-videobridge/lib/jetty-util-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jetty-xml-7.0.1.
v20091125.jar:/usr/share/jitsi-videobridge/lib/jicoco-1.1-20180821.201527-7.jar:/usr/share/jitsi-videobridge/lib/jitsi-android-osgi-1.0-20180322.162617-2.jar:/usr/share/j
itsi-videobridge/lib/jitsi-configuration-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jit
si-fileaccess-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-netaddr-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-packetlogging-2.13.f6042d3.jar:/usr/
share/jitsi-videobridge/lib/jitsi-protocol-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-jabber-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi
-protocol-media-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-resourcemanager-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-stats-1.0-20180523.024051-
5.jar:/usr/share/jitsi-videobridge/lib/jitsi-ui-service-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-util-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jna
-4.1.0.jar:/usr/share/jitsi-videobridge/lib/jnsapi-0.0.3-jitsi-smack4.2-3.jar:/usr/share/jitsi-videobridge/lib/jose4j-0.5.1.jar:/usr/share/jitsi-videobridge/lib/json-simp
le-1.1.1.jar:/usr/share/jitsi-videobridge/lib/jul-to-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/jxmpp-core-0.6.0.jar:/usr/share/jitsi-videobridge/lib/jxmpp-jid-0.6.
0.jar:/usr/share/jitsi-videobridge/lib/jxmpp-util-cache-0.6.0.jar:/usr/share/jitsi-videobridge/lib/libidn-1.15.jar:/usr/share/jitsi-videobridge/lib/libjitsi-1.0-20181119.
181042-364.jar:/usr/share/jitsi-videobridge/lib/log4j-1.2.15.jar:/usr/share/jitsi-videobridge/lib/log4j-api-2.3.jar:/usr/share/jitsi-videobridge/lib/log4j-core-2.3.jar:/u
sr/share/jitsi-videobridge/lib/object-cloner-0.1.jar:/usr/share/jitsi-videobridge/lib/objenesis-2.2.jar:/usr/share/jitsi-videobridge/lib/okhttp-3.9.1.jar:/usr/share/jitsi
-videobridge/lib/okio-1.13.0.jar:/usr/share/jitsi-videobridge/lib/orange-extensions-1.3.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.framework-4.4.0.jar:/usr/s
hare/jitsi-videobridge/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.osgi.core-4.3.1.jar:/usr/share/jitsi-videobridge/lib/osgi-over-slf4j-1.7.7
.jar:/usr/share/jitsi-videobridge/lib/sdes4j-1.1.3.jar:/usr/share/jitsi-videobridge/lib/sdp-api-1.0.jar:/usr/share/jitsi-videobridge/lib/sigar-1.6.4.jar:/usr/share/jitsi-
videobridge/lib/slf4j-api-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-simple-1.6.1.jar:/usr/share/jitsi-videob
ridge/lib/smack-bosh-4.2.1.jar:/usr/share/jitsi-videobridge/lib/smack-core-4.2.2-b1c107f.jar:/usr/share/jitsi-videobridge/lib/smack-debug-4.2.1.jar:/usr/share/jitsi-video
root@point:/etc/apache2# netstat -lnp|grep 11798
tcp6       0      0 192.168.2.126:4443      :::*                    LISTEN      11798/java          
udp6       0      0 192.168.2.126:10000     :::*                                11798/java          
udp6       0      0 :::38104                :::*                                11798/java
damencho commented 5 years ago

Does your deployment have its public address on the local machine, the address that matches the dns you use to access the deployment? If you open 3 tabs on your PC, does it work?

ghost commented 5 years ago

@damencho Good check, thanks - yep, the public address is on the local machine. While testing locally I'm using localhost as the DNS name currently, and apache2 is listening there on port 443.

When I browse to https://localhost/testing in three separate windows, I'm able to share content in one screen and I see it correctly in the second and third windows.

ghost commented 5 years ago

Next configuration:

# cat /etc/jitsi/videobridge/sip-communicator.properties 
org.jitsi.videobridge.AUTHORIZED_SOURCE_REGEXP=focus@auth.localhost/.*
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.2.126
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=192.168.2.126

Results - same behavior:

jvb-natconfig.log

damencho commented 5 years ago

org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.2.126 org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=192.168.2.126

This is not correct, remove those lines if you are not putting the public address there. From your words, I understand that you test everything in the same network and there is no public address. If this is the case you don't need any setting. Seems to me that the streamer thing is not able to connect and send media for some reason, maybe the warning/errors you pasted above I don't know. This needs more debugging.

Open chrome://webrtc-internals and find the streamer ssrc/streams and see is anything received at all.

ghost commented 5 years ago

@damencho I'm not sure exactly what to look for here, but it does appear that some streams are being created in the human participant chrome browser:

chrome-streams

Let me know if I should dump more of this for debugging purposes.

damencho commented 5 years ago

Check recv ssrc, are there any bytes received

ghost commented 5 years ago

ssrc-recv

@damencho it appears not - bytesReceived is zero

ghost commented 5 years ago

@damencho any recommendations on next debugging steps? Glad to provide further details and investigation while this is fresh in my mind. For what it's worth, I do also have a local checkout of ice4j and can apply any relevant patches there for testing purposes (some of the previous log messages led me to believe the problem might be in that area)

damencho commented 5 years ago

Any errors on the other side, why ice fails there ... Not sure how ti debug the other client you use ...

ghost commented 5 years ago

@damencho Sounds good; perhaps I'll have to follow this in a chrome-webrtc issue tracker too; I'll leave this issue open until finding out more

ghost commented 5 years ago

Adding DTLS 1.2 support in libjitsi (by way of recompiling a custom bc-java as per the notes in https://github.com/jitsi/libjitsi/issues/441) seems to help a lot here. That said, there are other issues when trying to use the latest head version of bc-java at the time of writing, so I'm trying to narrow in on any commits which provide a workable system.

ghost commented 5 years ago

Just for the record:

[005:363] [16909] (opensslstreamadapter.cc:784): BeginSSL with peer.
[005:363] [16909] (opensslstreamadapter.cc:841): ContinueSSL
[005:364] [16909] (opensslstreamadapter.cc:872):  -- error want read
stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.