videobridge crash SIGSEGV after SCTP resource temporary unavailable

[ptualek]

Currently, we are running jitsi-videobridge on

Instance:
VM with 24vCPU, 48GB Memory 

JVM parameter:

-Djava.net.preferIPv4Stack=true -Duser.timezone=Asia/Bangkok -Xmx16g -Xms16g -Xmn8g -XX:MaxMetaspaceSize=512m -XX:MetaspaceSize=512m -Djava.security.egd=file:/dev/./urandom -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge

OS: Ubuntu 18 LTS

Kernel version
# uname -a
Linux prod-vroom2-vdo-bridge-0001 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

JVB version: Jitsi-videobridge 2.1
# dpkg -l | grep jitsi
ii  jitsi-videobridge2                    2.1-157-g389b69ff-1                             all          WebRTC compatible Selective Forwarding Unit (SFU)

Symptom:

We are running multiple conferences on this VM and JVM is crash randomly.

Log file jvb.log show below:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f0163d1cb1f, pid=13174, tid=0x00007f0027fff700
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x18eb1f]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid13174.log

How to reproduce?

• Can't reproduce. It happen randomly.

We already check resources CPU, Memory everything look fine.

So we figured this out /tmp/hs_err_pid13174.log is using libjnisctp.so and we already check it is latest version of current code in this reposity.

Could someone help investigate the problem?

here is a link to the file hs_err_pid13174.log

Thanks,

Bond

[bbaldino]

We've seen instances of stuff like this, but it was years ago--and unfortunately we didn't find anything specifically that seemed to cause it/make it go away. Since then in the bridge we've moved to using websockets for the 'bridge channel' (instead of datachannels). I'd recommend looking into setting that up (see here)

[ptualek]

We've seen instances of stuff like this, but it was years ago--and unfortunately we didn't find anything specifically that seemed to cause it/make it go away. Since then in the bridge we've moved to using websockets for the 'bridge channel' (instead of datachannels). I'd recommend looking into setting that up (see here)

Hi, @bbaldino thanks for the response. I checked your links and try on tests it doesn't work. Seem like nginx can't commuicate with prosody.

Do you have other link that up-to-date document?

[bbaldino]

@ptualek let's start a thread in the community for debugging that.

[cherubimro]

Guys this seems to be exactly https://github.com/sctplab/usrsctp/issues/133 . It seems to be an old bug.

I opened this bug report: https://github.com/sctplab/usrsctp/issues/463 also without knowing of 133.

it seems the crash is in a macro definition of libstcp and it happens when webrtc clients are not terminating sessions gracefully. also , we were only able to use google chrome as a reliable webrtc client.

A

[cherubimro]

Will it go away if I enable websockets?

[bbaldino]

Will it go away if I enable websockets?

Yes, if you configure websockets on the bridge we won't use SCTP.

[cherubimro]

Will it go away if I enable websockets?

Yes, if you configure websockets on the bridge we won't use SCTP.

Thank you so much for replying.

I added these lines in the videobridge2 sip-communicator.properties file:

org.jitsi.videobridge.rest.jetty.tls.port=443 org.jitsi.videobridge.rest.jetty.sslContextFactory.keyStorePath=/etc/jitsi/videobridge/ssl.store org.jitsi.videobridge.rest.jetty.sslContextFactory.keyStorePassword=KEY_STORE_PASSWORD org.jitsi.videobridge.rest.COLIBRI_WS_TLS=true org.jitsi.videobridge.rest.COLIBRI_WS_DOMAIN=jitsidomain:443

I restarted jitsi-videobridge2 and now I don't see any SCTP lines in the jvb.log but I don't see websockets in the client console either. The debug console for the client writes that websockets are not enabled: [modules/xmpp/XmppConnection.js] <_maybeEnableStreamResume>: Stream resume enabled, but WebSockets are not enabled.

I did include openBridgeChannel: 'websocket' in config.js in /etc/jitsi/meet as the only entry by uncommenting the openBridgeChannel line which was set initially to true.

Am I doing something wrong? I am using debian 10 buster and everything is up to date. The file /etc/jitsi/videobridge/ssl.store is not even created. Conferencing works as usual.

Thank's for writing great software!

Alin A.

[bgrozev]

The debug console for the client writes that websockets are not enabled: [modules/xmpp/XmppConnection.js] <_maybeEnableStreamResume>: Stream resume enabled, but WebSockets are not enabled.

This is for the the XMPP connection (different websocket).

Do you see [modules/RTC/BridgeChannel.js] <WebSocket.e.onopen>: websocket channel opened in the client logs?

[cherubimro]

The debug console for the client writes that websockets are not enabled: [modules/xmpp/XmppConnection.js] <_maybeEnableStreamResume>: Stream resume enabled, but WebSockets are not enabled.

This is for the the XMPP connection (different websocket).

Do you see [modules/RTC/BridgeChannel.js] <WebSocket.e.onopen>: websocket channel opened in the client logs?

No, I don't. I did refresh the page a lot. On meet.jit.si I can see that it is using websockets but there is no mention of WebSocket.e.onopen. I followed the tutorial on https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md .

A