search

jitsi / jitsi-videobridge

Jitsi Videobridge is a WebRTC compatible video router or SFU that lets build highly scalable video conferencing infrastructure (i.e., up to hundreds of conferences per server).
https://jitsi.org/jitsi-videobridge
Apache License 2.0
2.91k stars 989 forks source link

error when I have >=33 users #716

Open manang opened 6 years ago

manang commented 6 years ago

Hi, I'm trying to test jitsi bridge in internal environment. I have the following error after the connection of 33 clients

`JVB 2018-09-25 16:32:23.211 SEVERE: [1354] util.UtilActivator.uncaughtException().122 An uncaught exception occurred in thread=Thread[IceUdpTransportManager connect thread,5,main] and message was: unable to create new native thread java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at org.jitsi.impl.neomedia.RTPConnectorOutputStream$Queue.(RTPConnectorOutputStream.java:808) at org.jitsi.impl.neomedia.RTPConnectorOutputStream$Queue.(RTPConnectorOutputStream.java:726) at org.jitsi.impl.neomedia.RTPConnectorOutputStream.(RTPConnectorOutputStream.java:261) at org.jitsi.impl.neomedia.RTPConnectorUDPOutputStream.(RTPConnectorUDPOutputStream.java:44) at org.jitsi.impl.neomedia.transform.TransformUDPOutputStream.(TransformUDPOutputStream.java:52) at org.jitsi.impl.neomedia.transform.RTPTransformUDPConnector.createDataOutputStream(RTPTransformUDPConnector.java:132) at org.jitsi.impl.neomedia.MediaStreamImpl$2.createDataOutputStream(MediaStreamImpl.java:2643) at org.jitsi.impl.neomedia.MediaStreamImpl$2.createDataOutputStream(MediaStreamImpl.java:2613) at org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:331) at org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:311) at org.jitsi.impl.neomedia.AbstractRTPConnector.addTarget(AbstractRTPConnector.java:109) at org.jitsi.impl.neomedia.MediaStreamImpl.doSetTarget(MediaStreamImpl.java:1454) at org.jitsi.impl.neomedia.MediaStreamImpl.rtpConnectorChanged(MediaStreamImpl.java:2516) at org.jitsi.impl.neomedia.MediaStreamImpl.setConnector(MediaStreamImpl.java:2696) at org.jitsi.videobridge.RtpChannel.maybeStartStream(RtpChannel.java:1053) at org.jitsi.videobridge.Channel.transportConnected(Channel.java:948) at java.lang.Iterable.forEach(Iterable.java:75) at org.jitsi.videobridge.IceUdpTransportManager.onIceConnected(IceUdpTransportManager.java:1751) at org.jitsi.videobridge.IceUdpTransportManager.lambda$startConnectivityEstablishment$5(IceUdpTransportManager.java:1956) at java.lang.Thread.run(Thread.java:748) OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed. JVB 2018-09-25 16:32:23.212 INFO: [1357] org.ice4j.ice.ConnectivityCheckClient.log() Nomination confirmed for pair: 10.100.84.33:10136/udp/host -> 10.100.84.85:6353/udp/host (stream.RTCP). Loal ufrag bl3ll1co8p64gp JVB 2018-09-25 16:32:23.212 INFO: [1357] org.ice4j.ice.CheckList.log() Selected pair for stream stream.RTCP: 10.100.84.33:10136/udp/host -> 10.100.84.85:6353/udp/host (stream.RTCP) JVB 2018-09-25 16:32:23.213 INFO: [1357] org.ice4j.ice.Agent.log() CheckList of stream stream is COMPLETED JVB 2018-09-25 16:32:23.213 INFO: [1357] org.ice4j.ice.Agent.log() ICE state changed from Running to Completed. Local ufrag bl3ll1co8p64gp JVB 2018-09-25 16:32:23.213 INFO: [1357] org.jitsi.videobridge.IceUdpTransportManager.log() CAT=stat ice_state_change,conf_id=6d8fa4737376d371,content=video,ch_id=616fe068880a56ca,endp_id=Jitsi-Hammer_33 old_state=Running,new_state=Completed OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fdb7f337000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) #

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.

JVB 2018-09-25 16:32:23.213 INFO: [1362] org.jitsi.videobridge.Channel.log() CAT=stat transport_connected,conf_id=6d8fa4737376d371,content=video,ch_id=616fe068880a56ca,endp_id=Jitsi-Hammer_33,stream=1650580574

An error report file with more information is saved as:

/home/ubuntu/jitsi-videobridge/target/jitsi-videobridge-linux-x64-1.1-SNAPSHOT/hs_err_pid12678.log`

do you have idea how to solve this issue? Thank you very much

Angelo

damencho commented 6 years ago

Which is the version of videobridge that you use? What is the available memory on the machine?

manang commented 6 years ago

it is the master branch, I compiled it from scratch. I have 64gb of ram

damencho commented 6 years ago

Can you reproduce it every time? If the answer is yes, can you create a heapdump and send it to us so we can analyze it. Send it to me privately damencho AT jitsi.org. 

jmap -dump:live,format=b,file=${HEAP_FILE} ${PID}

The command needs to be executed by the same user that is running the jvb.

manang commented 6 years ago

Hi, could you tell me how I can found the heap file name? Thank you Angelo

damencho commented 6 years ago

@manang Thanks I received the file, but cannot find anything suspicious at my initial check. Are you running jicofo and jvb on the same machine? What is the available RAM of that machine?

manang commented 6 years ago

Jicofo and jitsi videobridge are in the same server. My server has 64GB of ram.

The error is about the capability of create new threads.

We are suspecting that the problem is the hardware and not the software. Let me try to change the hardware.

Thank you for your support.

Angelo

From: Дамян Минков notifications@github.com Reply-To: jitsi/jitsi-videobridge reply@reply.github.com Date: Wednesday, September 26, 2018 at 5:03 PM To: jitsi/jitsi-videobridge jitsi-videobridge@noreply.github.com Cc: manang amantellini@gmail.com, Mention mention@noreply.github.com Subject: Re: [jitsi/jitsi-videobridge] error when I have >=33 users (#716)

@manang Thanks I received the file, but cannot find anything suspicious at my initial check. Are you running jicofo and jvb on the same machine? What is the available RAM of that machine?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

damencho commented 6 years ago

Hum are you running ubuntu 18.04? There is some restriction about threads per user. I will check it out what was the fix and will update you.

manang commented 6 years ago

No, ubuntu 16.04

From: Дамян Минков notifications@github.com Reply-To: jitsi/jitsi-videobridge reply@reply.github.com Date: Wednesday, September 26, 2018 at 5:29 PM To: jitsi/jitsi-videobridge jitsi-videobridge@noreply.github.com Cc: manang amantellini@gmail.com, Mention mention@noreply.github.com Subject: Re: [jitsi/jitsi-videobridge] error when I have >=33 users (#716)

Hum are you running ubuntu 18.04? There is some restriction about threads per user. I will check it out what was the fix and will update you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

damencho commented 6 years ago

Is it running with systemd? Try service jitsi-videobridge status and you will have something like: Tasks: 228 (limit: 30405) I suspect the limit is too low on your end.

To fix that in case of systemd: Make sure you have in /etc/systemd/logind.conf for example UserTasksMax=80% and in /etc/systemd/system.conf for example DefaultTasksMax=30405

manang commented 6 years ago

No, I’m launching directly the script jvb

From: Дамян Минков notifications@github.com Reply-To: jitsi/jitsi-videobridge reply@reply.github.com Date: Wednesday, September 26, 2018 at 6:08 PM To: jitsi/jitsi-videobridge jitsi-videobridge@noreply.github.com Cc: manang amantellini@gmail.com, Mention mention@noreply.github.com Subject: Re: [jitsi/jitsi-videobridge] error when I have >=33 users (#716)

Is it running with systemd? Try service jitsi-videobridge status and you will have something like: Tasks: 228 (limit: 30405) I suspect the limit is too low on your end.

To fix that in case of systemd: Make sure you have in /etc/systemd/logind.conf for example UserTasksMax=80% and in /etc/systemd/system.conf for example DefaultTasksMax=30405

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

damencho commented 6 years ago

But if you are using systemd these limits still apply, are you using it? You can check for example DefaultTasksMax value for the user which you use to launch jvb by executing systemctl show as that user.

manang commented 6 years ago

Hi, I'm not using systemd to launch the videobridge. thank you for your answer

Il giorno mer 26 set 2018 alle ore 18:28 Дамян Минков < notifications@github.com> ha scritto:

But if you are using systemd these limits still apply, are you using it? You can check for example DefaultTasksMax value for the user which you use to launch jvb by executing systemctl show as that user.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jitsi/jitsi-videobridge/issues/716#issuecomment-424781623, or mute the thread https://github.com/notifications/unsubscribe-auth/AFpk67fXAL3mFbLP3dIG6xuONTlJYXl5ks5ue6smgaJpZM4W5Czc .

damencho commented 6 years ago

I didn't mean for launching the videobridge, is systemd installed on the system, cause this limits then affects all users on the system.

manang commented 6 years ago

The problem was that the variable /proc/sys/vm/max_map_count was set to 3096 This value is too low, so you can reset that variable with sudo sysctl -w vm.max_map_count= this number has to be 1 every 128kb of system memory, for example

vm.max_map_count=2097152 on a 256 GB system.

In my case I set 500000 because I have 64gb of ram.

Thank you for your support

Angelo