Problem on video recording , Chrome tab crashed , SESSION Chrome hung

[kelvinyasu]

There is some problem while I do recording... any idea what's the problem?

2020-04-08 17:39:19.523 INFO: [931] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {audio={download=173, upload=22}, download=2014, upload=22, video={download=1841, upload=0}}, all clients muted? false
2020-04-08 17:39:30.087 FINE: [931] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-04-08 17:39:35.265 INFO: [931] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {audio={download=154, upload=22}, download=1706, upload=22, video={download=1552, upload=0}}, all clients muted? false
2020-04-08 17:39:48.599 FINE: [931] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-04-08 17:39:49.947 INFO: [931] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {audio={download=166, upload=18}, download=1532, upload=18, video={download=1356, upload=0}}, all clients muted? false
2020-04-08 17:39:59.542 FINE: [931] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-04-08 17:40:02.559 FINE: [931] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-04-08 17:40:07.787 FINE: [931] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-04-08 17:40:48.362 SEVERE: [931] org.jitsi.jibri.selenium.JibriSelenium.invoke() Error while running call status checks: org.openqa.selenium.WebDriverException: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
  (Session info: chrome=80.0.3987.163)
  (Driver info: chromedriver=80.0.3987.106 (f68069574609230cf9b635cd784cfb1bf81bb53a-refs/branch-heads/3987@{#882}),platform=Linux 4.15.0-96-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '0f3b8603eeb4', ip: '172.18.0.4', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-96-generic', java.version: '1.8.0_242'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 80.0.3987.106 (f68069574609..., userDataDir: /tmp/.com.google.Chrome.9Vdigt}, cssSelectorsEnabled: true, databaseEnabled: false, goog:chromeOptions: {debuggerAddress: localhost:41887}, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), rotatable: false, setWindowRect: true, strictFileInteractability: false, takesHeapSnapshot: true, takesScreenshot: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unexpectedAlertBehaviour: ignore, unhandledPromptBehavior: ignore, version: 80.0.3987.163, webStorageEnabled: true}
Session ID: 37103ea83e103537abc8040eadbc72c8 with stack: 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:480)
org.jitsi.jibri.selenium.pageobjects.CallPage.getNumParticipants(CallPage.kt:71)
org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.isCallEmpty(EmptyCallStatusCheck.kt:44)
org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.run(EmptyCallStatusCheck.kt:29)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1$event$1.invoke(JibriSelenium.kt:201)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1$event$1.invoke(JibriSelenium.kt:112)
kotlin.sequences.TransformingSequence$iterator$1.next(Sequences.kt:172)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1.invoke(JibriSelenium.kt:301)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1.invoke(JibriSelenium.kt:112)
org.jitsi.jibri.util.extensions.SchedulerExecutorServiceExtsKt$sam$java_lang_Runnable$0.run(SchedulerExecutorServiceExts.kt)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-04-08 17:40:48.364 INFO: [2169] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Running to Error: SESSION Chrome hung
2020-04-08 17:40:48.365 INFO: [2169] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Running to Error: SESSION Chrome hung
2020-04-08 17:40:48.365 INFO: [2169] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq <iq to='jibribrewery@internal-muc.meet.ude.hk/focus' id='5G7pQ-4275' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error'/></iq>
2020-04-08 17:40:48.367 FINE: [2169] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-04-08 17:40:48.367 INFO: [2169] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-04-08 17:40:48.368 INFO: [2169] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-04-08 17:40:48.368 INFO: [2169] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-04-08 17:40:48.413 SEVERE: [931] org.jitsi.jibri.selenium.pageobjects.CallPage.numRemoteParticipantsMuted() error running numRemoteParticipantsMuted script: {error=invalid session id, message=invalid session id, stacktrace=#0 0x564586f6ad29 <unknown>
} class com.google.common.collect.Maps$TransformedEntriesMap

[rodrigopavezi]

I have the same issue happening on a kubernetes container. I tried giving it more Memory and CPU but still chrome tab is crashing.

[danilorsilva]

@rodrigopavezi Did you install jitsi on top of kubernetes? Is it working with 3 or more people on conference?

[rodrigopavezi]

@danilorsilva yes I got it working. I am planning to push here a branch with the helm I created. I hope to get it here later today. I will ping you when I get it done.

[danilorsilva]

If you can send me the yaml i would be very happy.

[rodrigopavezi]

@danilorsilva there you go https://github.com/jitsi/docker-jitsi-meet/pull/415. Have fun!! :-)

[danilorsilva]

Thanks @rodrigopavezi !

[zWaR]

We're experiencing something similar too: https://community.jitsi.org/t/jibri-chrome-hangs-and-crashes/80874

What we've found so far at least is that it looks like crashes could be caused by Chrome running out of space for /dev/shm. With the flags for Chrome in Jibri config (--no-sandbox or --disable-dev-shm-usage) it's possible to prevent Chrome from using space in /dev/shm but we're not sure if that can cause some other issues down the line. It would be helpful to receive some additional comments on /dev/shm usage and whether or not it's safe to prevent Chrome using it.

@rodrigopavezi does your memory increase for Docker container also proportionally increases allocation for /dev/shm? How many MBs do you allocate for it? As mentioned in the community forums, we're allocating 64MB for now.

Thanks for the input!

[thapakazi]

Any findings on these ?

I am getting similar result after recording for some while with just 2 users,

I see some logs on dmesg:

and chrome seems to be using lots of cpu

and I am using jibri - 8.0-61-g99288dc-1 with 2 participants.