search

jitsi / docker-jitsi-meet

Jitsi Meet on Docker
https://hub.docker.com/u/jitsi/
Apache License 2.0
3.04k stars 1.35k forks source link

Jibri not working/recording since stable-5142 release #871

Closed MominFarhan closed 3 years ago

MominFarhan commented 3 years ago

Tested on release 5142-1, 5142-2, 5142-3

Environmental variables that were changed .env

PUBLIC_URL= DOCKER_HOST_ADDRESS= ENABLE_RECORDING=1

Multiple tests were done after clearing the config folder ~/.jitsi-meet-cfg/, updating the passwords ./gen-passwords.sh, and in combination of the above environmental variables.

All Errors in Jibri logs

**2020-12-10 09:34:51.924 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Visiting url https://meet.jitsi/conventionalrotationsdenouncedefinitely#config.iAmRecorder=true&config.externalConnectUrl=null&config.startWithAudioMuted=true&config.startWithVideoMuted=true&interfaceConfig.APP_NAME="Jibri"&config.analytics.disabled=true&config.p2p.enabled=false&config.prejoinPageEnabled=false&config.requireDisplayName=false 2020-12-10 09:34:52.713 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:53.228 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:53.743 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:54.257 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:54.768 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:55.281 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:55.793 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:56.307 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:56.829 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:57.341 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:57.853 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:58.364 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:58.876 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:59.389 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:34:59.900 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:00.412 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:00.923 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:01.435 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:01.947 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:02.459 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:02.970 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:03.482 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:03.993 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:04.505 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:05.017 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:05.529 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:06.040 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:06.552 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:07.064 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:07.575 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:08.087 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:08.597 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:09.109 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:09.621 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:10.133 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:10.929 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:11.442 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:11.956 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:12.468 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:12.979 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:13.491 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:14.003 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:14.515 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:15.026 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:15.538 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:16.049 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:16.561 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:17.072 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:17.583 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:18.095 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:18.607 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:18.697 FINE: [17] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status 2020-12-10 09:35:19.118 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:19.629 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:20.141 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:20.653 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:21.165 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:21.677 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:22.187 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:22.699 FINE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Not joined yet: Cannot read property 'isJoined' of undefined 2020-12-10 09:35:22.705 SEVERE: [55] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Timed out waiting for call page to load 2020-12-10 09:35:22.710 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Starting up to Error: FailedToJoinCall SESSION Failed to join the call 2020-12-10 09:35:22.711 INFO: [55] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: FailedToJoinCall SESSION Failed to join the call 2020-12-10 09:35:22.713 INFO: [55] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error Error: FailedToJoinCall SESSION Failed to join the call, sending error iq 2020-12-10 09:35:22.714 FINE: [55] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording 2020-12-10 09:35:22.714 INFO: [55] org.jitsi.jibri.JibriManager.stopService() Stopping the current service 2020-12-10 09:35:22.714 INFO: [55] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer 2020-12-10 09:35:22.715 INFO: [55] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process 2020-12-10 09:35:22.715 INFO: [55] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value null 2020-12-10 09:35:22.716 INFO: [55] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium 2020-12-10 09:35:22.729 INFO: [55] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: [] 2020-12-10 09:35:22.792 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving call and quitting browser 2020-12-10 09:35:22.792 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Recurring call status checks cancelled 2020-12-10 09:35:22.811 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 22 log entries for type browser 2020-12-10 09:35:22.896 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 1121 log entries for type driver 2020-12-10 09:35:23.045 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client** 2020-12-10 09:35:23.045 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call 2020-12-10 09:35:23.090 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver 2020-12-10 09:35:23.166 INFO: [55] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit 2020-12-10 09:35:23.166 INFO: [55] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording 2020-12-10 09:35:23.169 SEVERE: [55] org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize() Failed to run finalize script: java.io.IOException: Cannot run program "/config/finalize.sh": error=2, No such file or directory with stack: java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) org.jitsi.jibri.util.ProcessWrapper.start(ProcessWrapper.kt:87) org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize(FileRecordingJibriService.kt:212) org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:197) org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262) org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:210) org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:86) org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37) org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29) org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53) org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29) kotlin.collections.CollectionsKtMutableCollectionsKt.filterInPlace$CollectionsKtMutableCollectionsKt(MutableCollections.kt:293) kotlin.collections.CollectionsKtMutableCollectionsKt.retainAll(MutableCollections.kt:284) org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53) org.jitsi.jibri.service.impl.StatefulJibriService.onServiceStateChange(StatefulJibriService.kt:40) org.jitsi.jibri.service.impl.StatefulJibriService.access$onServiceStateChange(StatefulJibriService.kt:26) org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:35) org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:26) org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26) org.jitsi.jibri.service.JibriServiceStateMachine.access$notify(JibriServiceStateMachine.kt:46) org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:100) org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:46) com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65) com.tinder.StateMachine.transition(StateMachine.kt:23) org.jitsi.jibri.service.JibriServiceStateMachine.transition(JibriServiceStateMachine.kt:112) org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:46) org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:26) org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37) org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29) org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53) org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29) kotlin.collections.CollectionsKtMutableCollectionsKt.filterInPlace$CollectionsKtMutableCollectionsKt(MutableCollections.kt:293) kotlin.collections.CollectionsKtMutableCollectionsKt.retainAll(MutableCollections.kt:284) org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53) org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange(JibriSelenium.kt:167) org.jitsi.jibri.selenium.JibriSelenium.access$onSeleniumStateChange(JibriSelenium.kt:118) org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:152) org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:118) org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26) org.jitsi.jibri.selenium.SeleniumStateMachine.access$notify(SeleniumStateMachine.kt:33) org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:78) org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:33) com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65) com.tinder.StateMachine.transition(StateMachine.kt:23) org.jitsi.jibri.selenium.SeleniumStateMachine.transition(SeleniumStateMachine.kt:83) org.jitsi.jibri.selenium.JibriSelenium$joinCall$1.run(JibriSelenium.kt:251) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

Ignore the finalize.sh error

MominFarhan commented 3 years ago

All Logs

All-logs.txt

saghul commented 3 years ago

Can you share the browser logs?

MominFarhan commented 3 years ago

Browser Logs

image

saghul commented 3 years ago

Sorry I meant from the jibri container. There is a browser logs file.

MominFarhan commented 3 years ago

The above logs shared in comment 1 & 2 are the Jibri logs. Second comment contain all the logs, whereas the first comment only contain the snippet of the logs where I suspect the problem reflects.

MominFarhan commented 3 years ago

Let me know the location/path of chrome logs inside the Jibri container, if that is what you are asking for? These are the browser logs I could provide which are accessible inside the /config mount of Jibri.

browser.0.txt

I hope they are the same :)

sierrabravo79 commented 3 years ago

Hi @MominFarhan

in your browser log I see

[SEVERE] https://meet.jitsi/libs/lib-jitsi-meet.min.js?v=4466 0 WebSocket connection to 'wss://169.48.111.109:8443/xmpp-websocket?room=conventionalrotationsdenouncedefinitely' failed: Error during WebSocket handshake: Unexpected response code: 403

Try to set XMPP_CROSS_DOMAIN=true in .env file and try again

hoffie commented 3 years ago

I was seeing the same issue with a reverse-proxy (nginx) in front of Jitsi, even after properly configuring WebSocket upgrade headers for /xmpp-websocket etc.

[SEVERE] https://meet.jitsi/libs/lib-jitsi-meet.min.js?v=4466 0 WebSocket connection to 'wss://169.48.111.109:8443/xmpp-websocket?room=conventionalrotationsdenouncedefinitely' failed: Error during WebSocket handshake: Unexpected response code: 403

Try to set XMPP_CROSS_DOMAIN=true in .env file and try again

This was the right pointer for me. I did not set this option as I am unsure about the security consequences. Without explicitly setting this option, the Prosody cross_domain_websocket value is derived from PUBLIC_URL. However, when running a reverse proxy in front of Jitsi, the original Origin header may get lost. Therefore, I was able to solve this by setting proxy_set_header Origin "https://meet.my-external-url.example.org/".

MominFarhan commented 3 years ago

Jibri Recording Files Are Corrupt // Stable-Release 5142-3

Jibri Logs

2020-12-14 08:25:49.478 INFO: [54] org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq() Received JibriIq from environment [MucClient id=xmpp.meet.jitsi hostname=xmpp.meet.jitsi] 2020-12-14 08:25:49.478 FINE: [54] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording 2020-12-14 08:25:49.479 INFO: [54] org.jitsi.jibri.JibriManager.stopService() Stopping the current service 2020-12-14 08:25:49.479 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer 2020-12-14 08:25:49.479 INFO: [54] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process 2020-12-14 08:25:49.495 SEVERE: [54] org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor() Error stopping process: java.io.IOException: Cannot run program "kill": error=2, No such file or directory with stack: java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) java.lang.Runtime.exec(Runtime.java:621) java.lang.Runtime.exec(Runtime.java:451) java.lang.Runtime.exec(Runtime.java:348) org.jitsi.jibri.util.ProcessWrapper.stop(ProcessWrapper.kt:101) org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor(ProcessWrapper.kt:112) org.jitsi.jibri.util.JibriSubprocess.stop(JibriSubprocess.kt:77) org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.stop(FfmpegCapturer.kt:125) org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:166) org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262) org.jitsi.jibri.api.xmpp.XmppApi.handleStopJibriIq(XmppApi.kt:263) org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq(XmppApi.kt:169) org.jitsi.jibri.api.xmpp.XmppApi.handleIq(XmppApi.kt:150) org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:569) org.jitsi.xmpp.mucclient.MucClient.access$800(MucClient.java:50) org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:533) org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

2020-12-14 08:25:49.495 SEVERE: [54] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Error trying to gracefully stop ffmpeg, destroying forcibly 2020-12-14 08:25:49.513 INFO: [54] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 137 2020-12-14 08:25:49.513 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium 2020-12-14 08:25:49.573 INFO: [17] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {}, num participants: 1, numMutedParticipants: 0, numJigasis: 0, all clients muted? false 2020-12-14 08:25:49.574 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Trying to retrieve key 'jibri.call-status-checks.all-muted-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.578 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Successfully retrieved key 'jibri.call-status-checks.all-muted-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.578 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Trying to retrieve key 'jibri.call-status-checks.no-media-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.579 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Successfully retrieved key 'jibri.call-status-checks.no-media-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.581 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: [] 2020-12-14 08:25:49.658 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving call and quitting browser 2020-12-14 08:25:49.659 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Recurring call status checks cancelled 2020-12-14 08:25:49.693 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 11 log entries for type browser 2020-12-14 08:25:49.829 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 779 log entries for type driver 2020-12-14 08:25:49.957 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client 2020-12-14 08:25:49.957 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call 2020-12-14 08:25:50.135 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver 2020-12-14 08:25:50.220 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit 2020-12-14 08:25:50.220 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording

Jibri-logs.txt

Jibri Browser Logs

browser.0.txt

prayagsingh commented 3 years ago

@saghul I'm facing this issue too. Final recording is corrupted and un-playable. Using jitsi/jibri:latest image.

saghul commented 3 years ago

My money is on the ffmpeg version. stable-5142 was still stretch, but with -1 we switched to Buster. The Jibri version has not changed. Can you try to run your setup with stable-5142-3, but a stable-5142 jibri?

prayagsingh commented 3 years ago

@saghul I tried stable-5142 and it's crashing. I'm running it on 8vCPU and 8Gb Memory VM and after 10 minutes it started consuming all the memory and finally crashed.

logs

2020-12-17 10:48:29.003 INFO: [18] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {audio={download=33, upload=22}, download=2181, upload=22, video={download=2148, upload=0}}, all clients muted? false
2020-12-17 10:48:29.991 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2020-12-17 10:48:46.326 FINE: [18] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-12-17 10:48:57.175 SEVERE: [18] org.jitsi.jibri.selenium.JibriSelenium.invoke() Error while running call status checks: org.openqa.selenium.WebDriverException: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
  (Session info: chrome=78.0.3904.97)
  (Driver info: chromedriver=78.0.3904.105 (60e2d8774a8151efa6a00b1f358371b1e0e07ee2-refs/branch-heads/3904@{#877}),platform=Linux 4.15.0-128-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '7917f14648ae', ip: '10.0.1.45', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-128-generic', java.version: '1.8.0_265'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: /tmp/.com.google.Chrome.lZDcuJ}, cssSelectorsEnabled: true, databaseEnabled: false, goog:chromeOptions: {debuggerAddress: localhost:36453}, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), rotatable: false, setWindowRect: true, strictFileInteractability: false, takesHeapSnapshot: true, takesScreenshot: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unexpectedAlertBehaviour: ignore, unhandledPromptBehavior: ignore, version: 78.0.3904.97, webStorageEnabled: true}
Session ID: 4b7a5c0a2af99845607bfdab6bd37469 with stack:
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:480)
org.jitsi.jibri.selenium.pageobjects.CallPage.getNumParticipants(CallPage.kt:74)
org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.isCallEmpty(EmptyCallStatusCheck.kt:44)
org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.run(EmptyCallStatusCheck.kt:29)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1$event$1.invoke(JibriSelenium.kt:196)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1$event$1.invoke(JibriSelenium.kt:119)
kotlin.sequences.TransformingSequence$iterator$1.next(Sequences.kt:172)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1.invoke(JibriSelenium.kt:308)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1.invoke(JibriSelenium.kt:119)
org.jitsi.jibri.util.extensions.SchedulerExecutorServiceExtsKt$sam$java_lang_Runnable$0.run(SchedulerExecutorServiceExts.kt)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-12-17 10:48:57.197 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Running to Error: ChromeHung SESSION Chrome hung
2020-12-17 10:48:57.198 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Running to Error: ChromeHung SESSION Chrome hung
2020-12-17 10:48:57.198 INFO: [56] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error Error: ChromeHung SESSION Chrome hung, sending error iq <iq to='jibribrewery@internal-muc.yp.nextmeet.in/focus' id='OV1VM-64' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error' should_retry='true'/></iq>
2020-12-17 10:48:57.200 FINE: [56] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-12-17 10:48:57.200 INFO: [56] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-12-17 10:48:57.200 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-12-17 10:48:57.201 INFO: [56] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-12-17 10:48:57.634 INFO: [56] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 137
2020-12-17 10:48:57.635 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2020-12-17 10:48:57.646 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: []
2020-12-17 10:48:57.768 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving call and quitting browser
2020-12-17 10:48:57.769 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Recurring call status checks cancelled
2020-12-17 10:48:57.772 SEVERE: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Error trying to get chromedriver logs: java.lang.ClassCastException: com.google.common.collect.Maps$TransformedEntriesMap cannot be cast to java.util.List with stack:
org.openqa.selenium.remote.RemoteLogs.getAvailableLogTypes(RemoteLogs.java:108)
org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser(JibriSelenium.kt:279)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:195)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:210)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:86)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:293)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:284)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.service.impl.StatefulJibriService.onServiceStateChange(StatefulJibriService.kt:40)
org.jitsi.jibri.service.impl.StatefulJibriService.access$onServiceStateChange(StatefulJibriService.kt:26)
org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:35)
org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:26)
org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
org.jitsi.jibri.service.JibriServiceStateMachine.access$notify(JibriServiceStateMachine.kt:46)
org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:100)
org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:46)
com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65)
com.tinder.StateMachine.transition(StateMachine.kt:23)
org.jitsi.jibri.service.JibriServiceStateMachine.transition(JibriServiceStateMachine.kt:112)
org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:46)
org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:26)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:293)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:284)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange(JibriSelenium.kt:176)
org.jitsi.jibri.selenium.JibriSelenium.access$onSeleniumStateChange(JibriSelenium.kt:119)
org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:161)
org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:119)
org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
org.jitsi.jibri.selenium.SeleniumStateMachine.access$notify(SeleniumStateMachine.kt:33)
org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:78)
org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:33)
com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65)
com.tinder.StateMachine.transition(StateMachine.kt:23)
org.jitsi.jibri.selenium.SeleniumStateMachine.transition(SeleniumStateMachine.kt:83)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$transitionState$1$1.run(JibriSelenium.kt:185)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-12-17 10:48:57.773 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2020-12-17 10:48:57.785 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2020-12-17 10:48:57.862 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2020-12-17 10:48:57.862 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording

screenshots Screenshot (433) Screenshot (434) Screenshot (435) Screenshot (436)

saghul commented 3 years ago

@bbaldino Any clues?

bbaldino commented 3 years ago

Jibri Recording Files Are Corrupt // Stable-Release 5142-3

Jibri Logs

2020-12-14 08:25:49.478 INFO: [54] org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq() Received JibriIq from environment [MucClient id=xmpp.meet.jitsi hostname=xmpp.meet.jitsi] 2020-12-14 08:25:49.478 FINE: [54] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording 2020-12-14 08:25:49.479 INFO: [54] org.jitsi.jibri.JibriManager.stopService() Stopping the current service 2020-12-14 08:25:49.479 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer 2020-12-14 08:25:49.479 INFO: [54] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process 2020-12-14 08:25:49.495 SEVERE: [54] org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor() Error stopping process: java.io.IOException: Cannot run program "kill": error=2, No such file or directory with stack: java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) java.lang.Runtime.exec(Runtime.java:621) java.lang.Runtime.exec(Runtime.java:451) java.lang.Runtime.exec(Runtime.java:348) org.jitsi.jibri.util.ProcessWrapper.stop(ProcessWrapper.kt:101) org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor(ProcessWrapper.kt:112) org.jitsi.jibri.util.JibriSubprocess.stop(JibriSubprocess.kt:77) org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.stop(FfmpegCapturer.kt:125) org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:166) org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262) org.jitsi.jibri.api.xmpp.XmppApi.handleStopJibriIq(XmppApi.kt:263) org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq(XmppApi.kt:169) org.jitsi.jibri.api.xmpp.XmppApi.handleIq(XmppApi.kt:150) org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:569) org.jitsi.xmpp.mucclient.MucClient.access$800(MucClient.java:50) org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:533) org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

2020-12-14 08:25:49.495 SEVERE: [54] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Error trying to gracefully stop ffmpeg, destroying forcibly 2020-12-14 08:25:49.513 INFO: [54] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 137 2020-12-14 08:25:49.513 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium 2020-12-14 08:25:49.573 INFO: [17] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {}, num participants: 1, numMutedParticipants: 0, numJigasis: 0, all clients muted? false 2020-12-14 08:25:49.574 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Trying to retrieve key 'jibri.call-status-checks.all-muted-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.578 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Successfully retrieved key 'jibri.call-status-checks.all-muted-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.578 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Trying to retrieve key 'jibri.call-status-checks.no-media-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.579 FINE: [17] org.jitsi.jibri.config.debug() ConfigSourceSupplier: Successfully retrieved key 'jibri.call-status-checks.no-media-timeout' from source 'config' as type java.time.Duration 2020-12-14 08:25:49.581 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: [] 2020-12-14 08:25:49.658 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving call and quitting browser 2020-12-14 08:25:49.659 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Recurring call status checks cancelled 2020-12-14 08:25:49.693 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 11 log entries for type browser 2020-12-14 08:25:49.829 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 779 log entries for type driver 2020-12-14 08:25:49.957 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client 2020-12-14 08:25:49.957 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call 2020-12-14 08:25:50.135 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver 2020-12-14 08:25:50.220 INFO: [54] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit 2020-12-14 08:25:50.220 INFO: [54] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording

Jibri-logs.txt

Jibri Browser Logs

browser.0.txt

So this issue is something someone else has reported recently for docker as well. I don't know what's going on here, but some docker change maybe? I don't know what would cause not being able to find kill. @saghul I thought you had a theory here on the linux version changing or something? Ultimately it's somewhat harmless in this case, as the explicit kill commands failed, but we use destroyForcibly, which is a built-in API in the Process class and it looks like that worked to stop ffmpeg.

bbaldino commented 3 years ago

@saghul I tried stable-5142 and it's crashing. I'm running it on 8vCPU and 8Gb Memory VM and after 10 minutes it started consuming all the memory and finally crashed.

logs

2020-12-17 10:48:29.003 INFO: [18] org.jitsi.jibri.selenium.JibriSelenium.run() Jibri client receive bitrates: {audio={download=33, upload=22}, download=2181, upload=22, video={download=2148, upload=0}}, all clients muted? false
2020-12-17 10:48:29.991 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2020-12-17 10:48:46.326 FINE: [18] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-12-17 10:48:57.175 SEVERE: [18] org.jitsi.jibri.selenium.JibriSelenium.invoke() Error while running call status checks: org.openqa.selenium.WebDriverException: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
  (Session info: chrome=78.0.3904.97)
  (Driver info: chromedriver=78.0.3904.105 (60e2d8774a8151efa6a00b1f358371b1e0e07ee2-refs/branch-heads/3904@{#877}),platform=Linux 4.15.0-128-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '7917f14648ae', ip: '10.0.1.45', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-128-generic', java.version: '1.8.0_265'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: /tmp/.com.google.Chrome.lZDcuJ}, cssSelectorsEnabled: true, databaseEnabled: false, goog:chromeOptions: {debuggerAddress: localhost:36453}, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), rotatable: false, setWindowRect: true, strictFileInteractability: false, takesHeapSnapshot: true, takesScreenshot: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unexpectedAlertBehaviour: ignore, unhandledPromptBehavior: ignore, version: 78.0.3904.97, webStorageEnabled: true}
Session ID: 4b7a5c0a2af99845607bfdab6bd37469 with stack:
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:480)
org.jitsi.jibri.selenium.pageobjects.CallPage.getNumParticipants(CallPage.kt:74)
org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.isCallEmpty(EmptyCallStatusCheck.kt:44)
org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.run(EmptyCallStatusCheck.kt:29)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1$event$1.invoke(JibriSelenium.kt:196)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1$event$1.invoke(JibriSelenium.kt:119)
kotlin.sequences.TransformingSequence$iterator$1.next(Sequences.kt:172)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1.invoke(JibriSelenium.kt:308)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$1.invoke(JibriSelenium.kt:119)
org.jitsi.jibri.util.extensions.SchedulerExecutorServiceExtsKt$sam$java_lang_Runnable$0.run(SchedulerExecutorServiceExts.kt)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-12-17 10:48:57.197 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Running to Error: ChromeHung SESSION Chrome hung
2020-12-17 10:48:57.198 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Running to Error: ChromeHung SESSION Chrome hung
2020-12-17 10:48:57.198 INFO: [56] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error Error: ChromeHung SESSION Chrome hung, sending error iq <iq to='jibribrewery@internal-muc.yp.nextmeet.in/focus' id='OV1VM-64' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error' should_retry='true'/></iq>
2020-12-17 10:48:57.200 FINE: [56] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-12-17 10:48:57.200 INFO: [56] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-12-17 10:48:57.200 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-12-17 10:48:57.201 INFO: [56] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-12-17 10:48:57.634 INFO: [56] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 137
2020-12-17 10:48:57.635 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2020-12-17 10:48:57.646 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: []
2020-12-17 10:48:57.768 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving call and quitting browser
2020-12-17 10:48:57.769 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Recurring call status checks cancelled
2020-12-17 10:48:57.772 SEVERE: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Error trying to get chromedriver logs: java.lang.ClassCastException: com.google.common.collect.Maps$TransformedEntriesMap cannot be cast to java.util.List with stack:
org.openqa.selenium.remote.RemoteLogs.getAvailableLogTypes(RemoteLogs.java:108)
org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser(JibriSelenium.kt:279)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:195)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:210)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:86)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:293)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:284)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.service.impl.StatefulJibriService.onServiceStateChange(StatefulJibriService.kt:40)
org.jitsi.jibri.service.impl.StatefulJibriService.access$onServiceStateChange(StatefulJibriService.kt:26)
org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:35)
org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:26)
org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
org.jitsi.jibri.service.JibriServiceStateMachine.access$notify(JibriServiceStateMachine.kt:46)
org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:100)
org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:46)
com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65)
com.tinder.StateMachine.transition(StateMachine.kt:23)
org.jitsi.jibri.service.JibriServiceStateMachine.transition(JibriServiceStateMachine.kt:112)
org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:46)
org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:26)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:293)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:284)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange(JibriSelenium.kt:176)
org.jitsi.jibri.selenium.JibriSelenium.access$onSeleniumStateChange(JibriSelenium.kt:119)
org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:161)
org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:119)
org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
org.jitsi.jibri.selenium.SeleniumStateMachine.access$notify(SeleniumStateMachine.kt:33)
org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:78)
org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:33)
com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65)
com.tinder.StateMachine.transition(StateMachine.kt:23)
org.jitsi.jibri.selenium.SeleniumStateMachine.transition(SeleniumStateMachine.kt:83)
org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$transitionState$1$1.run(JibriSelenium.kt:185)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-12-17 10:48:57.773 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2020-12-17 10:48:57.785 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2020-12-17 10:48:57.862 INFO: [56] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2020-12-17 10:48:57.862 INFO: [56] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording

screenshots Screenshot (433) Screenshot (434) Screenshot (435) Screenshot (436)

This looks to be another type of problem. In the past I think we had some users complain about rampant memory usage until a crash, but I can't remember what the fix was (we never encountered this problem ourselves). There may be some posts on the community about how it was solved.

prayagsingh commented 3 years ago

@bbaldino I tried unstable image too and results are even more scaring. This is happening when I stoped the recording.


2020-12-18 00:11:46.797 INFO: [62] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error Error: CouldntWriteMeetingMetadata SYSTEM Could not write meeting metadata, sending error iq <iq to='jibribrewery@internal-muc.meet.example.com/focus' id='OIiJl-155' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error' should_retry='true'/></iq>
2020-12-18 00:11:46.799 FINE: [62] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: error:recording
2020-12-18 00:11:46.800 INFO: [62] org.jitsi.jibri.status.JibriStatusManager.log() Received component health update: JibriManager has status UNHEALTHY (detail: )
2020-12-18 00:11:46.802 FINE: [62] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-12-18 00:11:46.803 INFO: [62] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-12-18 00:11:46.804 INFO: [62] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-12-18 00:11:46.805 INFO: [62] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-12-18 00:11:46.807 INFO: [62] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 255
2020-12-18 00:11:46.809 INFO: [62] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2020-12-18 00:11:46.816 INFO: [62] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: [prayag@myemail.com]
2020-12-18 00:11:46.823 SEVERE: [62] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Error writing metadata: com.fasterxml.jackson.databind.JsonMappingException: Class java.lang.String not subtype of [map type; class java.util.Map, [simple type, class java.lang.String] -> [simple type, class java.lang.Object]] (through reference chain: org.jitsi.jibri.service.impl.RecordingMetadata["participants"]->java.util.ArrayList[0]) with stack:
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:391)
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:363)
com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:338)
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3189)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:186)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:210)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:86)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:293)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:284)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:190)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:210)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:86)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
``
bbaldino commented 3 years ago

@bbaldino I tried unstable image too and results are even more scaring

Unstable image of what? Jibri latest unstable is the same as stable right now.

2020-12-18 00:11:46.823 SEVERE: [62] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Error writing metadata: com.fasterxml.jackson.databind.JsonMappingException: Class java.lang.String not subtype of [map type; class java.util.Map, [simple type, class java.lang.String] -> [simple type, class java.lang.Object]] (through reference chain: org.jitsi.jibri.service.impl.RecordingMetadata["participants"]->java.util.ArrayList[0]) with stack:

Is this with a custom web UI or with meet? This is complaining about the type of a participant object not being what we expect. Browser logs should have some debug information.

sierrabravo79 commented 3 years ago

Hi @bbaldino ,

as I wrote in another thread, also i am stuck with jibri container error when stopping recording:

2020-12-10 14:53:01.200 INFO: [53] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-12-10 14:53:01.205 SEVERE: [53] org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor() Error stopping process: java.io.IOException: Cannot run program "kill": error=2, No such fi
le or directory with stack: 
java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
java.lang.Runtime.exec(Runtime.java:621)
java.lang.Runtime.exec(Runtime.java:451)
java.lang.Runtime.exec(Runtime.java:348)
org.jitsi.jibri.util.ProcessWrapper.stop(ProcessWrapper.kt:101)
org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor(ProcessWrapper.kt:112)
org.jitsi.jibri.util.JibriSubprocess.stop(JibriSubprocess.kt:77)
org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.stop(FfmpegCapturer.kt:125)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:166)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262)
org.jitsi.jibri.api.xmpp.XmppApi.handleStopJibriIq(XmppApi.kt:263)
org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq(XmppApi.kt:169)
org.jitsi.jibri.api.xmpp.XmppApi.handleIq(XmppApi.kt:150)
org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:569)
org.jitsi.xmpp.mucclient.MucClient.access$800(MucClient.java:50)
org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:533)
org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-12-10 14:53:01.206 SEVERE: [53] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Error trying to gracefully stop ffmpeg, destroying forcibly

I'm on latest Jibri docker image

$ docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
jitsi/jibri         latest              feb9277a4a4d        8 days ago          962MB
$ docker inspect --format='{{index .RepoDigests 0}}' feb9277a4a4d
jitsi/jibri@sha256:e2c2158052578ead0461c23bbf0ffd8a3ea94c8ebc8f8c2711872d1e1779c298

package jibri 
ii  jibri                         8.0-61-g99288dc-1

No customizations on jibri.

Another Jitsi+Jibri setup on two VMs, works perfectly (again no customizations)

ii jibri 8.0-53-ga574be9-1

I hope it will be useful Thank you

prayagsingh commented 3 years ago

@bbaldino I tried unstable image too and results are even more scaring

Unstable image of what? Jibri latest unstable is the same as stable right now.

2020-12-18 00:11:46.823 SEVERE: [62] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Error writing metadata: com.fasterxml.jackson.databind.JsonMappingException: Class java.lang.String not subtype of [map type; class java.util.Map, [simple type, class java.lang.String] -> [simple type, class java.lang.Object]] (through reference chain: org.jitsi.jibri.service.impl.RecordingMetadata["participants"]->java.util.ArrayList[0]) with stack:

Is this with a custom web UI or with meet? This is complaining about the type of a participant object not being what we expect. Browser logs should have some debug information.

It's unstable base-java image then I built my own jibri image.

No it's not meet.jitsi. I'll check the browser logs and get back to you.

kelvinyasu commented 3 years ago

I got the same issue after updated the docker image too.. wanna see if anyone got the problem fixed by someway..

saghul commented 3 years ago

Can you try to rebuild the image from current master? I pushed a fix this morning.

prayagsingh commented 3 years ago

@bbaldino @saghul After running same docker images on different cloud platforms , I have reached to conclusion that something is wrong with one of the cloud platform on which jibri is crashing. I ran same docker images on IBM cloud and other is local cloud provider.

IBM Config 4vCPU, 8Gb memory, Ubuntu 18.04 LTS uname -a Linux nm3 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

Local Cloud Config(jibri crashing) 8vCPU, 8Gb memory, Ubuntu 18.04 LTS uname -a Linux VM-25ff0616-cb3a-43c3-b72d-2845bbd3d2d5 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

Started a meeting with 3 participants. 2 participants are on audio only and third one is both audio and video enabled. meeting went for 25 minutes and on IBM cloud, the memory consumption didn't touch 2Gb while on local cloud platform it touched 8Gb within 10 minutes and also using more than 80% of all 8vCPU's.

kelvinyasu commented 3 years ago

Can you try to rebuild the image from current master? I pushed a fix this morning.

I did try recompile the docker image today and it works... Thanks.. Trying to use the local build image first.

saghul commented 3 years ago

Added an extra fix 6f7b2b4a01b3e8566eb874f163a1a8cb22af4964 I'll release a -4 later today.