search

jitsi / jibri

Jitsi BRoadcasting Infrastructure
Apache License 2.0
609 stars 313 forks source link

Jibri failing to start recording #332

Open SchoolGuy opened 4 years ago

SchoolGuy commented 4 years ago

This Issue tracker is only for reporting bugs and tracking code related issues.

Before posting, please make sure you check community.jitsi.org to see if the same or similar bugs have already been discussed. General questions, installation help, and feature requests can also be posted to community.jitsi.org.

Description

When I click on "Start recording" in the Jitsi-Meet UI I get a "Preparing recording meeting..." forever.

Current behavior

Aug 13 21:30:23 meet2 launch.sh[6121]: Aug 13, 2020 9:30:23 PM org.jitsi.jibri.api.xmpp.XmppApi$handleStartJibriIq$1 run
Aug 13 21:30:23 meet2 launch.sh[6121]: SEVERE: Error starting Jibri service : java.lang.ExceptionInInitializerError with stack:
Aug 13 21:30:23 meet2 launch.sh[6121]: org.jitsi.jibri.service.impl.FileRecordingJibriService.<init>(FileRecordingJibriService.kt:105)
Aug 13 21:30:23 meet2 launch.sh[6121]: org.jitsi.jibri.JibriManager.startFileRecording(JibriManager.kt:134)
Aug 13 21:30:23 meet2 launch.sh[6121]: org.jitsi.jibri.api.xmpp.XmppApi.handleStartService(XmppApi.kt:281)
Aug 13 21:30:23 meet2 launch.sh[6121]: org.jitsi.jibri.api.xmpp.XmppApi.access$handleStartService(XmppApi.kt:67)
Aug 13 21:30:23 meet2 launch.sh[6121]: org.jitsi.jibri.api.xmpp.XmppApi$handleStartJibriIq$1.run(XmppApi.kt:191)
Aug 13 21:30:23 meet2 launch.sh[6121]: java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Aug 13 21:30:23 meet2 launch.sh[6121]: java.util.concurrent.FutureTask.run(FutureTask.java:266)
Aug 13 21:30:23 meet2 launch.sh[6121]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Aug 13 21:30:23 meet2 launch.sh[6121]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Aug 13 21:30:23 meet2 launch.sh[6121]: java.lang.Thread.run(Thread.java:748)
Aug 13 21:30:31 meet2 launch.sh[6121]: Aug 13, 2020 9:30:31 PM org.jitsi.jibri.api.xmpp.XmppApi handleJibriIq
Aug 13 21:30:31 meet2 launch.sh[6121]: INFO: Received JibriIq <iq to='focus@auth.meet2.opensuse.org/QCHwiOIt' from='jibribrewery@internal.auth.meet2.opensuse.org/focus' id='Zm9jdXNAYXV0aC5tZWV0Mi5vcGVuc3VzZS5vcmcvUUNId2lPSXQASVBQTlctOTU4ACZ8g5CBryA1WQTqQFBi+Sw=' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' action='stop'/></iq> from environment [MucClient id=meet2.opensuse.org hostname=meet2.opensuse.org]
Aug 13 21:30:31 meet2 launch.sh[6121]: Aug 13, 2020 9:30:31 PM org.jitsi.jibri.JibriManager stopService
Aug 13 21:30:31 meet2 launch.sh[6121]: INFO: Stopping the current service
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.api.xmpp.XmppApi handleJibriIq
Aug 13 21:30:42 meet2 launch.sh[6121]: INFO: Received JibriIq <iq to='focus@auth.meet2.opensuse.org/QCHwiOIt' from='jibribrewery@internal.auth.meet2.opensuse.org/focus' id='Zm9jdXNAYXV0aC5tZWV0Mi5vcGVuc3VzZS5vcmcvUUNId2lPSXQASVBQTlctOTgyACZ8g5CBryA1WQTqQFBi+Sw=' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' action='start' recording_mode='file' room='desperatepenguinsfinishruthlessly@conference.meet2.opensuse.org' session_id='bzxciempjgioizwl' app_data='{"file_recording_metadata":{"share":true}}'/></iq> from environment [MucClient id=meet2.opensuse.org hostname=meet2.opensuse.org]
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.api.xmpp.XmppApi handleStartJibriIq
Aug 13 21:30:42 meet2 launch.sh[6121]: INFO: Received start request
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.api.xmpp.XmppApi handleStartJibriIq
Aug 13 21:30:42 meet2 launch.sh[6121]: INFO: Sending 'pending' response to start IQ
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.api.xmpp.XmppApi$handleStartJibriIq$1 run
Aug 13 21:30:42 meet2 launch.sh[6121]: INFO: Starting service
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.api.xmpp.XmppApi handleStartService
Aug 13 21:30:42 meet2 launch.sh[6121]: INFO: Parsed call url info: CallUrlInfo(baseUrl=https://meet2.opensuse.org, callName=desperatepenguinsfinishruthlessly, urlParams=[])
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.JibriManager startFileRecording
Aug 13 21:30:42 meet2 launch.sh[6121]: INFO: Starting a file recording with params: FileRecordingRequestParams(callParams=CallParams(callUrlInfo=CallUrlInfo(baseUrl=https://meet2.opensuse.org, callName=desperatepenguinsfinishruthlessly, urlParams=[])), sessionId=bzxciempjgioizwl, callLoginParams=XmppCredentials(domain=recorder.meet2.opensuse.org, username=recorder, password=YOURSECRET3)) finalize script path: /usr/share/jitsi-jibri/finalize_recording.sh and recordings directory: /srv/recordings
Aug 13 21:30:42 meet2 launch.sh[6121]: Aug 13, 2020 9:30:42 PM org.jitsi.jibri.api.xmpp.XmppApi$handleStartJibriIq$1 run
Aug 13 21:30:42 meet2 launch.sh[6121]: SEVERE: Error starting Jibri service : java.lang.NoClassDefFoundError: Could not initialize class org.jitsi.jibri.selenium.JibriSelenium with stack:
Aug 13 21:30:42 meet2 launch.sh[6121]: org.jitsi.jibri.service.impl.FileRecordingJibriService.<init>(FileRecordingJibriService.kt:105)
Aug 13 21:30:42 meet2 launch.sh[6121]: org.jitsi.jibri.JibriManager.startFileRecording(JibriManager.kt:134)
Aug 13 21:30:42 meet2 launch.sh[6121]: org.jitsi.jibri.api.xmpp.XmppApi.handleStartService(XmppApi.kt:281)
Aug 13 21:30:42 meet2 launch.sh[6121]: org.jitsi.jibri.api.xmpp.XmppApi.access$handleStartService(XmppApi.kt:67)
Aug 13 21:30:42 meet2 launch.sh[6121]: org.jitsi.jibri.api.xmpp.XmppApi$handleStartJibriIq$1.run(XmppApi.kt:191)
Aug 13 21:30:42 meet2 launch.sh[6121]: java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Aug 13 21:30:42 meet2 launch.sh[6121]: java.util.concurrent.FutureTask.run(FutureTask.java:266)
Aug 13 21:30:42 meet2 launch.sh[6121]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Aug 13 21:30:42 meet2 launch.sh[6121]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Aug 13 21:30:42 meet2 launch.sh[6121]: java.lang.Thread.run(Thread.java:748)

Expected Behavior

Recording works.

Possible Solution

I saw in the forums a lot of issues around Selenium. The error reporting around this is vague and could be improved. Both stacktraces we got above are not telling us anything.

I also did put in the first line of init in JibriSelenium a Logger.info() but this is not put out anywhere. Thus I guess we fail before reaching it.

My wild guess is that this is caused by the follwing line: private val stateMachine = SeleniumStateMachine()

Also to verify that this is working I put in for Selenium the following Java System Property: -Dwebdriver.chrome.driver="/usr/local/bin/chromedriver" it made no difference, the stacktrace stays the same.

When monitoring with htop I can see that chrome is never starting up as well as I never see the Selenium Server being started.

Steps to reproduce

We followed the install guide in the README. I was able to reproduce this both with Java 1.8 and the Jibri Tag 8.0 (old config) and Java 11 on Git HEAD (new config).

Environment details

OS: openSUSE Leap 15.2

Java:

meet2:~/repos/jibri # java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (IcedTea 3.16.0) (build 1.8.0_252-b09 suse-lp152.1.2-x86_64)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
meet2:~/repos/jibri # javac -version
javac 1.8.0_252
bbaldino commented 4 years ago

I think I've seen strange errors like this before (NoClassDef for a class which is obviously there) and I would try:

  1. Make sure you do a clean before make
  2. Make sure the configured logging directory exists (weird, I know, but if that directory doesn't exist it causes strange errors--I've never investigated it).
SchoolGuy commented 4 years ago

@bbaldino Thanks for the quick response. I will try tomorrow morning and see if it helps!

SchoolGuy commented 4 years ago

Okay I tried now to do a mvn clean and then compiled again. After that I checked if the logging directory exists. Both actions did not change anything as far as I could tell.

What is weird is that we first get the java.lang.ExceptionInInitializerError and THEN when trying to start the recording a second time the NoClassDef error. When restarting via systemctl we also reset the behavior.

SchoolGuy commented 4 years ago

So I now tried to upgrade the Selenium POM to 3.141.59 which didn't made a difference. I will now do a downgrade of our Chrome to V78 which you are using fixed in your Docker Container (we took the latest stable). After that I am out of ideas.

SchoolGuy commented 4 years ago

Also a small note here: https://stackoverflow.com/questions/2210720/how-to-analyse-a-noclassdeffounderror-caused-by-an-ignored-exceptionininitialize

SchoolGuy commented 4 years ago

I more and more get the feeling that this is not Jibri related but rather a Selenium/Chrome problem. We don't have static initilizers as far as I can tell (maybe something autogenerated) in Jibri and thus the root-cause of this for me unknown...

bbaldino commented 4 years ago

Hm. What hash are you using? Tip of master?

EDIT: Oops, just saw you tried tip of master. I would definitely use Java 8 though.

SchoolGuy commented 4 years ago

@bbaldino That is the new current setup. I downgraded from 11 to 1.8 yesterday evening.

EDIT: I upgraded to Git HEAD this morning after some tries with Tag 8.0.

bbaldino commented 4 years ago

Can you list out the exact steps you're using to checkout and build? I can try and repro.

SchoolGuy commented 4 years ago

Leap 15.2

  1. zypper ar https://download.opensuse.org/repositories/home:/SchoolGuy:/jitsi/openSUSE_Leap_15.2/home:SchoolGuy:jitsi.repo
  2. zypper in nginx prosody lua51-zlib jitsi-meet jitsi-videobridge jitsi-jicofo
  3. Adjust the setup according to your Debian instructions.
SchoolGuy commented 4 years ago

In some minutes I will push a more extensive commit to this PR: https://github.com/jitsi/handbook/pull/82

SchoolGuy commented 4 years ago

This is our package: https://build.opensuse.org/package/show/home:SchoolGuy:jitsi/Jitsi-Jibri

SchoolGuy commented 4 years ago

Regarding Source installation: Take the maven_repo.tar,gz from the package and do the following:

  1. zypper ar http://dl.google.com/linux/chrome/rpm/stable/x86_64 Google-Chrome
  2. wget https://dl.google.com/linux/linux_signing_key.pub
  3. rpm --import linux_signing_key.pub
  4. zypper ref -f
  5. zypper in alsa ffmpeg google-chrome-stable chromedriver xf86-video-dummy xf86-input-void xdotool icewm alsa-utils
  6. git clone git@github.com:jitsi/jibri.git
  7. Unpack the maven_repo.tar.gz
  8. mvn package -DskipTests -Dassembly.skipAssembly=false
  9. mkdir /usr/share/jitsi/jibri
  10. cp resources/debian-package/opt/jitsi/jibri/* /usr/share/jitsi/jibri
  11. cp target/jibri-8.0-SNAPSHOT-jar-with-dependencies.jar /usr/share/jitis/jibri/jibri.jar

Then we copy the configs from an internal repo to the destinations along with the service files which are identical to your upstream ones.

SchoolGuy commented 4 years ago

jitsi-jiofo.conf

JICOFO_HOST=localhost

# sets the XMPP domain (default: none)
JICOFO_HOSTNAME=meet2.opensuse.org

# sets the secret used to authenticate as an XMPP component
JICOFO_SECRET=YOURSECRET3

# sets the port to use for the XMPP component connection
JICOFO_PORT=5347

# sets the XMPP domain name to use for XMPP user logins
JICOFO_AUTH_DOMAIN=auth.meet2.opensuse.org

# sets the username to use for XMPP user logins
JICOFO_AUTH_USER=focus

# sets the password to use for XMPP user logins
JICOFO_AUTH_PASSWORD=YOURSECRET3

# extra options to pass to the jicofo daemon
JICOFO_OPTS=""

# adds java system props that are passed to jicofo (default are for home and logging config file)
JAVA_SYS_PROPS="-Xmx3072m\
 -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi\
 -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=jicofo\
 -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi-jicofo\
 -Djava.util.logging.config.file=/etc/jitsi/jicofo/logging.properties"
SchoolGuy commented 4 years ago

jicofo sip-communicator.properties

org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.meet2.opensuse.org
org.jitsi.jicofo.SHORT_ID=55555
# workaround for certificate errors
org.jitsi.jicofo.ALWAYS_TRUST_MODE_ENABLED=true
org.jitsi.jicofo.jibri.BREWERY=JibriBrewery@internal.auth.meet2.opensuse.org
org.jitsi.jicofo.jibri.PENDING_TIMEOUT=90
SchoolGuy commented 4 years ago

jibri.conf

jibri {
  // A unique identifier for this Jibri
  // TODO: eventually this will be required with no default
  id = "8ece92a3-1736-4f03-8e75-d82897af849e"

  // Whether or not Jibri should return to idle state after handling
  // (successfully or unsuccessfully) a request.  A value of 'true'
  // here means that a Jibri will NOT return back to the IDLE state
  // and will need to be restarted in order to be used again.
  single-use-mode = false
  api {
    http {
      external-api-port = 2222
      internal-api-port = 3333
    }
    xmpp {
      // See example_xmpp_envs.conf for an example of what is expected here
        environments = [
        {
            // A user-friendly name for this environment
            name = "Luigi"

            // A list of XMPP server hosts to which we'll connect
            xmpp-server-hosts = [ "localhost" ]

            // The base XMPP domain
            xmpp-domain = "meet2.opensuse.org"

            // The MUC we'll join to announce our presence for
            // recording and streaming services
            control-muc {
                domain = "internal.auth.meet2.opensuse.org"
                room-name = "JibriBrewery"
                nickname = "1b47ce47-98a3-4769-949d-2326f3260390"
            }

            // The login information for the control MUC
            control-login {
                domain = "recorder.meet2.opensuse.org"
                username = "recorder"
                password = "YOURSECRET3"
            }

            // An (optional) MUC configuration where we'll
            // join to announce SIP gateway services
            //sip-control-muc {
            //    domain = "domain"
            //    room-name = "room-name"
            //    nickname = "nickname"
            //}

            // The login information the selenium web client will use
            call-login {
                domain = "recorder.meet2.opensuse.org"
                username = "recorder"
                password = "YOURSECRET3"
            }

            // The value we'll strip from the room JID domain to derive
            // the call URL
            strip-from-room-domain = "conference."

            // How long Jibri sessions will be allowed to last before
            // they are stopped.  A value of 0 allows them to go on
            // indefinitely
            usage-timeout = 1 hour

            // Whether or not we'll automatically trust any cert on
            // this XMPP domain
            trust-all-xmpp-certs = true
        }
    ]
    }
  }
  recording {
    recordings-directory = "/srv/recordings"
    # TODO: make this an optional param and remove the default
    finalize-script = "/usr/share/jitsi-jibri/finalize_recording.sh"
  }
  chrome {
    // The flags which will be passed to chromium when launching
    flags = [
      "--use-fake-ui-for-media-stream",
      "--start-maximized",
      "--kiosk",
      "--enabled",
      "--disable-infobars",
      "--autoplay-policy=no-user-gesture-required"
    ]
  }
  stats {
    enable-stats-d = true
  }
  // A list of subscribers interested in receicing webhook events
  webhook-subscribers = []
}
SchoolGuy commented 4 years ago

Passwords obscured obviously

SchoolGuy commented 4 years ago

Rest is copy paste from you guys.