Closed alfredogt closed 6 years ago
If you wish to make multiple recordings at the same time, simply spin up more instances/VMs running the jibri service. Jicofo will choose from the available pool of jibri recorders and assign them when the user enters their stream ID.
-Aaron
On Mar 27, 2017, at 12:39, Alfredo Guzman notifications@github.com wrote:
in the doc: "It is intended to be run on a separate machine (or a VM), with no other applications using the display or audio devices. Only one recording at a time is supported on a single jibri."
Is it definitely impossible to record more than one at a time?
If I wanted to record or trasminitr to youtube many sessions at the same time what should I do?
Thank you
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jitsi/jibri/issues/33, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8JeB5rnp98mrflp50jRvDWljLr8Wu2ks5rp_RBgaJpZM4MqlvE.
pool of jibri recorders? Where is that configured? Thanks!
Hi, any info on this ? Where can we add servers to the pool ?
You just need to install multiple jibri machines the same way, the all join a muc room which is monitored by jicofo and it chooses the next free one when recording is requested. But Aaron already mentioned all these.
Do we need to change any usernames etc in config files ? When starting like this i see on second server processes but on jitsi side it says it has failed recording.
Actually what happens is that they both start recording and it is fine , then i stop one and start it again , still fine. I want to stop second one but instead of asking me to stop recording it says "no recorders available" , then i stop the first one and press again on second one and it wants to start recording again even when it is still recording.
You only need to have different nicknames on the different jibris, user and password are the same. You can do this by adding: "nick": "jibri123456" to the config.json.
I just tried it and it still does not add "Recording" floating notification to second recording session , but it is recording and it can't stop.
Also "nick" variable is not used , i see that every server uses some random jibri-randomseed nick unique to its session.
This is from jicofo log:
Jicofo 2018-02-22 18:10:07.883 INFO: [124] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status: <jibri-recording-status xmlns='http://jitsi.org/protocol/jibri' status='on'/> in: astonishingpantsbitebravely@conference.external.jibri.zimbra-aaa.de Jicofo 2018-02-22 18:11:25.428 SEVERE: [99] org.jitsi.jicofo.recording.jibri.JibriSession.log() Jibri pending timeout! squaresnakespursuescornfully@conference.external.jibri.zimbra-aaa.de Jicofo 2018-02-22 18:11:25.428 INFO: [99] org.jitsi.jicofo.recording.jibri.JibriSession.log() Recording stopped for: squaresnakespursuescornfully@conference.external.jibri.zimbra-aaa.de Jicofo 2018-02-22 18:11:25.428 INFO: [99] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status: <jibri-recording-status xmlns='http://jitsi.org/protocol/jibri' status='failed'><error type='cancel'><internal-server-error xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></error></jibri-recording-status> in: squaresnakespursuescornfully@conference.external.jibri.zimbra-aaa.de Jicofo 2018-02-22 18:11:25.428 INFO: [99] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status: <jibri-recording-status xmlns='http://jitsi.org/protocol/jibri' status='busy'/> in: squaresnakespursuescornfully@conference.external.jibri.zimbra-aaa.de
The nick is not for the session it is for the jibri users that enter the muc control room, where jicofo is checking for available recorders.
Ok from the jibri log nick is not used form the config file , but even if it is used i still have same issue:
Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,358 INFO Creating a client for hostname: xmpp.jibri.zimbra-aaa.de with options: {'jid': 'jibri@auth.jibri.zimbra-aaa.de', 'roomname': 'TheBrewery', 'selenium_xmpp_prefix': 'recorder.', 'recording_directory': '/home/jibri/jibri_recordings', 'xmpp_domain': 'jibri.zimbra-aaa.de', 'jid_username': 'jibri', 'mucserver_prefix': 'conference.', 'selenium_xmpp_login': 'recorder@recorder.jibri.zimbra-aaa.de', 'usage_timeout': 0, 'boshdomain': 'recorder.jibri.zimbra-aaa.de', 'nick': 'jibri-3219265634743168', 'room': 'TheBrewery@conference.jibri.zimbra-aaa.de', 'brewery_prefix': 'conference.', 'jidserver_prefix': 'auth.', 'roompass': '', 'boshdomain_prefix': 'recorder.', 'password': 'testpass', 'selenium_xmpp_password': 'testpass', 'url': 'https://xmpp.jibri.zimbra-aaa.de/%ROOM%', 'selenium_xmpp_username': 'recorder'} Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,360 INFO jibri_watcher starting up... Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,392 INFO * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,443 WARNING jid property deprecated. Use boundjid.bare Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,490 INFO Negotiating TLS Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,491 INFO Using SSL version: TLS 1.0 Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,497 WARNING Could not find pyasn1 and pyasn1_modules. SSL certificate COULD NOT BE VERIFIED. Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,661 INFO JID set to: jibri@auth.jibri.zimbra-aaa.de/14cc01ee-5c21-447a-be83-d02edfee6c91 Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,662 WARNING Could not find pyasn1 and pyasn1_modules. SSL certificate expiration COULD NOT BE VERIFIED. Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,663 INFO Starting up client <jibrixmppclient.JibriXMPPClient object at 0x7f121ebd1b38>, connecting to room TheBrewery@conference.jibri.zimbra-aaa.de as jibri-3219265634743168 Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,665 INFO sending presence: <presence xml:lang="en" to="TheBrewery@conference.jibri.zimbra-aaa.de"><jibri-status xmlns="http://jitsi.org/protocol/jibri" status="idle" /></presence> Feb 22 17:53:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:53:42,665 INFO Started up client <jibrixmppclient.JibriXMPPClient object at 0x7f121ebd1b38> Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,615 INFO on_jibri_iq: <iq id="amlicmlAYXV0aC5qaWJyaS56aW1icmEtdm5jLmRlLzE0Y2MwMWVlLTVjMjEtNDQ3YS1iZTgzLWQwMmVkZmVlNmM5MQBVNGJVNS03OTEAHAkxS2IAebpngIH97vZEig==" type="set" from="thebrewery@conference.jibri.zimbra-aaa.de/focus" to="jibri@auth.jibri.zimbra-aaa.de/14cc01ee-5c21-447a-be83-d02edfee6c91"><jibri xmlns="http://jitsi.org/protocol/jibri" action="start" room="cruelparrotsimproveangrily@conference.external.jibri.zimbra-aaa.de" recording_mode="file" /></iq> Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,616 INFO sending presence: <presence xml:lang="en" to="TheBrewery@conference.jibri.zimbra-aaa.de"><jibri-status xmlns="http://jitsi.org/protocol/jibri" status="busy" /></presence> Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Client options for host: {'jid': 'jibri@auth.jibri.zimbra-aaa.de', 'roomname': 'TheBrewery', 'selenium_xmpp_prefix': 'recorder.', 'recording_directory': '/home/jibri/jibri_recordings', 'xmpp_domain': 'jibri.zimbra-aaa.de', 'jid_username': 'jibri', 'mucserver_prefix': 'conference.', 'selenium_xmpp_login': 'recorder@recorder.jibri.zimbra-aaa.de', 'usage_timeout': 0, 'boshdomain': 'recorder.jibri.zimbra-aaa.de', 'nick': 'jibri-3219265634743168', 'room': 'TheBrewery@conference.jibri.zimbra-aaa.de', 'brewery_prefix': 'conference.', 'jidserver_prefix': 'auth.', 'roompass': '', 'boshdomain_prefix': 'recorder.', 'password': 'testpass', 'selenium_xmpp_password': 'testpass', 'url': 'https://xmpp.jibri.zimbra-aaa.de/%ROOM%', 'selenium_xmpp_username': 'recorder'} Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Setting xmpp login from client options: recorder@recorder.jibri.zimbra-aaa.de Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Setting xmpp password from client options Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Setting boshdomain from client options recorder.jibri.zimbra-aaa.de Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Setting xmpp_domain from client options jibri.zimbra-aaa.de Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Setting mucserver_prefix from client options conference. Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,617 INFO Setting recording_directory from client options /home/jibri/jibri_recordings Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,765 INFO Start recording callback Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,765 INFO update_jibri_status Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,765 INFO looping through queue for host xmpp.jibri.zimbra-aaa.de Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,765 INFO Starting jibri Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,765 INFO token:::::::::::::: Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,766 INFO Starting selenium attempt 1/3 pjsua_flag:False Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,766 INFO overriding config.hosts.domain with boshdomain: recorder.jibri.zimbra-aaa.de Feb 22 17:55:18 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:18,766 INFO starting jibri selenium, url=https://xmpp.jibri.zimbra-aaa.de/cruelparrotsimproveangrily#config.iAmRecorder=true&config.externalConnectUrl=null&config.hosts.domain=recorder.jibri.zimbra-aaa.de, google_account=None, xmpp_login=recorder@recorder.jibri.zimbra-aaa.de Feb 22 17:55:28 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:28,091 INFO Launching URL: https://xmpp.jibri.zimbra-aaa.de/cruelparrotsimproveangrily#config.iAmRecorder=true&config.externalConnectUrl=null&config.hosts.domain=recorder.jibri.zimbra-aaa.de Feb 22 17:55:28 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:28,092 INFO setting jibri identifiers: display Recorder - email recorder@jitsi.org Feb 22 17:55:29 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:29,743 INFO setting jibri identifiers: xmpp_username_override recorder@recorder.jibri.zimbra-aaa.de Feb 22 17:55:30 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:30,878 INFO starting to wait for XMPPConnected Feb 22 17:55:30 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:30,886 INFO waiting +5 = 0 < 10 for XMPPConnected Feb 22 17:55:35 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:35,911 INFO XMPP connected, done waiting Feb 22 17:55:35 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:35,912 INFO starting to wait for DownloadBitrate Feb 22 17:55:35 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:35,916 INFO waiting +2 = 0 < 10 for DownloadBitrate Feb 22 17:55:37 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:37,924 INFO downloadbitrate > 0, done waiting Feb 22 17:55:37 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:37,924 INFO Selenium started successfully on attempt 1/3 Feb 22 17:55:37 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:37,925 INFO Starting ffmpeg attempt 1/3 Feb 22 17:55:37 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:37,925 INFO starting jibri ffmpeg with youtube-stream-id= Feb 22 17:55:38 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:38,009 INFO No frame= lines found from ffmpeg, not running yet Feb 22 17:55:39 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:39,018 INFO No frame= lines found from ffmpeg, not running yet Feb 22 17:55:40 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:40,026 INFO No frame= lines found from ffmpeg, not running yet Feb 22 17:55:41 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:41,035 INFO No frame= lines found from ffmpeg, not running yet Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,046 INFO ffmpeg process started successfully Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,047 INFO update_jibri_status Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,048 INFO looping through queue for host xmpp.jibri.zimbra-aaa.de Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,048 INFO queueing status started for host xmpp.jibri.zimbra-aaa.de Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,048 INFO queueing job for jibri_watcher Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,049 INFO queued job for jibri_watcher. Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,049 INFO jibri_start_callback completed... Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,049 INFO jibri_watcher received job, now watching ffmpeg and selenium with a timeout of 0. Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,067 INFO starting to wait for DownloadBitrate Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,075 INFO downloadbitrate > 0, done waiting Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,723 INFO got msg from main: started Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,724 INFO sending status update: <iq id="132f0ea8-b116-412d-9def-6e95cba0bee8-7" type="set" to="thebrewery@conference.jibri.zimbra-aaa.de/focus"><jibri xmlns="http://jitsi.org/protocol/jibri" status="on" /></iq> Feb 22 17:55:42 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:42,725 INFO sending status::::===on Feb 22 17:55:47 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:47,097 INFO starting to wait for DownloadBitrate Feb 22 17:55:47 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:47,101 INFO downloadbitrate > 0, done waiting Feb 22 17:55:52 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:52,197 INFO starting to wait for DownloadBitrate Feb 22 17:55:52 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:52,200 INFO downloadbitrate > 0, done waiting Feb 22 17:55:57 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:57,281 INFO starting to wait for DownloadBitrate Feb 22 17:55:57 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:55:57,287 INFO downloadbitrate > 0, done waiting Feb 22 17:56:02 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:56:02,601 INFO starting to wait for DownloadBitrate Feb 22 17:56:02 recorder.jibri.zimbra-aaa.de app.py[982]: 2018-02-22 17:56:02,810 INFO downloadbitrate > 0, done waiting
Hi , can you help with this ?
Hi I made cleaner test where in warnings you can see the whole process , please respond.
`##### 2 sessions
Jicofo 2018-03-02 15:26:48.770 INFO: [69] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
Jicofo 2018-03-02 15:27:25.038 INFO: [53] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
Jicofo 2018-03-02 15:28:30.442 INFO: [98] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
Jicofo 2018-03-02 15:29:22.561 INFO: [99] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
Jicofo 2018-03-02 15:30:52.562 SEVERE: [105] org.jitsi.jicofo.recording.jibri.JibriSession.log() Jibri pending timeout! 58-2@conference.external.jibri.zimbra-aaa.de Jicofo 2018-03-02 15:30:52.562 INFO: [105] org.jitsi.jicofo.recording.jibri.JibriSession.log() Recording stopped for: 58-2@conference.external.jibri.zimbra-aaa.de
..................
Jicofo 2018-03-02 15:30:52.563 INFO: [105] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
Jicofo 2018-03-02 15:32:11.541 INFO: [109] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
Jicofo 2018-03-02 15:33:51.039 INFO: [38] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.sendRemoveSourceIQ().558 Notify remove SSRC 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0 SID: 99ggjtgsfmcmc Sources{ audio: [ssrc=2034517262 ] video: [ssrc=1669193102 ssrc=205460022 ssrc=3853850348 ssrc=51791073 ssrc=180095571 ssrc=1126842249 ] }@595975689 source_Groups{ video:[ SourceGroup(FID)[ ssrc=1669193102 ssrc=205460022 ]SourceGroup(FID)[ ssrc=3853850348 ssrc=180095571 ]SourceGroup(FID)[ ssrc=51791073 ssrc=1126842249 ]SourceGroup(SIM)[ ssrc=1669193102 ssrc=3853850348 ssrc=51791073 ] ] }@890407653
Jicofo 2018-03-02 15:34:11.046 INFO: [102] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Timing out single participant: 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0
Jicofo 2018-03-02 15:34:11.046 INFO: [102] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Terminating: 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0
Jicofo 2018-03-02 15:34:11.047 INFO: [102] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession().597 Terminate session: 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0
Jicofo 2018-03-02 15:34:11.047 INFO: [102] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removing 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0 sources Sources{ audio: [ssrc=259209160 ] }@1925268313
Jicofo 2018-03-02 15:34:11.047 INFO: [102] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Expiring channels for: 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0
Jicofo 2018-03-02 15:34:11.048 INFO: [102] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0
Jicofo 2018-03-02 15:34:12.731 INFO: [38] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberLeft sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@3d9ec882 member=ChatMember[58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0, jid: null]@1058616936]
Jicofo 2018-03-02 15:34:12.731 INFO: [38] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0 is leaving
Jicofo 2018-03-02 15:34:12.731 WARNING: [38] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Participant not found for 58-2@conference.external.jibri.zimbra-aaa.de/recorder-ade5f0 terminated already or never started ?
Jicofo 2018-03-02 15:34:16.684 INFO: [89] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status:
`
So the second recording start starts on jibri server but it is not communicated and recording seems off from jitsi side , only way to stop recording is to close chatroom and let the timeout on jibri happen.
@alfredogt @damencho
Here are the full logs from jibri servers and jicofo from xmpp server:
jicofo_full_log_91-1_2.txt recorder1-91-1.txt recorder1-91-2.txt
Please respond and continue conversation to find resolution for multiple jibri recording sessions. Second recording session is not working correctly.
It may well be that Jicofo doesn't properly handle the case of multiple jibri recorders in a single room. This wasn't ever a use-case we considered, beyond a primary and backup streamer instance. The main use case that's been tested for multiple Jibri instances is to have them recording in separate rooms at the same time. Thank you for the detailed logs, we will examine them and consider what the best next step might be. It's possible that the solution will be block the starting of a second jibri if one is already started.
@aaronkvanmeerten Thank you for response but in our scenario those are 2 separate rooms (2 for testing , later we will add much more jibri servers per need.
The recorder1-91-2.txt should be named recorder2 actually , just a typo in log file name when uploading and 91-1 ,91-2 are rooms.
jibri configuration has different nicks
"nick":"jibri_server2", and "nick":"jibri_server1",
Hi @aaronkvanmeerten @damencho @alfredogt .
This is very urgent for us , are you willing to get in touch and that we arrange some sort of business cooperation to resolve multiple recording sessions without errors?
The issue here is that status update goes to first room "91-1@" instead of the second "91-2@".
Jicofo 2018-03-02 16:29:01.325 INFO: [38] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Received sources from 91-2@conference.external.jibri.zimbra-aaa.de/recorder-2863e0 Sources{ audio: [ssrc=1793974955 ] }@595840060 Jicofo 2018-03-02 16:29:01.325 WARNING: [38] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() No jingle session yet for 91-2@conference.external.jibri.zimbra-aaa.de/059c768e Jicofo 2018-03-02 16:29:01.440 INFO: [38] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Received sources from 91-2@conference.external.jibri.zimbra-aaa.de/059c768e Sources{ audio: [ssrc=3438484023 ] video: [ssrc=367772780 ssrc=218423136 ssrc=1730057572 ssrc=1269277709 ssrc=3425349228 ssrc=657124843 ] }@158414941 Jicofo 2018-03-02 16:29:01.441 INFO: [38] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.sendAddSourceIQ().455 Notify add SSRC 91-2@conference.external.jibri.zimbra-aaa.de/recorder-2863e0 SID: a8qlaecoe4j2u Sources{ audio: [ssrc=3438484023 ] video: [ssrc=367772780 ssrc=218423136 ssrc=1730057572 ssrc=1269277709 ssrc=3425349228 ssrc=657124843 ] }@158414941 source_Groups{ video:[ SourceGroup(FID)[ ssrc=367772780 ssrc=218423136 ]SourceGroup(FID)[ ssrc=1730057572 ssrc=3425349228 ]SourceGroup(FID)[ ssrc=1269277709 ssrc=657124843 ]SourceGroup(SIM)[ ssrc=367772780 ssrc=1730057572 ssrc=1269277709 ] ] }@1861131331 Jicofo 2018-03-02 16:29:01.441 INFO: [38] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.sendAddSourceIQ().455 Notify add SSRC 91-2@conference.external.jibri.zimbra-aaa.de/059c768e SID: 1rr302t27lcet Sources{ audio: [ssrc=1793974955 ] video: [] }@981486357 source_Groups{ }@132088405 Jicofo 2018-03-02 16:29:08.584 INFO: [116] org.jitsi.jicofo.recording.jibri.JibriSession.log() Updating status from JIBRI: <iq to='focus@auth.jibri.zimbra-aaa.de/focus21251986190' from='thebrewery@conference.jibri.zimbra-aaa.de/jibri-07721004909600315' id='Zm9jdXNAYXV0aC5qaWJyaS56aW1icmEtdm5jLmRlL2ZvY3VzMjEyNTE5ODYxOTAAMGY3MmIyMDktNDZkNy00Nzk4LTlkOGMtZjkxNDg4ZDlkNGEzLUQAOnLZC4jkyD2stnCd49FtIQ==' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='on'/></iq> for 91-1@conference.external.jibri.zimbra-aaa.de Jicofo 2018-03-02 16:29:08.585 INFO: [116] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status: <jibri-recording-status xmlns='http://jitsi.org/protocol/jibri' status='on'/> in: 91-1@conference.external.jibri.zimbra-aaa.de Jicofo 2018-03-02 16:30:25.903 SEVERE: [105] org.jitsi.jicofo.recording.jibri.JibriSession.log() Jibri pending timeout! 91-2@conference.external.jibri.zimbra-aaa.de Jicofo 2018-03-02 16:30:25.904 INFO: [105] org.jitsi.jicofo.recording.jibri.JibriSession.log() Recording stopped for: 91-2@conference.external.jibri.zimbra-aaa.de Jicofo 2018-03-02 16:30:25.904 INFO: [105] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status: <jibri-recording-status xmlns='http://jitsi.org/protocol/jibri' status='failed'><error type='cancel'><internal-server-error xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></error></jibri-recording-status> in: 91-2@conference.external.jibri.zimbra-aaa.de Jicofo 2018-03-02 16:30:25.904 INFO: [105] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Publish new JIBRI status: <jibri-recording-status xmlns='http://jitsi.org/protocol/jibri' status='busy'/> in: 91-2@conference.external.jibri.zimbra-aaa.de
It may well be that Jicofo doesn't properly handle the case of multiple jibri recorders in a single room.
This. One recording session per jibri JID is supported. If you try to have more than one, the requests get routed to the wrong place. :( https://github.com/jitsi/jicofo/blob/master/src/main/java/org/jitsi/jicofo/recording/jibri/JibriRecorder.java#L112
Hi @bgrozev but the thing is that we use separate ones , I changed the config and 2 recorders now have separate usernames: 'jid': 'jibri2@auth.jibri.zimbra-aaa.de' other this: 'jid': 'jibri@auth.jibri.zimbra-aaa.de'
even the 'nick' is added to config but it is not used.
Oh, I see! Looking into it further, we use the occupant JIDs of the jibri's and we compare them as bare JIDs. And the bare JIDs of the two jibri's are the same, because they are in the same MUC.
To confirm that this is the problem can you please try this patch? https://github.com/bgrozev/jicofo/tree/fix-jibri-iq-routing
If that works I'll make a ticket to implement a more proper fix.
@bgrozev
so far this works ! I will make detailed tests tomorrow but for now it all seems ok. Thanks for your response.
Thanks for the report! I've opened #264 to get this merged into master.
This should be fixed in master. Please re-open if you find that it isn't.
Can someone clarify if a single machine can or cannot run multiple jibri processes? Right now, it seems that jibri process has an internal jetty listening to ports 2222 and 3333. It prevents another jibri process started. If this restriction is removed somehow, any other components (e.g. device modules, chrome...) may cause the limitation?
@vincentwlau that's not a configuration we've tested as jibri's services can be pretty sensitive to contention, so we've found it easier to just spin up multiple.
@bbaldino Thanks for the response. But single jibri per machine is not a viable solution since our customer requires 1000+ concurrent recording sessions. Can you elaborate where the contention is? If jibri is not a right solution, do you have any suggestions?
My team is willing to look into this contention issue. If we can solve it, we would like to contribute the solution to the community. Would anyone like to share any insights where the contention is, or point us to the direction where we can explore ourselves?
couple things i can think of off the bat: 1) configurable ports for the http apis. this was on my todo list already and would be a good standalone change. my thought was to do this via command-line args 2) cpu contention. we haven't done any measurements as to how sensitive ffmpeg is to load. i.e. how many simultaneous live streams/recordings can ffmpeg do without causing problem. same thing goes for selenium. 3) device conflicts. making sure there isn't anything weird with device contention between the selenium processes. this shouldn't be an issue, since jibri should join as muted, but i've seen this not take effect when testing locally. 4) each jibri will need its own logging.properties file (so they don't overwrite one another) and config.json (so they used different nicknames in the mucs)
that's all that came to me immediately, i'll see if i come up with anything else.
@bbaldino Thank you very much for the information; I will try out #2 and #3.
For #1, using command-line args is a good idea. Any thought on how launch.sh and shutdown.sh (and graceful_shutown.sh) handle multiple jibri processes on a single machine? I have an idea. The config.json can have two optional new elements "internal_http_api_port" (default is 3333) and "external_http_api_port" (default is 2222.) The launch.sh and shutdown.sh can look for config[0-9]*.json files to start and stop all additional instances. To be backward compatible, launch.sh will always look for config.json and shutdown.sh will always do curl on port 3333.
For #4, we will change the logging handler patterns from "xxx.%g.txt" to "xxx.%g.%u.txt" where xxx is "log", ffmpeg", "pjsua" and "browser".
for your questions on #1 i think @aaronkvanmeerten would have some good thoughts, but putting the ports in config.json instead of command line args could work well (i don't think we need to pass them as args if they're in config.json)
one concern about your idea for #4 is: how will we know which jibri is logging to which file? i guess maybe we don't care. i'd have to think about whether we do or not. also, i'm not sure the exact semantics of %u
--is that generated once per process?
From FileHandler javadoc: %u is a unique number to resolve conflicts between simultaneous Java processes. It starts with 0 and incremented if there is a conflict. But you are right that it is not smart enough to reuse the number when an instance is restarted. Probably need to invent a new format "%i" for an instance identifier; it needs more thinking.
The main issue with using multiple jibris on a single instance is actually the distinct number of X desktops and ALSA loopback devices configured on the instance. As long as there are distinct desktops and distinct devices, and jibri is re-written to be configured to use a specific device per instance, I believe this is possible. There was an excellent fork of the python version that provided for this functionality, although we didn't end up merging it as we ran into trouble testing it and it then it got stale/out of date.
When you refer a distinct desktop, does it mean a unique display number? By any chance, do you know how to configure more than one ALSA loopback device? That is a good start for us to explore.
@bbaldino For the concurrent logging issue, will it be better to have config.json containing an array of the elements? Then in the arg line, there will be an option of instance identifier (0, 1...) One thing that I dislike this scheme is breaking the compatibility of config.json. It is just a thought.
Yes, I do mean a unique display number. By default the jibri service is configured to start up a supporting X11 server on :0
As far as multiple ALSA loopback devices, I believe you can do it with the enable= option, as referenced here: https://stackoverflow.com/questions/18321094/adding-multiple-loopback-alsa-devices-in-ubuntu
in /etc/modprobe.d/sound.conf add the following to enable for example 5 loopback devices options snd-aloop enable=1,1,1,1,1 index=10,11,12,13,14
Then you'd have to set up the .asoundrc in the jibri homedir to reference all the devices like we've done for the first one.
@bbaldino For the concurrent logging issue, will it be better to have config.json containing an array of the elements? Then in the arg line, there will be an option of instance identifier (0, 1...) One thing that I dislike this scheme is breaking the compatibility of config.json. It is just a thought.
Yeah, maybe we can think about this more. I'd say for now don't worry about the logging issue (the other problems will likely be more difficult anyway) and we can figure that one out down the line.
Thank you both for the valuable input. I really appreciate your time helping me.
Does anyone have a recommended (minimum) hardware requirement (# of cores, CPU speed, RAM, swap space) for Jibri server? We had a single core, 2.3GHz, 4GB RAM, 0B swap-space 64-bit Ubuntu cloud server and it hung after 20 seconds of recording with two participants due to out of swap space.
We are currently using the c5.xlarge AWS instance for recordings. This is a 4 vCPU, 8GB of RAM instance. We do not configure any swap either. Considering that this instance must run Chrome and render the high quality video stream before having ffmpeg capture it, the main restriction is how much memory Chrome uses for swap, and how much memory ffmpeg needs for capture, and finally how much memory the jibri java uses during operations.
@aaronkvanmeerten Thank you very much for information. Just FYI. We were able to run Jibri with 2 vCPU (2.3GHz), 8GB RAM, 0 swap and its memory usage never dipped below 6GB. So, we think it should be able run with 4GB RAM as well.
Dears,
Can you share more details for configuring Jibri in one instance and be able to record many sessions simultaneously, i tried to execute the Jibri second instance on the same machine but it throws exception as the address is already in use for 3333 port,also i tried to change the ALSO loopback configuration and no sense.Could you share some detailed information for this matter ?
Thanks
@arts111188 Although we modified the source in our private experiment to allow ports 2222 and 3333 configurable via the config.json, we stopped pursuing further to support different the x11 display and audio loopback for ffmpeg because the CPU consumption for a recording session is very high and it won't be scaled well for our requirement. We are trying to take a different approach to "record" multiple sessions without x11 and audio loopback from a single server. Unlike Jibri's real-time recording, we are merely archiving the media streams for offline mpeg encoding. It is sort of a hybrid of Jibri and Jirecon. We are still experimenting with this idea, so it is too soon to share.
Hello everyone, Finally, the solution that was applied was to create an instance in AWS for each jibri session. My need was for maximum 12 or 15 active jibri sessions at a time. I had problems with Jicofo, so I used a separate socket connection to start / stop jibri. I have not had the opportunity to try the new version of jibri that JAVA uses. I hope to contribute soon. Greetings.
@alfredogt , any idea for creating "N" instances (100 to 200) dynamically for multiple recording at the same time based on session created? Is it efficient? Also, what's your opinion in cost wise?
Hello, @andaralex In our project we do not need so many instances created at the same time.
We made some scripts to create and turn off new instances for each recording, we made some modifications to Jibri to make each instance connect to the specific session.
Is it efficient ?: Yes, a complete instance for each session, we had 99% efficiency, except for creation or network errors. we use the OVH service that have low prices, we had to write to support to enable more than 10 instances at the time, we were enabled 50 instances.
Regarding costs, I consider it viable, because it pays by the hour. The cost of each hour of an instance in OVH is really low
Hello, @alfredogt
Can you explain how you have made modifications to associate a specific jibri's instance to a specific room please ?
Greetings.
Hello,
I answer to myself : simply use HTTP API instead of XMPP API !
Greetings.
Hi, I need to run 3 jibri instances in a single VM. Could anyone please help me or guide me how to configure that, or what modification is needed? I have installed a single instance in a VM and it is working fine. Thanks for your help.
This isn't really possible with Jibri in its current state. We recommend using separate VMs for this type of functionality.
This isn't really possible with Jibri in its current state. We recommend using separate VMs for this type of functionality.
@aaronkvanmeerten Thanks for a quick response. But a single jibri instance can be used to record a single meeting at a time. So if I need to record 50 meetings at a same time I need to install 50 separate jibri instances in 50 separate VMs? It is not really feasible. Isn't there any other way so that we can install at least 5 instances in a single VM? Please help on this regards.
Yes that's accurate. Jibri uses a kernel module (Alsa loopback) for doing the audio capture, and this is set up per kernel. So it MAY be possible to set up a single VM with as many alsa loopback devices as you want per instance, and then run the docker container of jibri with the proper devices mapped, we don't suggest it as the CPU and memory constraints of jibri means if you have many recordings on a single VM at one time you'll end up with bad recordings (stuttering video, dropped audio, etc). So in general recommend running jibri in an autoscaling setup if you wish to do multiple recordings at the same time.
in the doc: "It is intended to be run on a separate machine (or a VM), with no other applications using the display or audio devices. Only one recording at a time is supported on a single jibri."
Is it definitely impossible to record more than one at a time?
If I wanted to record or trasminitr to youtube many sessions at the same time what should I do?
Thank you