All conferences on a multi-JVB setup crash when a single JVB goes down

[kordian-kowalski]

(this is a re-post of https://community.jitsi.org/t/shutting-down-a-jvb-crashes-a-conference-on-another-jvb/95948/5)

I'm not sure if this is specifically caused by a bug in jicofo, but here's the problem:

On a vanilla Jitsi installation (installed on clean Debian on AWS via APT), after adding additional JVBs on separate machines, an issue pops up that makes all conferences on the whole deployment crash when a single JVB goes down.

My initial test that I described on the forums assumes that there is a conference being hosted on the JVB that is to be shut down, but later random tests show that it may not be the case - killing an empty JVB results in similar effects.

Sometimes a conference will restart correctly after the reconnection period in jitsi-meet, but often with no video in video tiles. In this case it seems that all participants need to leave the room (so that it's cleaned up by jicofo I guess)

In any case, I always see this in jicofo logs: jvbbrewery@internal.auth.OUR-DOMAIN/jvb-02 to allocate channels for: OctoParticipant[relays=[]]@607377124 which makes no sense as we don't have octo enabled. (In fact I disabled it explicitly as suggested by a commenter, to no effect)

Our test setup:

• AWS instance with full Jitsi stack except JVB
• 3 AWS instances with a JVB only, using the same XMPP credentials but with unique nicknames (jvb-01, jvb-02, jvb-03)

Reproducing the issue:

• Opened 4 browser windows
• Navigated to vanilla.OUR-DOMAIN/conf-01 on 2 of them (conf-01 got scheduled on jvb-03)
• Navigated to vanilla.OUR-DOMAIN/conf-02 on the other 2 (conf-02 got scheduled on jvb-02)
• Stopped machine hosting jvb-03 (== conf-01)

What I expected:

• conf-01 to be rescheduled on jvb-01
• conf-02 to continue unaffected

What happened:

• conf-01 got rescheduled on jvb-01 and continued normally
• conf-02 was interrupted, rescheduled on jvb-01, after reconnecting participants there was no audio/video (empty tiles)
• in the logs, there are some entries about a channel being allocated for an OctoParticipant on jvb-02

Re-posting full logs:

• jicofo.log
• jvb-01
• jvb-02

[kordian-kowalski]

I just tested the same on older versions of Jitsi stack:

jitsi-meet-web=1.0.4466-1
jitsi-meet-web-config=1.0.4466-1
jitsi-videobridge2=2.1-376-g9f12bfe2-1
jicofo=1.0-644-1
jitsi-meet-prosody=1.0.4466-1
jitsi-meet-turnserver=1.0.4466-1

And the issue is not present there. Also, the failover to a working JVB is nearly instant (as opposed to a couple of seconds for latest Jitsi packages)

[kordian-kowalski]

After some testing it seems that simply downgrading jicofo to 1.0-644-1 (leaving other components on latest releases) solves the conference crashing issue.

Looks like some regression?

[flyinghuman]

same here. i updated some servers in Multi-Setup and the other conferences crashed. Please fix this. Thanks!

[damencho]

This had been fixed in the latest stable.

[gpolitis]

This is fixed here https://github.com/jitsi/jicofo/pull/713. @flyinghuman if you update your deployments as @damencho suggested and still experience problems please open a new issue with all the details and we'll look into it. Thanks!