Third person joining results in loss of audio for said THIRD PERSON

[tonewarper]

This is related to these issues: https://github.com/jitsi/jitsi-meet/issues/14212 and https://github.com/jitsi/jitsi-meet/issues/14195

I'm making new report because in my case, it's NOT a loss of audio for the first two participants, but only the 3rd participant. The lack of audio is bidirectional (can't hear, can't be heard). As soon as the number of participants goes from 2 to 3, this particular participant's mic appears muted to other participants, despite them not being muted. And the first two participants are unaffected.

Server information:

Like 14212 and unlike 14195, my experience is from the main hosted Meet.Jit.si, not an SDK/installation. Our participant group is using a mix of devices, browsers, and OSs.

Additional information:

It is reliably reproducible (at least in my environment), and also began to be experienced recently.

It seems this isn't caused by a participant's browser, device, or account used, as it happens regardless what laptop or smartphone app they join from. Then again, it does seem biased towards a particular user of ours. Their audio works fine if they are among the first 2 participants, but then when the 3rd joins, it's them who loses the audio. If they're the 3rd joiner, still, it's them, not anyone of the first 2.

Unfortunately I'm not a techie enough to know how to find debugging logs or dig into articles to learn how, as I'm a business end-user not an engineer, but if anyone here would be so kind as to take the time to list out in exact steps how to send any required logs, provide additional info, or would like to hop on a Test call together with me and my participant group (I really recommend this as the fastest way to see it firsthand), I'm happy to contribute troubleshooting that way. I'm also aware this isn't a support channel so I'm fine if this can't be addressed without me providing things like bug reports/screenshots without handholding/help; I just really like Jitsi and wanted to take the time to report the issue.

Here is the room we've been having trouble with most recently, in case its most recent logs can be pulled up centrally: https://meet.jit.si/PreventiveHealthcareforDEIandBusinessExcellence

[scoreym]

We've been running into the same issue recently. I don't really have much to add other than that we are encountering the some behavior when three people connect. Only two are able to hear each other and the third is muted and unable to unmute.

[saghul]

Do you observe a pattern in the type of the 3rd device? Same browser perhaps?

[scoreym]

It doesn't seem to matter which browser type or version. Also haven't tested with any of the users using a non-browser client.

[tonewarper]

Do you observe a pattern in the type of the 3rd device? Same browser perhaps?

Concur with @scoreym ; the issue remained with the 3rd participant when we tested with their different Chrome versions on different Windows laptops, and even via official Jitsi mobile app. We changed the roles so that I would be the 3rd participant not among the 1st two, and it happened to me, using Chrome on Mac. Making us suspect the culprit is somewhere in the server/code itself, or a wider compatibility/security issue with a broad range of devices/browsers/environments.

[kazin-kharizma]

Can confirm that this happens on self hosted installs we operate, ones that our partner agencies operate and several times on the main Jitsi instance. Cloudron has discontinued offering Jitsi in its app repository as a result and I hear Linode and Digital Ocean will be as well.

The crazy part is that even though this and the video quality dropping for the Jitsi Video Bridge being some of the most top listed issues, Jitsi always gives the same unhelpful vague response putting the fault on the user despite their guides not being effective at troubleshooting the issue. This all started because of changes to the software, so help users make the fixes needed or admit you’re working on it.

[saghul]

Cloudron has discontinued offering Jitsi in its app repository as a result and I hear Linode and Digital Ocean will be as well.

Where have you heard this?

[saghul]

The crazy part is that even though this and the video quality dropping for the Jitsi Video Bridge being some of the most top listed issues, Jitsi always gives the same unhelpful vague response putting the fault on the user despite their guides not being effective at troubleshooting the issue. This all started because of changes to the software, so help users make the fixes needed or admit you’re working on it.

We wouldn't introduce bugs just because, why would we do that?

If this isn't fixed is because we cannot reproduce it, nor we do have logs (yet). We use our very own software every single day and so far got no reports of this, so it's hard to track down, that's all.

[kazin-kharizma]

Cloudron has discontinued offering Jitsi in its app repository as a result and I hear Linode and Digital Ocean will be as well.

Where have you heard this?

I’m in the Cloudron repo daily and worked with them to find an alternative solution when the Jitsi package they had was no longer meeting their security and quality controls. In the same topic on the Cloudron forums discussing Jitsi, we wondered why Jitsi remains a viable alternative for other providers, only to have someone chime in and say that Linode’s had issues as well and that new offerings are coming. As for Digital Ocean, I had a Droplet there and spoke to their software team after countless issues and was told, “we aren’t going to wait much longer for the matter to be addressed before looking elsewhere. The Disroot team is also looking at fixes.

I have posted several issues that were ignored here form various accounts, personal, developer, corporate. It’s always the same thing, pass the buck. I’m not saying Jitsi deliberately did anything, I’m merely saying that they are not concerned with user experience to those who self host the app.

[kazin-kharizma]

The crazy part is that even though this and the video quality dropping for the Jitsi Video Bridge being some of the most top listed issues, Jitsi always gives the same unhelpful vague response putting the fault on the user despite their guides not being effective at troubleshooting the issue. This all started because of changes to the software, so help users make the fixes needed or admit you’re working on it.

We wouldn't introduce bugs just because, why would we do that?

If this isn't fixed is because we cannot reproduce it, nor we do have logs (yet). We use our very own software every single day and so far got no reports of this, so it's hard to track down, that's all.

If you want logs, I can assure you, I’ve sent logs. Logs are there and I have tons of the sound issue, the JVB issue and more. The last JVB issue answer was basically, “oh colibre2 changed stuff and we can’t fix it” and so based on what you say here, I ask you this, if it works internally and externally well for you guys, why aren’t you providing any guidance to your self hosted customers to correct their instances? I shouldn’t have to create a work order for a network analyst to discover things based on deciphering the most obscure and cryptic advice.

I’m not angry but I’m annoyed that I produced perfect GitHub issue reports that made me look like an idiot and that my issues remain the top searched issues for this. Alas, I’ve moved to another provider and so it’s all kinda moot now but I do miss the stability of Jitsi. Now it can’t even stay up for more than 10 minutes.

[saghul]

Thanks for elaborating, please see below.

I’m in the Cloudron repo daily and worked with them to find an alternative solution when the Jitsi package they had was no longer meeting their security and quality controls.

Has anyone from their end gotten in touch with us about that? I personally scout a number of repos several times a day, and others do the same, but we could've missed something.

In the same topic on the Cloudron forums discussing Jitsi, we wondered why Jitsi remains a viable alternative for other providers, only to have someone chime in and say that Linode’s had issues as well and that new offerings are coming. As for Digital Ocean, I had a Droplet there and spoke to their software team after countless issues and was told, “we aren’t going to wait much longer for the matter to be addressed before looking elsewhere. The Disroot team is also looking at fixes.

I don't know how they install / configure Jitsi Meet, but taking a quick check I don't see how the DO setup would end up with a valid hostname, since it doesn't ask for one as it's unattended.

Jitsi Meet is composed of a number of moving parts, and having one misfire can cause really weird errors. Not saying the software doesn't have any bugs! We do install the same Debian packages available for the broader community.

I have posted several issues that were ignored here form various accounts, personal, developer, corporate. It’s always the same thing, pass the buck.

Sorry about that. Can you please link them here so we can revisit?

I’m not saying Jitsi deliberately did anything, I’m merely saying that they are not concerned with user experience to those who self host the app.

I understand why you might feel that way, but let me assure you that is not the case.

If you want logs, I can assure you, I’ve sent logs. Logs are there and I have tons of the sound issue, the JVB issue and more. The last JVB issue answer was basically, “oh colibre2 changed stuff and we can’t fix it”

Please do link the issue. I am not familiar with the problem, but since we releasse all components in tandem, a change in colibri2 shouldn't have caused problems. I'd be interested to know more details here.

based on what you say here, I ask you this, if it works internally and externally well for you guys, why aren’t you providing any guidance to your self hosted customers to correct their instances? I shouldn’t have to create a work order for a network analyst to discover things based on deciphering the most obscure and cryptic advice.

I'm not sure we currently have a good understanding of what the problem actually is, so we cannot provide guidance until that is better understood. based on what you say it could be something as simple as a default flipping in our codebase and the config not being updated to match. We try to avoid those, but we've made mistakes.

Now it can’t even stay up for more than 10 minutes.

I don't know what kind of reaction you are looking for here. That is certainly not true.

[kazin-kharizma]

Thanks for elaborating, please see below.

I’m in the Cloudron repo daily and worked with them to find an alternative solution when the Jitsi package they had was no longer meeting their security and quality controls.

Has anyone from their end gotten in touch with us about that? I personally scout a number of repos several times a day, and others do the same, but we could've missed something.

In the same topic on the Cloudron forums discussing Jitsi, we wondered why Jitsi remains a viable alternative for other providers, only to have someone chime in and say that Linode’s had issues as well and that new offerings are coming. As for Digital Ocean, I had a Droplet there and spoke to their software team after countless issues and was told, “we aren’t going to wait much longer for the matter to be addressed before looking elsewhere. The Disroot team is also looking at fixes.

I don't know how they install / configure Jitsi Meet, but taking a quick check I don't see how the DO setup would end up with a valid hostname, since it doesn't ask for one as it's unattended.

Jitsi Meet is composed of a number of moving parts, and having one misfire can cause really weird errors. Not saying the software doesn't have any bugs! We do install the same Debian packages available for the broader community.

I have posted several issues that were ignored here form various accounts, personal, developer, corporate. It’s always the same thing, pass the buck.

Sorry about that. Can you please link them here so we can revisit?

I’m not saying Jitsi deliberately did anything, I’m merely saying that they are not concerned with user experience to those who self host the app.

I understand why you might feel that way, but let me assure you that is not the case.

If you want logs, I can assure you, I’ve sent logs. Logs are there and I have tons of the sound issue, the JVB issue and more. The last JVB issue answer was basically, “oh colibre2 changed stuff and we can’t fix it”

Please do link the issue. I am not familiar with the problem, but since we releasse all components in tandem, a change in colibri2 shouldn't have caused problems. I'd be interested to know more details here.

based on what you say here, I ask you this, if it works internally and externally well for you guys, why aren’t you providing any guidance to your self hosted customers to correct their instances? I shouldn’t have to create a work order for a network analyst to discover things based on deciphering the most obscure and cryptic advice.

I'm not sure we currently have a good understanding of what the problem actually is, so we cannot provide guidance until that is better understood. based on what you say it could be something as simple as a default flipping in our codebase and the config not being updated to match. We try to avoid those, but we've made mistakes.

Now it can’t even stay up for more than 10 minutes.

I don't know what kind of reaction you are looking for here. That is certainly not true.

I am going to get back to you on all the other items that you so painstakingly took the time to answer for me. It is more than I have had in support from you guys in quite some time. Support used to be great. I cannot speak for the Digital Ocean and Linode installs except to say that their current guides worked perfectly before and now they don't work at all.

What I will say is I don't like this:

Now it can’t even stay up for more than 10 minutes.

I don't know what kind of reaction you are looking for here. That is certainly not true.

This is gaslighting a user experience. It is bordering on obfuscation and ever closely to patronising. When I share with you the logs I have (which I will dig up again), you will see that the JVB cannot maintain at all and drops within minutes of it being queued (aka when not P2P). Further to that, the audio issues with iPhone are insane, with the echo. Nevertheless, please do not gaslight your users. I would not have left a beloved software that I told everyone about and had even worked to create our own custom landing page for all those years ago if it worked and I had received the support I was looking for.

As for Cloudron, you may check their forums and search for Jitsi and find the conversation. Jitsi is also no longer in their AppStore.

I thank you for your detailed reply and will send you more details when I have pulled them. Do not gaslight your user's experience. Regardless of what caused it, the experience is real to them and at least in my case, I am not being hyperbolic.

[saghul]

Sorry, my intention was not to gaslight you. I do think you are being hyperbolic and just like do disapprove of my use of language I disapprove of yours.

Let's leave it at that and focus on getting to the bottom of this.

We can't go and look at how all cloud providers do Jitsi Meet deployments, if they want to engage they know where to find us. What we care deeply about is that our users get a good experience when installing Jitsi Meet by themselves, be that via our Debian packages or Docker. If there is a scenario that results in a non-functioning setup or a botched update, we need to fix that.

[kazin-kharizma]

Sorry, my intention was not to gaslight you. I do think you are being hyperbolic and just like do disapprove of my use of language I disapprove of yours.

Let's leave it at that and focus on getting to the bottom of this.

We can't go and look at how all cloud providers do Jitsi Meet deployments, if they want to engage they know where to find us. What we care deeply about is that our users get a good experience when installing Jitsi Meet by themselves, be that via our Debian packages or Docker. If there is a scenario that results in a non-functioning setup or a botched update, we need to fix that.

I wanted you to know that in a desire to put to bed once and for all this issue of the Jitsi Video Bridge having issues (image attached and the issue with the audio cutting out or echoing when people join on iPhone, I have gone ahead and created two brand spanking new Jitsi instances on a Hetzner server and one on a Linode server. Now, what logs, details and precise explorations do you need to address this issue and find a solution because I can report that the issue happens on both server instances without fail. I will provide you everything you request without hesitation and we will see whether Jitsi is indeed committed to solving these matters for its self hosted clients.

Side note, as I even type this out, without doing anything, the Jitsi instace I was in quite, as expected that it would.

Is Jitsi truly prepared to solve this matter, I offer my full cooperation to that end and welcome them to work with me. So, what do you require?

If you require me to self-host via Docker, or Debian on my HPE ProLiant DL360p server at home, or my own ThinkCentre M720q in order to work with me on this matter, I will prepare instances of both for you.

[damencho]

Can you give details, what server type and region did you chose when creating the vms for Linode and Hetzner?

[saghul]

Are those instances accessible? Being able to reproduce ourselves would certainly help.

As for the iPhone echo, it should be fixed in the latest (24.2.2) release. It was a problem with processing duplicate audio 🤦

[kazin-kharizma]

I appreciate the update on the iPhone echo issue, @damencho, and I'm glad to hear it has been addressed in the latest release (24.2.2). That’s great news!

Regarding making the instances accessible, I think that perhaps a public GitHub thread isn't the ideal place for detailed testing and sharing of such data. I'm open to any suggestions you might have for how we could better facilitate this. To give you more context, we initially deployed more testing servers using a single instance of Jitsi installed from the Hetzner app store. You can find the repository for the Hetzner cloud deployment here: Hetzner Cloud Apps - Jitsi. This server was set up in Finland running on Ubuntu 20.04.

Following a recommendation by @saghul, we then moved to an Ubuntu 22.04 Docker-CE instance, also in Finland, where we installed Portainer and Jitsi according to the official Jitsi documentation available at Jitsi DevOps Guide - Docker. The Hetzner Git repository for Docker-CE can be found here: Hetzner Cloud Apps - Docker-CE.

After discontinuing our Linode instance (of which we only have past logs), we performed a clean installation of Jitsi on my bare metal server—an HPE ProLiant DL360p Gen10, following the OpenSUSE guidelines provided here: Jitsi DevOps Guide - OpenSUSE. To confirm that the issue wasn’t specific to the use of OpenSUSE Tumbleweed, we also set up a parallel instance using OpenSUSE Leap 15.5 for comparison.

Unfortunately, across all these setups, we’ve consistently encountered an issue with the Jitsi Video Bridge defaulting to a low FPS setting. This issue has been documented across various threads, including Issue #1599 and Issue #1637, along with numerous other ongoing discussions in the Jitsi Video Bridge Issues and Docker Jitsi Meet Issues.

I have often seen how, in many of these cases, once a minor discrepancy in configuration is found, the thread concludes with an indication that custom configurations cannot be supported. This has been a significant frustration, and as discussed earlier, it has led to certain misunderstandings. However, I am willing to give Jitsi one last try to resolve this matter, potentially regaining a corporate client in the process.

We are ready to test further on any required setup—be it a bare metal, Hetzner install, or a cloud service deployment like Cloudron to thoroughly investigate this issue. We can provide jvb.conf, .env logs, Docker logs, journal entries, or anything else necessary. I’m also working on ensuring we can grant you access to these instances, following necessary security and privacy reviews, as these are primarily sandbox/test zones.

I sincerely hope we can work together to address this issue comprehensively. Your support in this matter.

[damencho]

I don't see any information here about the versions that were used. Were you using the latest stable? You were using install methods not provided by us, even the opensuse one is community-contributed. And are those using the latest stable version?

I may test on those providers in the specified regions, but I will be using debian packages to install jitsi-meet following the quick install guide, this way I know I'm using the latest stable, the versions we run in production.

[damencho]

You can send me links to damencho at jitsi dot org, so you don't make those public.

[saghul]

Likewise, feel free to seend access intructions to saghul @ jitsi dot org.

[kazin-kharizma]

I don't see any information here about the versions that were used. Were you using the latest stable? You were using install methods not provided by us, even the opensuse one is community-contributed. And are those using the latest stable version?

I may test on those providers in the specified regions, but I will be using debian packages to install jitsi-meet following the quick install guide, this way I know I'm using the latest stable, the versions we run in production.

@damencho, thanks for focusing on the installation methods and versions. We consistently use the latest stable versions, whether .deb or Docker images. While I understand the preference for Debian packages for consistency, our previous interactions left us feeling sidelined when our issues were not resolved despite providing extensive logs.

The documentation doesn't clearly state the support status for methods like the OpenSUSE install, leading to confusion among users who follow these guides but feel unsupported. This issue, particularly the contradictions about Docker support mentioned by @saghul, needs clearer communication to enhance user trust and loyalty.

Over the last two years, I've tried various installation methods. Our initial Jitsi setup followed the Quick Install guide for Debian/Ubuntu, but we faced persistent issues. After similar unresolved problems, we eventually switched to another solution, although I'm still testing to isolate these issues for community benefit.

@saghul and @damencho, once we complete our security checks, I'll email you access details for testing. Could you specify what logs or access you need to investigate the JVB errors so that I can share what we see to line-up against what you see? The issue, noted in GitHub issue #1531, affects many users, especially on cloud platforms but even in Docker and Debian/Ubuntu variants. It would help if Jitsi clarified unsupported installation methods in the DevOps Guide to prevent confusion and enhance support.

Looking forward to collaborating on resolving these issues effectively.

I don't see any information here about the versions that were used. Were you using the latest stable? You were using install methods not provided by us, even the opensuse one is community-contributed. And are those using the latest stable version?

@damencho, your query about the versions used points to a broader issue I've previously discussed with @saghul. The unconscious bias in tech interactions often leaves non-developers feeling marginalized. While it's clear from my previous comments that we've moved away from using Jitsi organizationally, my personal dedication to resolving these persistent issues remains strong.

I've been testing nearly a dozen Jitsi instances across various deployment methods and following each option in the DevOps Guide. This issue persists across many setups, not just isolated cases or specific versions. I urge us to look beyond the immediate setups and consider this a more widespread problem that needs our collective attention. Let’s collaborate effectively to understand and resolve this for the betterment of the entire Jitsi community. If it were as simple as what distro was used, or region that was picked, or version, we would have adapted. Hundreds of people with this issue cannot all have the same common issue and if they do, let us find what it is instead of gaslighting them.

[damencho]

The documentation doesn't clearly state the support status for methods like the OpenSUSE install, leading to confusion among users who follow these guides but feel unsupported. This issue, particularly the contradictions about Docker support mentioned by @saghul, needs clearer communication to enhance user trust and loyalty.

Yes, we should address that in the documentation. It can be something I'm missing. We will discuss it.

I've been testing nearly a dozen Jitsi instances across various deployment methods and following each option in the DevOps Guide. This issue persists across many setups, not just isolated cases or specific versions. I urge us to look beyond the immediate setups and consider this a more widespread problem that needs our collective attention. Let’s collaborate effectively to understand and resolve this for the betterment of the entire Jitsi community. If it were as simple as what distro was used, or region that was picked, or version, we would have adapted. Hundreds of people with this issue cannot all have the same common issue and if they do, let us find what it is instead of gaslighting them.

I understand you, but we need to know which version is used. For example, several bugs in FF introduced similar behavior and we were addressing those. Browsers nowadays change every 2 to 4 weeks, and software needs to follow that.

Some of the links with install instructions we didn't even know existed, and we do not have control over them. And that's what I was mentioning, I don't even know the version used there. I have seen instructions targeting 2-3 years old versions and I'm not sure is that the case.

my personal dedication to resolving these persistent issues remains strong.

Thank you for that.

[damencho]

As a start we just need the links where we can observe the issue and see how it looks from the client side.

[damencho]

I just created a server on hetzner, in Finland (I'm in the US) using cx21 type of instance, shared CPU 4GB of ram. Ubuntu 22.04. I followed the quick install guide and installed jitsi-meet. I opened 9 tabs from my browser and I don't see any issue. My ping to there is 140-145ms. What is the load on the server when you start seeing the issue?

[kazin-kharizma]

I understand you, but we need to know which version is used. For example, several bugs in FF introduced similar behavior and we were addressing those. Browsers nowadays change every 2 to 4 weeks, and software needs to follow that.

Some of the links with install instructions we didn't even know existed, and we do not have control over them. And that's what I was mentioning, I don't even know the version used there. I have seen instructions targeting 2-3 years old versions and I'm not sure is that the case.

I have great respect for the work that developers do to stay at the bleeding edge of this rapidly changing world, please don't think otherwise. :) It is not an easy job or passion to have. Working in Atomic distributions myself lately and playing around with NixOS has taught me both the excitement and the headache that comes with rapidly changing factors.

Thank you for your reply and for defending your own labours with such humility.

[kazin-kharizma]

I just created a server on hetzner, in Finland (I'm in the US) using cx21 type of instance, shared CPU 4GB of ram. Ubuntu 22.04. I followed the quick install guide and installed jitsi-meet. I opened 9 tabs from my browser and I don't see any issue. My ping to there is 140-145ms. What is the load on the server when you start seeing the issue?

Might I suggest we take this to email now then? I have meetings for the next few hours but I will send you an email now, that we might connect on these early. :)