Audio crackly in BigBlueButton 2.2

[sydneysider]

Unfortunately, we're developing new issues with BBB. Audio crackling for some users. Unfortunately, we're to the point where my users have zero faith in BBB which is a total bummer. I really love the product, but seemingly too unstable.

[ffdixon]

You didn't offer any details on the users, but we're interested to try and help. A few questions

1. Do you have users who are consistently having troubles, or occasionally having troubles?
2. Are all the users in a session having troubles, or only some

The user's audio experience is dependent on (a) user's browser, (b) operating system, (c) computer, (d) connection to the internet (wired or wireless), (e) internet connection between their connection and your BigBlueButton server, (f) your BigBlueButton server's incoming bandwidth, (g) your server, and (h) the version of BigBlueButton running on the server.

The trick is to narrow down which step along the way is causing users to have poor bandwidth. Some of these you can control -- (f), (g), and (h) -- but the rest are dependent on the users.

Usually (a) to (e) are good for the average user, but 'good' heavily depends on where they live. Can you share more details about your users?

You haven't provided any details on your server, so the following would be helpful

1. What is the version of BigBlueButton you are using?
2. Does your server meet the minimum configuration required by BigBlueButton (Yes/No)?
3. Did you get errors during installation?
4. Did you make any custom changes to BigBlueButton outside of using bbb-conf?

If you have users that are consistently having troubles, then have them try out

https://test.bigbluebutton.org/

There experience with the above server -- better/same/worse experience -- will help isolate differences in (f) and (g).

There is a bit of investigation needed here. If you are willing to provide more information, we're interested in figuring out why users are not having good audio.

[sydneysider]

Apologies for the brevity. That's the thing, it's too intermittent (100 users), some in different countries, some at home, some in the office. Some on wifi, some are wired. The easiest common denominator was their version of Chrome, with an updated version some issues seemed to fix quickly.

I can tell you the AWS instance have set up C5Large (10G throughput) was set up fine with bbb-install.sh - 4 cores, 8gb ram. I haven't seen any actual hardware performance come close to hitting those numbers.

We're running v2.2, and look, it runs fine when you test. If I run your test at test.bigbluebutton.org, that runs fine too. But unfortunately, we have users that require instant working connections 100% of the time.

1) 2.2 2) Meets minimum requirements, see above. 3) No errors, ran from bbb-install.sh script. 4) No custom changes.

Unfortunately, the problem is always going to come down the lack of experience with the user - they just want things to work, and when they don't, they're ready to trash it - which is annoying. Because I get no real feedback outside of "crackly audio" "green fuzziness video". But unfortunately, between all these tests among multiple users, the general agreement is that it's a bad experience in terms of its inconsistency.

[ffdixon]

We empathize.

From a user's perspective, if the software loaded, it must therefore work. If it doesn't, the problem must be with the software. It's a common conclusion.

Consistently sending and receiving media packets requires a stable network. A browser can load a web page in 250 ms or 500 ms and the users can't tell the difference. But given network latency of 250 ms or 500 ms the users can hear the difference. Combine high latency, dropped packets, and low bandwidth together, these network issues will cause poor audio for a user.

Right now with BigBlueButton, any network issues are largely invisible to the users.

We're working to make potential network issues visible to the user, see

https://github.com/bigbluebutton/bigbluebutton/issues/5286

Chrome especially has evolved its WebRTC statistics and we're building upon this to give the user feedback when the client detects possible network issues (this will become default in Chrome 74), see about issue for more details.

This way, if the user is notified of a potential issue and starts to hear poor audio, they can make the connection on what is causing the problem. It's their network.

If the user experiencing poor audio asks others in the session about their audio, and if others say it's OK, then that user can conclude it's not the server, and it's not the software, it's something on their end (most likely their network).

Giving feedback on potential network problems won't resolve audio issues, but it does give users valuable information on how they might improve their audio during the session (i.e. try a different network access point if they are on wireless).

We'll keep this issue open until the work for #5286 lands in the client. As soon as it does and is available in 2.2-beta, we'll let you know.

In a strange way, we need people who are having audio troubles to help us test our work. Hopefully you'll be able to try it out when is ready and give us your feedback.

[Deepcuts]

Having the same issue with latest beta. Running on ESXI 6.7 Ubuntu 16.04 with kernel 4.15.0-66 BigBlueButton Server 2.2.0-rc-1 (1569) CPU is an Intel Xeon D-2146NT 8 cores/16 T Allocated 32 GB of RAM Storage is on a Samsung 960 PRO SSD

At first I thought the presenters just have bad mics and/or internet. Tested from a computer that has 1 Gbps/2 ms tested connection to the server hosting BBB. Same result: audio is crackling loudly from time to time.

Sample recording via desktop microphone from two speakers playing an audiobook https://soundcloud.com/razvan-constantin-9/bbb-crackling Cracklings at 00:11 00:29 00:36 00:53 Recorded voice from an actual presenter shows even more crackling.

Tried disabling comfort audio noise. No change

[ffdixon]

Can you do a recording test on

https://demo.bigbluebutton.org/

and let us know if it sounds any different. Also, the upload sounds to soundcloud were not available when we clicked on them to listen.

[Deepcuts]

Sorry, the recording was set to private. Fixed now. Will try to record on demo.bigbluebutton.org and get back with details.

[ffdixon]

Also, see the following discussion on bigbluebutton-dev

https://github.com/blindsidenetworks/wordpress-plugin_bigbluebutton

regarding the possible adjustments to the audio.

[capilkey]

I think the correct link is this one, https://groups.google.com/d/msg/bigbluebutton-setup/3Y7VBllwpX0/e05YaXQWBgAJ. That's where discussion was going on about configuration changes to FreeSWITCH.

[Deepcuts]

Thank you for pointing me to that discussion. I have tried every possible combination based on https://groups.google.com/d/msg/bigbluebutton-setup/3Y7VBllwpX0/e05YaXQWBgAJ without any kind of success. Searching some more, it seems the crackling audio is a long standing "feature" of freeswitch. I was able to track users with similar problems back to at least 2013. I should point out that the crackling is also present during live session, not only on the recording and also, that VM or dedicated server makes no difference.

Still looking for a solution.

[ffdixon]

Thanks for the feedback. We're always looking to improve the audio as well. For most users, they occasional crackling isn't a bit deal as the audio sounds much richer than the phone. Still, we'd like to reduce (or eliminate) as much unwanted sounds as possible.

[Semelovich]

Check this out https://groups.google.com/d/msg/bigbluebutton-dev/joJyywgYTTk/roFBwEq4EQAJ Kudos to Roberto!

We used almost the same config with small changes (we don't need increased bitrate and we don't use the video-mcu-stereo profile he is using. + we decided to keep vbr and set packet loss to 5%)

https://pastebin.com/8Fs9qG0w

The sound is MUCH better but there is still work to be done to make it better.

So, guys, you should really start looking into these settings ;) At least this will give you an idea what to look for and there. Perhaps you could get in touch with your Freeswitch contacts , I dunno.

Regards.

[ritzalam]

CONFERENCE>CONF.XML

<profile name="cdquality">
      <param name="domain" value="$${domain}"/>
      <param name="rate" value="48000"/>
      <param name="interval" value="20"/>
      <param name="energy-level" value="0"/>
      <!-- <param name="sound-prefix" value="$${sounds_dir}/en/us/callie"/> -->
      <!-- param name="muted-sound" value="conference/conf-muted.wav"/>
      <param name="unmuted-sound" value="conference/conf-unmuted.wav"/>
      <param name="alone-sound" value="conference/conf-alone.wav"/>

      <param name="moh-sound" value="$${hold_music}"/>
      <param name="enter-sound" value="tone_stream://%(200,0,500,600,700)"/>
      <param name="exit-sound" value="tone_stream://%(500,0,300,200,100,50,25)"/>
      <param name="kicked-sound" value="conference/conf-kicked.wav"/>

      <param name="locked-sound" value="conference/conf-locked.wav"/>
      <param name="is-locked-sound" value="conference/conf-is-locked.wav"/>
      <param name="is-unlocked-sound" value="conference/conf-is-unlocked.wav"/>
      <param name="pin-sound" value="conference/conf-pin.wav"/>
      <param name="bad-pin-sound" value="conference/conf-bad-pin.wav"/> -->
      <param name="caller-id-name" value="$${outbound_caller_name}"/>
      <param name="caller-id-number" value="$${outbound_caller_id}"/>
      <!-- param name="comfort-noise" value="true"/ -->
      <param name="comfort-noise" value="0"/>

OPUS.CONF.XML

<configuration name="opus.conf">
      <settings> <!--param name="asymmetric-sample-rates" value="0"/-->
        <param name="use-vbr" value="1"/>
        <param name="use-dtx" value="0"/>
    <param name="complexity" value="10"/>
    <param name="packet-loss-percent" value="5"/>
        <param name="keep-fec-enabled" value="0"/>
        <param name="use-jb-lookahead" value="0"/>
        <param name="advertise-useinbandfec" value="0"/>
    <param name="adjust-bitrate" value="0"/>
      </settings>
</configuration>

BBB_CONFERENCE.XML

<include>
    <extension name="bbb_conferences_ws">
      <condition field="${bbb_authorized}" expression="true" break="on-false"/>
      <condition field="${sip_via_protocol}" expression="^wss?$"/>
      <condition field="destination_number" expression="^(\d{5,6})$">
<action application="set" data="rtp_jitter_buffer_plc=true"/>
<!--action application="set" data="rtp_jitter_buffer_during_bridge=true"/>
<action application="set" data="jitterbuffer_msec=15:100"/>-->
<action application="jitterbuffer" data="60"/>
        <action application="answer"/>
        <action application="conference" data="$1@cdquality"/>
      </condition>
    </extension>
    <extension name="bbb_conferences">
      <condition field="${bbb_authorized}" expression="true" break="on-false"/>
      <condition field="destination_number" expression="^(\d{5,6})$">
<action application="set" data="rtp_jitter_buffer_plc=true"/>
<!--action application="set" data="rtp_jitter_buffer_during_bridge=true"/>        
<action application="set" data="jitterbuffer_msec=15:100"/>-->
<action application="jitterbuffer" data="60"/>
        <action application="answer"/>
        <action application="conference" data="$1@cdquality"/>
      </condition>
    </extension>
</include>

BBB_ECHO_TO_CONFERENCE.XML

<include>
  <extension name="ECHO_TO_CONFERENCE">
    <condition field="${bbb_from_echo}" expression="true" break="on-false"/>
    <condition field="destination_number" expression="^(ECHO_TO_CONFERENCE)$">
<action application="set" data="rtp_jitter_buffer_plc=true"/>
<!--action application="set" data="rtp_jitter_buffer_during_bridge=true"/>
<action application="set" data="jitterbuffer_msec=15:100"/> -->
<action application="jitterbuffer" data="60"/>
      <action application="answer"/>
      <action application="conference" data="${vbridge}@cdquality"/>
    </condition>
  </extension>
</include>

[Semelovich]

The default opus template https://github.com/fusionpbx/fusionpbx/blob/master/resources/templates/conf/autoload_configs/opus.conf.xml seems to work fine too

But you need to have settings in conference.conf.xml (energy level mostly) ,bbb_conference.conf.xml and bbb_echo_to_conference.conf.xml as in the files Roberto posted.

So, looking at the Roberto's changes, I presume it's mostly about jitter buffer setting and soem minor things.

[iorobertob]

Hi there, I read this here and I feel I need to give the right credit to my colleague @Vytenis, it was mostly him in a joint exploration to find those settings.

Now, we are still facing the issues with disabling the AGC and AEC in the browser side (which still degrades the audio quality). In the same thread, Stephen mentioned editing of the bbb webrtc bridge and sip JS files, looking for the getUserMedia function, but still we haven't been able to get that to work. If anyone had some progress on that front it would be great to discuss that.

[prlanzarin]

@iorobertob The files you're looking for to control AGC/AEC in PCs are here: https://github.com/bigbluebutton/bigbluebutton/blob/develop/bigbluebutton-html5/public/compatibility/sip.js, https://github.com/bigbluebutton/bigbluebutton/blob/develop/bigbluebutton-html5/imports/api/audio/client/bridge/sip.js. Can you confirm these were the ones you're fiddling with?

[iorobertob]

Hi @prlanzarin thanks a lot for your reply, actually not files i am fiddling with, in Ubuntu, are not in the dev fork, but in the installed BBB distro: /var/www/bigbluebutton/client/lib/bbb_webrtc_bridge_sip.js and /var/www/bigbluebutton/client/lib/sip.js

As we haven't forked the dev [yet], at least didn't think we'd have to go that far.

[prlanzarin]

@iorobertob Are you using the HTML5 client or the Flash client? Those files apply only to the Flash client.

[bruno-berchielli]

I'm suffering with the same problema, crackly audio on a fresh install of BBB 2.2 (c5.2xlarge).

With the suggestions made by @Semelovich @iorobertob the problem seems to be fixed.

Of course this problem still require deeper investigation. Will post any new findings about this.

Thanks

[iorobertob]

@prlanzarin Oh I see... we are using HMTL5. So then I understand the way forward would be to fork the dev and fiddle with the files you mentioned earlier, right?

Thanks again!!!

[prlanzarin]

@iorobertob I think you can alter the bundled files directly for the HTML5 client as well in /usr/share/meteor/bundle/programs/web.browser/app/compatibility/sip.js. Be aware of caching problems. Also restart bbb-html5 after changing stuff.

Optional AGC/AEC is something I think would be cool in BBB, though. If you end up forking it and extending the HTML5 client with it, maybe you could open a PR.

[davidpesce]

I'm not sure if this is helpful, but the crackling audio seems to occur when there is an iOS device connected. We can replicate this within 10mins of starting a meeting when using two iphones.

[bruno-berchielli]

Hello, would like to share that by only ajusting /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml in the "cdquality" profile to

<param name="energy-level" value="20"/> (default 200)

Most of the crackly the sound is gone. It's not perfect, but far better.

[Semelovich]

Attention!

It looks like these changes (https://github.com/bigbluebutton/bigbluebutton/issues/7007#issuecomment-605628743) might cause audio\video out of sync issue!

They do fix more or less crackly sounds but out of sync issue appears!

See latest posts https://groups.google.com/forum/#!topic/bigbluebutton-setup/3Y7VBllwpX0

[iorobertob]

Hi there, following this thread, yes we also noticed sync issues, and are still to explore the jitter config.

Besides that, thanks to @prlanzarin !!! after your advice I was able to disable AGC, AEC and NS by hacking the sip.js file on

/usr/share/meteor/bundle/programs/web.browser/app/compatibility/sip.js

and force my own constraints into the mediaStream. I also made the changes in the forked version, but I think I would still be a long way from putting something together for a PR.

For the time being, the changes (see post https://groups.google.com/forum/#!topic/bigbluebutton-setup/3Y7VBllwpX0) work and now audio quality is perfect, only facing dropped samples, but at least no auto gaining for now.

[iorobertob]

Just a note, as mentioned in that google group thread: these constraints included "echoCancellation: false", which would induce feedback in a normal conference. Four our music purposes this is fine, of the routing we do within our DAWs, but just in case someone runs into that. Aand, these last changes do not necessarily contribute to fix crackling, just to the overall quality of the sound, so maybe this should be dealt with in a separate issue/thread?

[nplesa]

Hi , We are using BBB version 2.2. The system is working fine except an audio issue that we can not properly identify:

During the conference, before someone start speaking we can hear a small noise (half second) . We usually don't hear this noise when someone speaks continously but it appears as soon as it stop and speak again. We don't know where this issue comes from and it is not related to the server hardware performance/bandwidth.

[spinteractive]

Hi, I just got the latest BBB up on AWS C5.2xLarge. I tried a demo meeting with 3 users. The audio started crackling after 2 few minutes after each test. We are using iPhones. We tested with Zoom and it was perfect :(

I'm not sure if I understood a fix or settings to adjust on the server in this thread. Can someone explain?

[Semelovich]

Hi, I just got the latest BBB up on AWS C5.2xLarge. I tried a demo meeting with 3 users. The audio started crackling after 2 few minutes after each test. We are using iPhones. We tested with Zoom and it was perfect :(

I'm not sure if I understood a fix or settings to adjust on the server in this thread. Can someone explain?

For the time being there is no fix. Fixing it the way we tried introduces other issues (audio-video delay). Making proper changes to BBB+Freeswitch configs requires a freeswitch expert.

[chaosgrid]

I can confirm that we have the exact same audio crackling issue. I mean, it is not conference-breaking but very annoying nonetheless. To me, it seems like a buffer under-run issue. I noticed that bbb voice latency is relatively low compared to i.e. Jitsi (Jitsi has 1000ms+ compared to bbb on the same server instance).

Now, one would assume that to fix this, it would be very simple: increase the buffer size. I, too, have played around with freeswitch jitterbuffer sizes to no avail. I also tried the latest settings posted here: https://groups.google.com/d/msg/bigbluebutton-setup/3Y7VBllwpX0/k0ImGHhyAgAJ

Anyways: My theory right now is that the jitterbuffer in freeswitch is fundamentally broken, or there is something broken with the client-webrtc-bbb-freeswitch routing process.

Here is a simple test everybody can do: Set the jitterbuffer to a fixed 2000msec. Just use the settings from here https://pastebin.com/r5m5Hzgj but replace the jitterbuffer line with this:

<action application="jitterbuffer" data="2000:2000:0"/>

In theory, this should direct freeswitch to create a jitterbuffer with 2000msec fixed length, and no drift. So the expected result should be smooth audio with a delay of at least 2 seconds.

As you might notice, this will not work at all and it gives weird sound errors with echos. As long as this simple test does not work, it does not make any sense to fine-tune jitterbuffer settings when obviously the buffer does not seem to work at all. Or maybe I am misunderstanding the jitterbuffer alltogether?

[ioslabnet]

First of all, thanks for this product to all contributors. We love big blue button. For this subject i can confirm that we are experiencing the same audio crackling issue with a clean setup 2.2. Posted solution at google forms by Alex seems to solve it. However we are not sure about whether it will cause any sync problems or not.

[ritzalam]

Hi everyone,

To really get to the bottom of this and hopefully fix it is to reproduce the issue in the latest FreeSWITCH.

If there are no issues is the latest FreeSWITCH, then the issue is with BBB version of FS or the way BBB uses FS.

For use to get help from FreeSWITCH we have to install the latest FreeSWITCH and do the recording using their WebRTC client. We just use FS out of the box but we're a few weeks behind plus we're on an unsupported distribution (Ubuntu). The libraries might be different between distributions.

If you can install the latest FreeSWITCH and make a recording, please provide the steps that you did so we can also replicate.

When we'll have more time (we're swamped right now due to covid), we'll be able to run it ourselves and get help from FreeSWITCH developers in fixing the issue.

Richard

On Tue, Apr 14, 2020 at 9:46 AM ioslabnet notifications@github.com wrote:

First of all, thanks for this product to all contributors. We love big blue button. For this subject i can confirm that we are experiencing the same audio crackling issue with a clean setup 2.2. Posted solution at google forms by Alex seems to solve it. However we are not sure about whether it will cause any sync problems or not.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bigbluebutton/bigbluebutton/issues/7007#issuecomment-613452147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABNCYDRBRME4XLQXX42B4LRMRSJTANCNFSM4G5Q3OMQ .

--

BigBlueButton Developer http://www.bigbluebutton.org https://github.com/bigbluebutton http://code.google.com/p/bigbluebutton

[Lukas2000]

Hello, would like to share that by only ajusting /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml in the "cdquality" profile to

<param name="energy-level" value="20"/> (default 200)

Most of the crackly the sound is gone. It's not perfect, but far better.

This seemed to help in our tests today. I will also try Alex's solution. For everyone else's reference, Alex recommends these settings: https://pastebin.com/r5m5Hzgj

[chaosgrid]

I can confirm that we have the exact same audio crackling issue. I mean, it is not conference-breaking but very annoying nonetheless. To me, it seems like a buffer under-run issue. I noticed that bbb voice latency is relatively low compared to i.e. Jitsi (Jitsi has 1000ms+ compared to bbb on the same server instance).

Now, one would assume that to fix this, it would be very simple: increase the buffer size. I, too, have played around with freeswitch jitterbuffer sizes to no avail. I also tried the latest settings posted here: https://groups.google.com/d/msg/bigbluebutton-setup/3Y7VBllwpX0/k0ImGHhyAgAJ

Anyways: My theory right now is that the jitterbuffer in freeswitch is fundamentally broken, or there is something broken with the client-webrtc-bbb-freeswitch routing process.

Here is a simple test everybody can do: Set the jitterbuffer to a fixed 2000msec. Just use the settings from here https://pastebin.com/r5m5Hzgj but replace the jitterbuffer line with this:

<action application="jitterbuffer" data="2000:2000:0"/>

In theory, this should direct freeswitch to create a jitterbuffer with 2000msec fixed length, and no drift. So the expected result should be smooth audio with a delay of at least 2 seconds.

As you might notice, this will not work at all and it gives weird sound errors with echos. As long as this simple test does not work, it does not make any sense to fine-tune jitterbuffer settings when obviously the buffer does not seem to work at all. Or maybe I am misunderstanding the jitterbuffer alltogether?

So, after extensive testing, I don't think the jitterbuffer (or a buffer under-run) is to blame for the crackling noises. But I found out something interesting about the jitterbuffer: It is broken if you use "use-dtx=1" (in opus.conf). At least if you use the Chrome browser (which correctly implements dtx), this will direct the browser to not send audio packets when there is silence (discontinuous transmission). However, it seems the jitterbuffer - as implemented in Freeswitch - cannot handle this correctly and will add echo noise if the stream just interrupts. Also, if you continue speaking, it will play back the last bit of the buffer, resulting in a weird experience.

Hello, would like to share that by only ajusting /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml in the "cdquality" profile to <param name="energy-level" value="20"/> (default 200) Most of the crackly the sound is gone. It's not perfect, but far better.

This seemed to help in our tests today. I will also try Alex's solution. For everyone else's reference, Alex recommends these settings: https://pastebin.com/r5m5Hzgj

With the jitterbuffer working correctly, I can confirm that the freeswitch noise-gate function seems to be at fault, at least partially. Every time somebody starts and stops speaking (so the energy-level gate is triggered), a little "crack" is added. But even when I set the energy-level to 0 and also this to be sure: <param name="conference-flags" value="audio-always"/> I still have some cracks from time to time (but it is noticeably better!). No other settings (opus, jitterbuffer, etc..) seem to make a difference. I suspect this is in fact a FreeSwitch problem. Btw. a great way to test is to simply run a fixed sine wave over the wire using i.e. this online tone generator: https://www.szynalski.com/tone-generator/

So for the time being, I strongly recommend to have <param name="use-dtx" value="0"/> in /opt/freeswitch/etc/freeswitch/autoload_configs/opus.conf.xml (I think that's the standard value) and also set the energy-level all the way to 0 in the cdquality profile under /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml

/edit: Actually, according to https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+And+The+Opus+Audio+Codec dtx is enabled by default. Make sure to disable it as explained above!

[papimla]

Upvoting this issue. I think it should be considered release critical because it is degrading the user experience significantly and is widespread.

@ffdixon would you be so kind as to rename this issue ? I originally thought the issue was only in Beta 2.2, but it turns out to be with all 2.2.x releases. Thanks :smiley:

[ffdixon]

Just chatting with the FreeSWITCH devs on this issue. @chaosgrid can you add <action application="set" data="jb_use_timestamps=true"/> to /opt/freeswitch/etc/freeswitch/dialplan/default/bbb_echo_to_conference.xml as shown below

<include>
  <extension name="ECHO_TO_CONFERENCE">
    <condition field="${bbb_from_echo}" expression="true" break="on-false"/>
    <condition field="destination_number" expression="^(ECHO_TO_CONFERENCE)$">
      <action application="set" data="jb_use_timestamps=true"/>
      <action application="set" data="jitterbuffer_msec=20:400"/>
      <action application="answer"/>
      <action application="conference" data="${vbridge}@cdquality"/>
    </condition>
  </extension>
</include>

Restart BigBlueButton and test again.

[chaosgrid]

@ffdixon

In my testing, the jitterbuffer did not really make a difference in the end. But I just want to note that this line is false: <action application="set" data="jitterbuffer_msec=20:400"/> This will not activate a jitter buffer (you can test this with high values like 2000:2000). The correct syntax within the dialplan config is this: <action application="jitterbuffer" data="20:400"/>

At this point, I'm pretty much convinced that the audio mixing in freeswitch is at fault. You can easily test this on demo.bigbluebutton.org as well, which has the same crackling issues. I'm not sure what I am supposed to test any further since I have no idea of the inner workings of freeswitch and to be honest I lack the time and knowledge to setup a freeswitch testing environment without BigBlueButton.

In general, I do not understand the fascination with freeswitch. No offense, but for conferencing, what freeswitch actually does in the background is not clear at all. There seems to be some mute-everybody-except-the-loudest-person mixing going on which you cannot influence via configuration (I searched...) Also, I do not really see the benefit in only sending out one audio stream since there will only be a few people talking together at most, so bandwidth-wise the worst per person is only around 5x100kbit = 0,5mbit downstream. In my opinion, just forwarding audio streams could be beneficial: less CPU load, less components (like freeswitch) that can introduce problems.

Unrelated to the crackling issue: If you run your server on VMWare esxi, have a look at this excellent document to get your server performance to as close as possible to bare-metal/native: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/latency-sensitive-perf-vsphere55-white-paper.pdf

[chaosgrid]

To get an idea of the severity of this, consider this 10 second signal comparison:

Note that this is very unlikely to be caused by network problems because I have the exact same behaviour on my own Jitsi and BigBlueButton instances.

/edit: Oh, and for Jitsi I connected 3 clients, otherwise you are in peer2peer-Mode which is not fair for this comparison.

[znerol]

In general, I do not understand the fascination with freeswitch.

One of my favorite features of BBB is the option to connect FreeSWITCH to a PBX. Like this people can use their phones to join the conference. Also this allows us to dial in with good conference phones (hardware).

[znerol]

Comparing FreeSWITCH vanilla with the one shipping with BBB, I find the following:

Vanilla (on buster):

# apt list --installed | grep -i opus
freeswitch-mod-opus/testing,now 1.10.2~release~14~f7bdd3845a~buster-1~buster+1 amd64 [installed]
freeswitch-mod-opusfile/testing,now 1.10.2~release~14~f7bdd3845a~buster-1~buster+1 amd64 [installed]
libopus0/stable,now 1.3-1 amd64 [installed,automatic]
libopusfile0/stable,now 0.9+20170913-1 amd64 [installed,automatic]
# ldd /usr/lib/freeswitch/mod/mod_opus* | grep -i opus
/usr/lib/freeswitch/mod/mod_opus.so:
    libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f5d937b1000)
/usr/lib/freeswitch/mod/mod_opusfile.so:
    libopusfile.so.0 => /usr/lib/libopusfile.so.0 (0x00007fba8f96a000)
    libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007fba8e018000)

BBB (on xenial):

# apt list --installed | grep -i opus
libopus0/xenial,now 1.1.2-1ubuntu1 amd64 [installed,automatic]
libopusenc0/xenial,now 0.2.1-1bbb1 amd64 [installed,automatic]
libopusfile0/xenial,now 0.7-1 amd64 [installed,automatic]
# ldd /opt/freeswitch/lib/freeswitch/mod/mod_opus* | grep -i opus
/opt/freeswitch/lib/freeswitch/mod/mod_opus.la:
/opt/freeswitch/lib/freeswitch/mod/mod_opus.so:
    libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f417cc19000)
/opt/freeswitch/lib/freeswitch/mod/mod_opusfile.la:
/opt/freeswitch/lib/freeswitch/mod/mod_opusfile.so:
    libopusfile.so.0 => /usr/lib/libopusfile.so.0 (0x00007f7f0ab89000)
    libopusenc.so.0 => /usr/lib/x86_64-linux-gnu/libopusenc.so.0 (0x00007f7f0a97d000)
    libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f7f07934000)

The libopusenc0 package comes from the bbb ppa (source is likely bigbluebutton/libopusenc). Not sure why BBB FreeSWITCH is linking that library at all, upstream doesn't.

@ffdixon does code from libopusenc0 run in the conference signal path?

[ritzalam]

On Fri, Apr 17, 2020 at 6:37 AM znerol notifications@github.com wrote:

Comparing FreeSWITCH vanilla with the one shipping with BBB, I find the following:

Vanilla (on buster):

apt list --installed | grep -i opus

freeswitch-mod-opus/testing,now 1.10.2~release~14~f7bdd3845a~buster-1~buster+1 amd64 [installed] freeswitch-mod-opusfile/testing,now 1.10.2~release~14~f7bdd3845a~buster-1~buster+1 amd64 [installed] libopus0/stable,now 1.3-1 amd64 [installed,automatic] libopusfile0/stable,now 0.9+20170913-1 amd64 [installed,automatic]

ldd /usr/lib/freeswitch/mod/mod_opus* | grep -i opus

/usr/lib/freeswitch/mod/mod_opus.so: libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f5d937b1000) /usr/lib/freeswitch/mod/mod_opusfile.so: libopusfile.so.0 => /usr/lib/libopusfile.so.0 (0x00007fba8f96a000) libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007fba8e018000)

BBB (on xenial):

apt list --installed | grep -i opus

libopus0/xenial,now 1.1.2-1ubuntu1 amd64 [installed,automatic] libopusenc0/xenial,now 0.2.1-1bbb1 amd64 [installed,automatic] libopusfile0/xenial,now 0.7-1 amd64 [installed,automatic]

ldd /opt/freeswitch/lib/freeswitch/mod/mod_opus* | grep -i opus

/opt/freeswitch/lib/freeswitch/mod/mod_opus.la: /opt/freeswitch/lib/freeswitch/mod/mod_opus.so: libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f417cc19000) /opt/freeswitch/lib/freeswitch/mod/mod_opusfile.la: /opt/freeswitch/lib/freeswitch/mod/mod_opusfile.so: libopusfile.so.0 => /usr/lib/libopusfile.so.0 (0x00007f7f0ab89000) libopusenc.so.0 => /usr/lib/x86_64-linux-gnu/libopusenc.so.0 (0x00007f7f0a97d000) libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f7f07934000)

The libopusenc0 package comes from the bbb ppa (source is likely bigbluebutton/libopusenc https://github.com/bigbluebutton/libopusenc). Not sure why BBB FreeSWITCH is linking that library at all, upstream doesn't.

@ffdixon https://github.com/ffdixon does code from libopusenc0 run in the conference signal path?

We had to build our own library. Otherwise, FS throws opus errors and can't record.

We are trying to figure out how to take FS without building our own libs.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bigbluebutton/bigbluebutton/issues/7007#issuecomment-615174813, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABNCYFWAW6RERUYJ4GVGKDRNAWPDANCNFSM4G5Q3OMQ .

--

BigBlueButton Developer http://www.bigbluebutton.org https://github.com/bigbluebutton http://code.google.com/p/bigbluebutton

[hostbbb]

I'm not sure if this is helpful, but the crackling audio seems to occur when there is an iOS device connected. We can replicate this within 10mins of starting a meeting when using two iphones.

Had customer on IOS 12 with crackling, updating to 13 solves the issue

[hostbbb]

I can confirm that we have the exact same audio crackling issue. I mean, it is not conference-breaking but very annoying nonetheless. To me, it seems like a buffer under-run issue. I noticed that bbb voice latency is relatively low compared to i.e. Jitsi (Jitsi has 1000ms+ compared to bbb on the same server instance). Now, one would assume that to fix this, it would be very simple: increase the buffer size. I, too, have played around with freeswitch jitterbuffer sizes to no avail. I also tried the latest settings posted here: https://groups.google.com/d/msg/bigbluebutton-setup/3Y7VBllwpX0/k0ImGHhyAgAJ Anyways: My theory right now is that the jitterbuffer in freeswitch is fundamentally broken, or there is something broken with the client-webrtc-bbb-freeswitch routing process. Here is a simple test everybody can do: Set the jitterbuffer to a fixed 2000msec. Just use the settings from here https://pastebin.com/r5m5Hzgj but replace the jitterbuffer line with this: <action application="jitterbuffer" data="2000:2000:0"/> In theory, this should direct freeswitch to create a jitterbuffer with 2000msec fixed length, and no drift. So the expected result should be smooth audio with a delay of at least 2 seconds. As you might notice, this will not work at all and it gives weird sound errors with echos. As long as this simple test does not work, it does not make any sense to fine-tune jitterbuffer settings when obviously the buffer does not seem to work at all. Or maybe I am misunderstanding the jitterbuffer alltogether?

So, after extensive testing, I don't think the jitterbuffer (or a buffer under-run) is to blame for the crackling noises. But I found out something interesting about the jitterbuffer: It is broken if you use "use-dtx=1" (in opus.conf). At least if you use the Chrome browser (which correctly implements dtx), this will direct the browser to not send audio packets when there is silence (discontinuous transmission). However, it seems the jitterbuffer - as implemented in Freeswitch - cannot handle this correctly and will add echo noise if the stream just interrupts. Also, if you continue speaking, it will play back the last bit of the buffer, resulting in a weird experience.

Hello, would like to share that by only ajusting /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml in the "cdquality" profile to <param name="energy-level" value="20"/> (default 200) Most of the crackly the sound is gone. It's not perfect, but far better.

This seemed to help in our tests today. I will also try Alex's solution. For everyone else's reference, Alex recommends these settings: https://pastebin.com/r5m5Hzgj

With the jitterbuffer working correctly, I can confirm that the freeswitch noise-gate function seems to be at fault, at least partially. Every time somebody starts and stops speaking (so the energy-level gate is triggered), a little "crack" is added. But even when I set the energy-level to 0 and also this to be sure: <param name="conference-flags" value="audio-always"/> I still have some cracks from time to time (but it is noticeably better!). No other settings (opus, jitterbuffer, etc..) seem to make a difference. I suspect this is in fact a FreeSwitch problem. Btw. a great way to test is to simply run a fixed sine wave over the wire using i.e. this online tone generator: https://www.szynalski.com/tone-generator/

So for the time being, I strongly recommend to have <param name="use-dtx" value="0"/> in /opt/freeswitch/etc/freeswitch/autoload_configs/opus.conf.xml (I think that's the standard value) and also set the energy-level all the way to 0 in the cdquality profile under /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml

/edit: Actually, according to https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+And+The+Opus+Audio+Codec dtx is enabled by default. Make sure to disable it as explained above!

I know on chrome AEC if jitter gets past 135? (cant remember exact) the echo cancellation has to retrain, hence you here echo come back into to mix... this happens last year when they introduced the new AEC3, and did some A/B testing, lots of users starting getting audio issues.

Has anyone compare the audio differences with FireFox, and Chrome? is issue more apparent on Chrome? I would assume 790-80% user base is is yesing by now.

[hostbbb]

To get an idea of the severity of this, consider this 10 second signal comparison:

Note that this is very unlikely to be caused by network problems because I have the exact same behaviour on my own Jitsi and BigBlueButton instances.

/edit: Oh, and for Jitsi I connected 3 clients, otherwise you are in peer2peer-Mode which is not fair for this comparison.

how is audio captured? from recording files of meetings, or the input or output source on laptop?

[znerol]

The correct syntax within the dialplan config is this: <action application="jitterbuffer" data="20:400"/>

Correct. The other syntax is used in order to set channel variables according to FreeSWITCH docs. Btw., the jitterbuffer of a specific stream can be checked using the fs_cli:

uuid_jitterbuffer <uuid> debug:DEBUG

It is also possible to change the jitterbuffer from withir the fs_cli, e.g.:

uuid_jitterbuffer <uuid> 20:400

It looks like jitterbuffer improves the situation in my case quite a bit (using the correct dialplan syntax of course). Also lowering the energy-level has a positive effect. So currently I'm using the following fixes: fs-conf.diff.txt

[dtaht]

I do think bufferbloat, worldwide, is causing folk no end of hurt. jitterbuffer or no jitterbuffer, lots of jitter makes for a mess, and I have seen bloat - particularly on wifi and lte - in the 10+sec range, with nearly a second fairly typical in cable and dsl. I know it's a bit late in the crisis to convince everyone experiencing issues to go install a router with smart queue management (fq_codel or sch_cake) on it, but it would be a better world, and for those of you that can turn on some form of decent qos/sqm, I'd urge you to urge your users to do so.

I'm also seeing the server vms themselves have increasingly bad network and cpu scheduling behavior. I put in a call to linode just now on one of mine.

An example of a lousy AP, now fixed... https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/

And if you need a laugh and an explanation of what you are trying to compensate for...

https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/

[ffdixon]

@chaosgrid Good investigation.

The correct syntax within the dialplan config is this:
<action application="jitterbuffer" data="20:400"/>

We're going to make changes in BigBlueButton 2.2.6 to use this correct syntax. We're currently testing with the following value from znerol (feedback welcome if you think the above is better).

<action application="jitterbuffer" data="60"/>

[Semelovich]

So currently I'm using the following fixes: fs-conf.diff.txt

action application="jitterbuffer" data="60"

I tried that. It does reduce crackling but it makes it worse in general if you are using a not ideal 3g network with packet loss and jitter. We had a teacher who had to use her crappy 3g connection and the issues with audio were pretty severe until we started using a "X:X:X' jitter buffer formula with a large maximum value, which in turn introduced crackling =)

What about audio-video being out of sync? Have you experienced it with your JB (fixed at 60) settings?

PS I've been testing with action application="jitterbuffer" data="60:400:20" (+energy level=0) And here are my observation: 1) Teachers using wired connections with no packet tloss=> practically non-existent crackling, barely audible.

2) Another teacher was using a 3g connection => crackling was there, not as severe as using the default BBB settings though.

I can't comment on audio-video being out of sycn though coz they didn't use webcams.

PPS Btw about jitsi. Its audio is great when it works, yes.
When out teacher had some serious issues with his connection Jitsi couldn't deliver at all (we tried to switch to it mid-session) audio was constantly freezing, whereas BBB\freeswitch at least tried and we could hear ~90%. There was lots of crackling and audio getting interrupted often but it was at least something .

[chaosgrid]

@chaosgrid Good investigation.

The correct syntax within the dialplan config is this:
<action application="jitterbuffer" data="20:400"/>

We're going to make changes in BigBlueButton 2.2.6 to use this correct syntax. We're currently testing with the following value from znerol (feedback welcome if you think the above is better).

<action application="jitterbuffer" data="60"/>

@ffdixon

IIRC if you specify 60, it means internally 60ms length with maximum of 300ms - which is fine I guess, as long as use-dtx is "0" in opus.conf (I think that's the default in BBB config). But please note that it did not really seem to make a large difference regarding the audio cracks - they seem to be there no matter the jitterbuffer which is very odd and we should rather find the root cause of the cracks. Also, as said elsewhere, the added jitterbuffer actually causes more audio latency and therefore audio-video sync issues. I'm actually not sure the jitterbuffer helps at all so I'm unsure whether it should be activated in general since it could result in the majority of users experiencing audio-video sync issues with no significant improvement in audio quality (note that the standard config right now does not activate any jitterbuffer because of the wrong syntax).

[znerol]

it could result in the majority of users experiencing audio-video sync issues

This is a side-effect resulting from the architecture. Video is SFUed between users and thus there is only one jitterbuffer involved at the receiving end (i.e. in the users browser). Since audio is mixed server-side some additional latency is practically unavoidable.

[chaosgrid]

it could result in the majority of users experiencing audio-video sync issues

This is a side-effect resulting from the architecture. Video is SFUed between users and thus there is only one jitterbuffer involved at the receiving end (i.e. in the users browser). Since audio is mixed server-side some additional latency is practically unavoidable.

Exactly, and when you add an audio buffer on the server-side while you don't have that extra buffer for video, there is an increased chance that audio gets delayed more compared to video (since video is simply forwarded). Another reason I would prefer an audio-sfu solution (could be optional).