Handle audio issues like feedback (semi) automatically

[ann0see]

Continuing #797

In short: How can we improve the following situation:

1. Inexperienced user connects to a Jamulus session with wrongly configured inputs/outputs —> feedback
2. Inexperienced users haven’t set new client level
3. How do we ensure that he knows about this issue and how to fix it?

I hope that’s right?

Possible solution: automatic feedback detection/suppression?

[ann0see]

It would be difficult for a simple algorithm to detect feedback. Jamulus already indicates (latches) a clipping condition. We could detect when a channel is "too loud" (clipping for a "long" time) and suppress it so that it doesn't make the session unusable for the other musicians. Suppress could mean turn the volume down to something very low (5%?) so that someone could troubleshoot that channel with continuous sound. Suppress could be something more drastic (e.g. kill or disconnect) but then it will be harder to troubleshoot.

@gilgongo We can do more analysis on "abnormal" traffic if the packets are numbered. Then we can do things like (1) detect out of order packets, (2) detect erratic interpacket times, (3) detect a long delay and then see all the late packets causing xxxx, etc. This is interesting for understanding when bad Internet transport is affecting our sound. This could also be a rabbit-hole.

I think numbering the packets and collecting interarrival rates in a log could be really helpful with minimal processing. This still does not help in creating code to detect feedback.

Is it possible some people on this thread are using the term feedback to mean some other audio defect?

Originally posted by @gene96817 in https://github.com/corrados/jamulus/issues/797#issuecomment-752248008

[ann0see]

Detecting feedback would be great. But Jamulus should not automatically mute or lower the volume of this user (this could become really annoying in my opinion). We could add a red warning like the "you are muted" warning which says: "Your signal has been very loud for quite some time! Do you hear feedback? Try to mute yourself and check if you‘ve set up your in/outputs correctly!" [Don‘t show this message again] [Mute myself]

[ann0see]

Detecting feedback will take the same kind of code as eliminating feedback. Let's clarify this thread to dealing with a user that is overwhelming the session with too much sound. Then we can decide what conditions of "too much sound" are bad and need/deserve automatic suppression. Some degree of automatic suppression will be needed, especially for a naive or inexperienced user that will respond slowly to muting themselves.

For fans of loud music, perhaps the threshold needs to be adjustable (i.e. triggering volume and duration or even disabled). Acoustic musicians probably want a lower threshold. People with tinnitus probably need a lower threshold too.

Originally posted by @gene96817 in https://github.com/corrados/jamulus/issues/797#issuecomment-752255460

[gene96817]

If we really want feedback detection, perhaps we should build echo cancellation into the client. However, echo cancellation added delay. Alternatively, we could use the echo cancellation to detect echo but not insert that code in the data path. When the echo cancellation kicks in, we can use that condition to trigger the messages (or indicators) to the user.

What is good about this approach is the client computer should have plenty of compute cycles for this. All the telephony and video conferencing apps have built-in echo cancellation. Most if this is in the codec. Does the codec we use have echo cancellation?

[ann0see]

Echo cancellation would probably result in the typical audio quality issues of video conferencing tools.

If we can use it to detect problems that‘s good.

We should find ways to show the user what he‘s done wrong.

[gene96817]

I was referring to echo cancellation as a class of well-known algorithms that can automatically detect an echo. The problem with video conferencing tools for singing has nothing to do with echo cancellation. It is their assumption that in a well-run meeting, only one person should be speaking at a time. Note, we do not want to run the audio to the server through the echo canceller because the echo canceller has a delay line as part of the algorithm to find the matching pattern (that is the echo). (The delay line stores the original signal for the pattern matching to find the echo.)

[ann0see]

The delay line stores the original signal for the pattern matching to find the echo.

Thanks for the clarification!

[gilgongo]

@ann0see

We can do more analysis on "abnormal" traffic if the packets are numbered.

Do we need to open a ticket for packet numbering or is that already being worked on?

This is interesting for understanding when bad Internet transport is affecting our sound.

By "interesting" do you mean that it would allow a user to be able to know whether (for example) to adjust their client's jitter buffer and not the server one? Or to reduce their audio quality setting rather than their buffer delay setting? Otherwise (as I keep asking), what's the point in having information if you can't reliably act on it? And more to the point - would such adjustments be able to be done automatically without user intervention? That would be cool if so...

[ann0see]

Do we need to open a ticket for packet numbering or is that already being worked on?

No. I don't think someone is working on it. Feel free to open one

[gilgongo]

@ann0see I think an engineer needs to open that ticket really as I'm not qualified to do so.

[gene96817]

Could someone show me what the payload of the UDP packets looks like? I'd be glad to work with someone interested in coding a solution .

[WolfganP]

Could someone show me what the payload of the UDP packets looks like? I'd be glad to work with someone interested in coding a solution

See https://github.com/softins/jamulus-wireshark/

[ann0see]

Will convert this to a discussion since I think it's not yet ready.