AEC doesn't work well with &buffer or &sync

briggeml commented 3 years ago

Two links for example: https://obs.ninja/?push=da6786&vb=2000&ab=510&buffer=0&sync=0&aec=1&denoise=0&autogain=0&codec=vp9&q=0&fps=25&record&view=6f1e59 https://obs.ninja/?push=6f1e59&vb=2000&ab=510&buffer=0&sync=0&aec=1&denoise=0&autogain=0&codec=vp9&q=0&fps=25&record&view=da6786

steveseguin commented 3 years ago

Not sure I can reproduce this issue yet; I'll keep trying though when I get my laptop setup in the other room.

Looking at the audio settings once loaded though, it does say the echo cancellation is on.

briggeml commented 3 years ago

Yes, it's enabled in the settings, but in fact the does not work, I can hear my echo. You need a setup with speakers (not headphones) on both sides. However, in director room AEC working fine. Also we have made audio-video tests confirming this, we can send you on request.

steveseguin commented 3 years ago

thank you. I'll try to figure it out. Much appreciate the bug report

briggeml commented 3 years ago

As I mentioned before at https://github.com/steveseguin/obsninja/issues/492 it would be great to have some metrics showing how the AEC works (like this is done in Extron for example):

A shallow analysis of https://chromium.googlesource.com/external/webrtc/+/3f08dc656dc22edf658a8393b5b03a46b23aa4e8/webrtc/modules/audio_processing/aec/echo_cancellation.cc shows that this is possible without modifying the C-sources

steveseguin commented 3 years ago

What metrics in particular? More importantly, what Javascript functions are callable to access this data?

I'd love to add metrics to help you out, but I'm at a loss of which stats or how to access them. These are the stats I have access to as far as I can tell.

briggeml commented 3 years ago

We need at least this to output:

ERL Meter — ERL (echo return loss) is the ratio of the far end signal at the reference input to the far end signal received at the mic input (in dB). This meter should read between -10 dB and +10 dB for proper AEC operation.
ERLE Meter — ERLE (echo return loss enhancement) is the amount of potential echo signal that the AEC algorithm is cancelling (in dB).
TER Meter — TER (total echo reduction) is the sum of ERL and ERLE and represents the total amount of echo reduction (in dB).

WebRtcAec_get_echo_status do most of that work (I don't know if you can get the result in JS)

int WebRtcAec_get_echo_status(void handle, int status);

steveseguin commented 3 years ago

Thank you.

I don't know of any way to access this data without modifying the C code; perhaps I'd be able to do that with the Electron Capture app though. I'll spend some time searching to see if I can find an accessible option.

Thank you for the request and direction

briggeml commented 3 years ago

This bug is quite critical, but it is not even marked as a bug. Is there anything I can do to help, collect tests, samples?

steveseguin commented 3 years ago

Hi @briggeml

Thank you for following up on this issue report.

I've tested and confirmed with others at the time of posting this issue that the faux rooms do work with AEC. AEC does work within the context of a faux-room, however, I apologize for not looking closer at your custom links though as I suspect I see some issues there.

Try removing the following parameters from the links you are using.

&buffer=0&sync=0

These commands could cause the echo cancellation to fail or not perform well, as they introduce audio delay into a pipeline that will reduce the ability of echo cancellation to work as expected. The buffer and sync values were mainly designed for use in a view-only capture scenario, and not with in the context of a performer link. Buffer and sync contain and use audio delay nodes.

What I can do is to investigate how to improve AEC when combined with audio delay nodes, although when I last looked at this I recall Chrome was highly restrictive as to what I could and could not do here. I have very limited control over AEC or how Chrome will let me route audio via web audio nodes, but I'll try to find time to explore some alternative options at some point soon.

I'd be happy to test some links with you; just poke me on Discord

briggeml commented 3 years ago

Thanks, Steve, AEC works after I removed these params! However, the question remains, what else params conflicting or cause unpredictable behavior with AEC (or each other)?

steveseguin commented 3 years ago

re: conflicts with AEC, I'll list the issues that come to mind:

The &stereo&aec is a combo that might cause issues, as stereo audio sometimes will convert to mono when using AEC. Maybe on some browsers AEC won't work at all if stereo channels are used. This is on a per-browser basis.

Some mobile / embedded devices will use their own hardware-based aec; such as some bluetooth headphones, and in those cases I cannot disable AEC effectively.

If the audio is delayed, sometimes by 500 to 100ms even without the buffer flag added, the AEC might fail.

AEC might not work well for the first 30 seconds of a call, as it dynamically adapts.

Auto-gain can use AEC to have problems, as if someone hasn't spoken for a while, then speaks, the AEC might not be calibrated correctly and it will take a few moments for it to adjust.

If you have audio sources playing in two different windows (each with its own audio context), in Chrome the AEC won't work. The AEC only works usually if used in the same browser window. This could potentially occur with IFRAMES?

re: better noise cancellation and echo cancellation, I think Google Meets (enterprise?) does noise reduction in the cloud, which offers better results than in-browser? Dolby.io also offers these types of services; https://dolby.io/products/media-processing

I'd like to add deep-learning custom audio filters to OBS.Ninja at some point, as I do deeply enjoy that field, and the first thing I'll add is likely a declick filter. I'm not sure I'll be able to offer improved AEC though; seems pretty challenging.

steveseguin / vdo.ninja

AEC doesn't work well with &buffer or &sync #733

int WebRtcAec_get_echo_status(void handle, int status);