MaximKurbanov commented 1 year ago

Package version

3.0.0-pre.6

Environment

* OS: Android
* Unity version:2021.3

Hi!
I have big problem with my voice chat App. When 2 people voice each other by mobile phone. The voice sound of the speaker is recorded back to the microphone, and causing echo. After many times the sound is echoed and resonant with each other, causing loud noises. I've been looking for solutions like AEC(Acoustic Echo Cancellation), Echo Suppression, but it's not work for Unity.
Have you ever had this problem or has any solutions, please let me know. Thanks so much!!!

Steps To Reproduce

Run app on 2 mobile devices

Current Behavior

No response

Expected Behavior

No response

Anything else?

No response

karasusan commented 1 year ago

@MaximKurbanov

651

Is this the same issue?

Joshua-Douglas commented 1 year ago

Hello,

Thank you so much for your tireless efforts with this package @karasusan! My team has been using this package extensively and we've found that it works very well. I'm wondering if this bug might be a good opportunity for our team to contribute back to this repository.

I've noticed that the current implementation hard-codes the AudioOptions within the WebRTCPlugin Context object.

rtc::scoped_refptr<AudioSourceInterface> Context::CreateAudioSource()
    {
        // avoid optimization specially for voice
        cricket::AudioOptions audioOptions; // echo_cancellation will default to nullptr
        audioOptions.auto_gain_control = false;
        audioOptions.noise_suppression = false;
        audioOptions.highpass_filter = false;
        return UnityAudioTrackSource::Create(audioOptions);
    }

Do you think noise_suppression, echo_cancellation, etc would work as expected if these options were set to true? If so, then we might be able to close out this issue by exposing the AudioOptions and allowing end-users to specify echo_cancellation if they need it (this would also close #587 I believe).

If you think this would be a good approach, and if you are open to contributions, then I would be happy to submit a fix. I'd be happy to outline the changes in a design document before submitting a PR, so you could approve the changes before I begin implementing.

karasusan commented 1 year ago

@Joshua-Douglas Hi, always we welcome to your contribution. Making these audio options to public is a first step to solve this issue. I'm not sure these opitions can fix it, therefore we need to test these options.

If you make a PR, we will be glad to review and test.

Joshua-Douglas commented 1 year ago

Hello @karasusan, that is great to hear - thanks!

Would you rather I submit the PR directly, or provide you with an outline of my planned changes before I submit? I don't mind sharing my outlined design before implementing, since these changes will impact the public API.

Either option works for me. I'll keep you updated.

karasusan commented 1 year ago

@Joshua-Douglas I'd like to see your planned changes of the public API. Maybe we need to make a new class to express AudioOptions.

karasusan commented 1 year ago

@Joshua-Douglas Have you already develop for this issue? I'm going to start this task before long.

Joshua-Douglas commented 1 year ago

@karasusan I have not started. If you are ready to start soon, then that would be great! I have another open source project than I'm supporting, so you would probably get it done much faster.

Can't wait to see your changes! It would be great if this does enable echo cancellation🤞!

karasusan commented 1 year ago

@Joshua-Douglas I have already started this issue. This is a draft PR.

978

Can you try that If I make a package file to test this fix?

MaximKurbanov commented 1 year ago

@Joshua-Douglas I have already started this issue. This is a draft PR.

feat: Add AudioOptions class #978

Can you try that If I make a package file to test this fix?

I can.

karasusan commented 1 year ago

I found the issue in our implementation for audio input from Unity. We need to fix it to use these audio options. Sorry but this fix is dropped from the next release version pre.7.

We will focus on this issue in the next version.

Shushpancheak commented 12 months ago

We have tried to implement AudioProcessingModule in native code, and added the following code in what I believe the two points at which your program either receives the audio, or sends it:

UnityAudioTrackSource.cpp

        rtc::scoped_refptr<::webrtc::AudioProcessing> audioProcessingModule = GetAudioProcessingModule();

        if (audioProcessingModule) {
            const size_t sliceDurationMs = 10;
            const size_t sliceLength = (nSampleRate * nNumChannels * sliceDurationMs) / 1000;
            const ::webrtc::StreamConfig streamConfig = StreamConfig(nSampleRate, nNumChannels);

            auto cfg = GetAdditionalAudioProcessingConfig();
            bool invertStreams = cfg.invertStreams;

            for (size_t offset = 0; offset < nNumFrames; offset += sliceLength) {
                size_t sliceLength = std::min(sliceLength, nNumFrames - offset);

                if (!invertStreams) {
                    audioProcessingModule->ProcessStream(&_convertedAudioData[offset], streamConfig, streamConfig, &_convertedAudioData[offset]);
                } else {
                    audioProcessingModule->ProcessReverseStream(&_convertedAudioData[offset], streamConfig, streamConfig, &_convertedAudioData[offset]);
                }
            }
        }

AudioTrackSinkAdapter.cpp

        rtc::scoped_refptr<::webrtc::AudioProcessing> audioProcessingModule = GetAudioProcessingModule();

        if (audioProcessingModule) {
            const size_t sliceDurationMs = 10;
            const size_t sliceLength = (sampleRate * channels * sliceDurationMs) / 1000;
            const ::webrtc::StreamConfig streamConfig = StreamConfig(sampleRate, channels);

            auto cfg = GetAdditionalAudioProcessingConfig();
            bool invertStreams = cfg.invertStreams;

            for (size_t offset = 0; offset < readLength; offset += sliceLength) {
                size_t sliceLength = std::min(sliceLength, readLength - offset);

                if (!invertStreams) {
                    audioProcessingModule->ProcessReverseStream(&_bufferIn[offset], streamConfig, streamConfig, &_bufferIn[offset]);
                } else {
                    audioProcessingModule->ProcessStream(&_bufferIn[offset], streamConfig, streamConfig, &_bufferIn[offset]);
                }
            }
        }

The audioProcessingModule is a Singleton to which we apply some AudioOptions, including AcousticEchoCancellation. However, we still get the echo. @karasusan Maybe you can pinpoint us to the location of the native code where we should look at to fix the problem with your implementation for audio input/output?

karasusan commented 12 months ago

@Shushpancheak I think UnityAudioTrackSource is a right place to process noise cancellation. Maybe there are bugs in the code.

Mahesh218 commented 9 months ago

Could you please let us know if there has been a delivery date set for this fix? We're currently facing some difficulties due to this issue. If possible, could you kindly provide us with a tentative date? Thank you very much for your assistance @karasusan

Shushpancheak commented 8 months ago

Are there any updates?

DanMcPete commented 7 months ago

We are still held back by this problem on ios and android mobile devices. webrtc package is rendered unusable on mobile devices when microphone picks up speaker audio without filter. Can you give any update on this fix? Thank you for your help.