search

kclyu / rpi-webrtc-streamer

This repo's objective is providing something like Web Cam server on the most popular Raspberry PI hardware. By integrating [WebRTC](https://webrtc.org/native-code/) and Raspberry PI, we can stream the Raspberry camera feed to browser or native client which talks WebRTC.
Other
623 stars 108 forks source link

Echo Cancellation not working #89

Open mintu19 opened 5 years ago

mintu19 commented 5 years ago

I am using raspberry pi to work on video call. I have enabled audio processing and echo cancellation. I have mic and speaker inside same body. But echo cancellation is not working and my voice is echoing.

mintu19 commented 5 years ago

Log: (specific part)

Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (webrtcvoiceengine.cc:324): WebRtcVoiceEngine::ApplyOptions: AudioOptions {aec: true, agc: true, ns: true,
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_impl.cc:849): BuiltInAECIsAvailable
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_generic.cc:17): virtual bool webrtc::AudioDeviceGeneric::BuiltInAECIsAvailable() const: Not
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_impl.cc:852): output: 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (apm_helpers.cc:118): Echo control set to 1 with mode 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (apm_helpers.cc:128): EC metrics set to 1
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_impl.cc:865): BuiltInAGCIsAvailable
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_generic.cc:27): virtual bool webrtc::AudioDeviceGeneric::BuiltInAGCIsAvailable() const: Not
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_impl.cc:868): output: 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (apm_helpers.cc:82): AGC set to 1 with mode 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_impl.cc:881): BuiltInNSIsAvailable
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [786] (audio_device_generic.cc:37): virtual bool webrtc::AudioDeviceGeneric::BuiltInNSIsAvailable() const: Not s
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:217] [791] (paced_sender.cc:112): PacedSender resumed.
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_device_impl.cc:884): output: 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (apm_helpers.cc:153): NS set to 1
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:512): NetEq capacity is 50
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:518): NetEq fast mode? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:536): Delay agnostic aec is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:546): Extended filter aec is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:556): Experimental ns is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:562): Intelligibility Enhancer is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_processing_impl.cc:710): Highpass filter activated: 1
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (fixed_gain_controller.cc:50): Gain to apply: 0 db.
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_processing_impl.cc:724): Gain Controller 2 activated: 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_processing_impl.cc:726): Pre-amplifier activated: 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_device_impl.cc:764): RecordingIsInitialized
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_device_impl.cc:826): Recording
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_device_impl.cc:745): InitRecording
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (audio_device_impl.cc:764): RecordingIsInitialized
mintu19 commented 5 years ago

any solution?

kclyu commented 5 years ago

echo cancellation is a very sensitive topic to answer. It seems that what hardware to use and what environment to use is the main issue. In addition, rws uses S/W method, so it can not be a special solution because the contents of issue can be echo reduction. For this problem, it seems to be the best way to find a way to reduce the interference between the speaker and the mic as much as possible.

mintu19 commented 5 years ago

I calculate 200 ms around echo tail for the echo. Also noticed that loud volume equals less echo but very loud volume equals clear echo. Low volume introduces chirping kind of effect.

Have also used the same body (mic and speaker and sound card inside same body) with pc on web app, and there is no echo when used with web version of webrtc.

Reducing interference between them means moving mic closer?

kclyu commented 5 years ago

If you are currently using deb, you might be able to compile it with the latest native stack and try it out. The audio related parts of the native stack are now changed and improved, but RWS and PC will not have the same results. I want to do the related tests, but I still do not have enough time to do audio tests.

mintu19 commented 5 years ago

which deb? Can you guide me in short how to do that?

kclyu commented 5 years ago

Are you cross compiling rws directly?

mintu19 commented 5 years ago

oh rws deb. Yes sorry, I am using rws deb file. How to compile it with latest native stack?

kclyu commented 5 years ago

README_building.md explains how to cross compile in Ubuntu environment.

mintu19 commented 5 years ago

oh You are asking to rebuild rws. ok. Latest version or deb commit? Also can i build it directly on pi or is it very resource consuming?

kclyu commented 5 years ago

You can not build WebRTC native stack on pi. The only way to build rws is to cross compile.

mintu19 commented 5 years ago

ok. I will try and let you know. Deb commit or latest? I think there are changes in latest.

Also do you know how to enable these things if i want to try them.

Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:536): Delay agnostic aec is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:546): Extended filter aec is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:556): Experimental ns is enabled? 0
Jun 08 11:02:22 raspberrypi webrtc-streamer[310]: [842:218] [786] (webrtcvoiceengine.cc:562): Intelligibility Enhancer is enabled? 0
kclyu commented 5 years ago

There are a lot of changes in the related part of WebRTC native code, so I need to look at the WebRTC native code source in order to answer correctly. I do not know if PI can handle all audio related settings because CPU usage is high.

mintu19 commented 5 years ago

ok i will try building and searching by myself. Suggest me which commit I should build because I think latest commit have configurational changes as well.

mintu19 commented 5 years ago

My observations after testing current commit:

1) Media_config.conf option "audio_echo_cancellation" should actually be "audio_echo_cancel" as defined in config_media.def?? Also change in high pass filter which should be "highpass_filter" instead "high_passfilter".

2) I have added few more filters with aec (streamer.cc)

if( config_media->GetAudioEchoCancel() == true ) {
            options.echo_cancellation = absl::optional<bool>(true);
            options.extended_filter_aec = absl::optional<bool>(true);
            options.delay_agnostic_aec = absl::optional<bool>(true);
            options.residual_echo_detector = absl::optional<bool>(true);
}

3) Can also add support for experimental NS same way options.experimental_ns = absl::optional<bool>(true); 4) Auto Gain control is check with Echo Cancel? Should be GetAudioGainControl()

if( config_media->GetAudioEchoCancel()  == true )
            options.auto_gain_control = absl::optional<bool>(true);

5) Even after echo cancellation, now echo is completely suppressed. But instead of echo i am getting drilling noise in the end. No noise till i keep speaking but noise at the end. Which i think can be due to 2 reasons either because of suppression or because of echo delay.

Now i am not able to find how to set_stream_delay_ms in RPI-Webrtc-Streamer. Can you suggest me with a quick look at code?

kclyu commented 5 years ago
  1. The content of the media_config.conf file has changed since the last deb distribution was created, so the document has not yet been modified. I need to fix it later before next deb deployment.

2.,3.,4, Audio related settings will be modified later with the options of auidoOptions.

  1. Why do not you try adding and testing audioOptions jitter buffer related settings? What you're talking about seems more related to the jitter buffer (NetEq).

Although set_stream_delay_ms is configurable in AudioProcessing, it is not exactly checked, but the delay is related to audio capture delay. As mentioned above, i think, it is better to check the jitter buffer config.

To test set_stream_delay_ms, 

170 peer_connection_factory_ = webrtc :: CreatePeerConnectionFactory (
171       network_thread_.get (), worker_thread_.get (),
172        signaling_thread_.get (),
173        adm_.get () / * adm * /,
174        audio_encoder_factory, audio_decoder_factory,
175       std :: move (video_encoder_factory), std :: move (video_decoder_factory),
176       nullptr / * audio_mixer * /, nullptr / * audio_processor * /);

AudioProcessing is the audio_processor, 176 line in streamer.cc. You need to create that object and pass it to CreatePeerConnection. iOS and android do not use this setting either. It is not intended to be used by the user anyway.

I think you are doing a lot of audio tests compared to others, in fact I think that software audio processing will be a bit overhead with Raspberry PI CPU. If you share test results with hardware information such as CPU usage, mic / speaker, etc., it will be very helpful to others.

mintu19 commented 5 years ago

I tried changing jitter settings but no solution. I am trying to use audio echo delay setting. But having problem with that right now.

Also I am getting another new problem now that is, audio and video is out of sync (possibly due to audio processing). Audio comes later than video.

I have also tried audio with uv4l where there is no echo and no noise. But since it is closed source, can't recreate that code.

i have so far tried to use all combo bw all jitter, echo and ns options... CPU usage is around 30-35% per core in every combo (is it strange?). I am using Raspberry PI model 3 A+ kernel: 4.19.49-v7+. But still after all combo there is some residual echo giving problem.

kclyu commented 5 years ago

Have you tried both NS/EC enabled? You have tested the configuration of raspberry pi, and I wonder if you set it on the browser side.

mintu19 commented 5 years ago

Yes i have enabled all ec and ns filters. It is verified in log.

I have tested in on browser/mobile app which was working with ec and ns in uv4l. Need to set something more?

Also while using audio_processing, I get segmentation fault somewhere and program stops. (How to enable trace/debug to find error?)

Edit: Add

So i have debugged where seg fault is there. If i even declare a audio processing class pointer, rtc::scopedrefptr apm; even if i don't use it, it gives seg fault.

mintu19 commented 5 years ago

I tried various options yet no luck with AEC.

blademckain commented 4 years ago

@kclyu can u share your WebRTC native-code library builded for Raspberry pi 3 ( armv7 ) and Raspberry pi zero ( armv6 )

Thx