APM process error

[nanguantong]

Version 0.5.0 AudioProcessing （processReverseStream and processStream） has problem that audio sounds abnormal and aec no effect compared with v0.4.0.

AudioProcessingStreamConfig far_frame_config = new AudioProcessingStreamConfig(8000, 1);
AudioProcessingStreamConfig near_frame_config = new AudioProcessingStreamConfig(8000, 1);

And v0.4.0 aec has no effect in some cases compared with c++ native webrtc (chrome master branch) using the same call.

[nanguantong]

Error as following:

Java_dev_onvoid_webrtc_media_audio_AudioProcessing_processReverseStream

**result = apm->ProcessStream(dstFrame, srcConfig, dstConfig, dstFrame);**

in JNI_AudioProcessing.cpp

[nanguantong]

Error as following:

Java_dev_onvoid_webrtc_media_audio_AudioProcessing_processReverseStream

**result = apm->ProcessStream(dstFrame, srcConfig, dstConfig, dstFrame);**

in JNI_AudioProcessing.cpp

Please feel free to fix this bug and publish the release or snapshot to repo，thanks.

[devopvoid]

According to your example you provide equal stream configs. With that config result = apm->ProcessStream(srcFrame, srcConfig, dstConfig, dstFrame); is being executed. In that case nothing has changed in the native code.

Please check your input/output buffer sizes.

[devopvoid]

And v0.4.0 aec has no effect in some cases compared with c++ native webrtc (chrome master branch) using the same call.

This is a vague statement. I use this library und AEC works fine. As previously noted, you don't have to use the JNI version of AudioProcessing since the native code of WebRTC will configure the PeerConnection with AEC enabled. You can use AudioProcessing to tune or disable some processing features.

Without a minimal working example I can't reproduce your situation.

[nanguantong]

And v0.4.0 aec has no effect in some cases compared with c++ native webrtc (chrome master branch) using the same call.

This is a vague statement. I use this library und AEC works fine. As previously noted, you don't have to use the JNI version of AudioProcessing since the native code of WebRTC will configure the PeerConnection with AEC enabled. You can use AudioProcessing to tune or disable some processing features.

Without a minimal working example I can't reproduce your situation.

In my case，I need not use PeerConnection， just only use apm to process near and far frame in the server-side，so the two client sides buffer need sync and do apm smoothly.

[nanguantong]

Java_dev_onvoid_webrtc_media_audio_AudioProcessing_processReverseStream

result = apm->ProcessStream(dstFrame, srcConfig, dstConfig, dstFrame); ------> modify to result = apm->ProcessReverseStream(dstFrame, srcConfig, dstConfig, dstFrame);

[devopvoid]

You are absolutely right. I overlooked it.

[nanguantong]

Please publish packet to the repo？

[devopvoid]

It's published as 0.6.0-SNAPSHOT. For 0.6.0 it is too early.

[nanguantong]

It's published as 0.6.0-SNAPSHOT. For 0.6.0 it is too early.

0.6.0-SNAPSHOT repo is Incompleted in https://oss.sonatype.org/content/repositories/snapshots/dev/onvoid/webrtc/webrtc-java/0.6.0-SNAPSHOT/.

[nanguantong]

And v0.4.0 aec has no effect in some cases compared with c++ native webrtc (chrome master branch) using the same call.

I tested many cases, found that this version webrtc(M95/4638) had no effect in two clients AEC (server-side mode). Please feel free to update webrtc M95/4638 to the latest version.

        config.echoCanceller.enabled = true;
        config.echoCanceller.enforceHighPassFiltering = true;

        config.gainControl.enabled = true;

        config.highPassFilter.enabled = true;

        config.noiseSuppression.enabled = true;
        config.noiseSuppression.level = AudioProcessingConfig.NoiseSuppression.Level.HIGH;

        config.residualEchoDetector.enabled = true;
        config.transientSuppression.enabled = true;
        config.levelEstimation.enabled = true;
        config.voiceDetection.enabled = true;

        audioProcessing.applyConfig(config);

AudioProcessingStreamConfig far_frame_config = new AudioProcessingStreamConfig(8000, 1);
AudioProcessingStreamConfig near_frame_config = new AudioProcessingStreamConfig(8000, 1);

[nanguantong]

And v0.4.0 aec has no effect in some cases compared with c++ native webrtc (chrome master branch) using the same call.

I tested many cases, found that this version webrtc(M95/4638) had no effect in two clients AEC (server-side mode).

Please feel free to update webrtc M95/4638 to the latest version.

        config.echoCanceller.enabled = true;

        config.echoCanceller.enforceHighPassFiltering = true;

        config.gainControl.enabled = true;

        config.highPassFilter.enabled = true;

        config.noiseSuppression.enabled = true;

        config.noiseSuppression.level = AudioProcessingConfig.NoiseSuppression.Level.HIGH;

        config.residualEchoDetector.enabled = true;

        config.transientSuppression.enabled = true;

        config.levelEstimation.enabled = true;

        config.voiceDetection.enabled = true;

        audioProcessing.applyConfig(config);

AudioProcessingStreamConfig far_frame_config = new AudioProcessingStreamConfig(8000, 1);

AudioProcessingStreamConfig near_frame_config = new AudioProcessingStreamConfig(8000, 1);

Please feel free to publish the build packet to the repo？

[devopvoid]

A new 0.6.0-SNAPSHOT is up.

[nanguantong]

A new 0.6.0-SNAPSHOT is up.

It's incompleted.

Could not find webrtc-java-0.6.0-SNAPSHOT-linux-x86_64.jar (dev.onvoid.webrtc:webrtc-java:0.6.0-SNAPSHOT:20211130.151623-6).

[devopvoid]

It's all there

  <snapshotVersion>
    <classifier>linux-x86_64</classifier>
    <extension>jar</extension>
    <value>0.6.0-20211130.150026-4</value>
    <updated>20211130150026</updated>
  </snapshotVersion>

[nanguantong]

It's all there

  <snapshotVersion>
    <classifier>linux-x86_64</classifier>
    <extension>jar</extension>
    <value>0.6.0-20211130.150026-4</value>
    <updated>20211130150026</updated>
  </snapshotVersion>

I use gradle that update error in my project.

[devopvoid]

Try to use 0.6.0-20211130.150026-4 as version

[nanguantong]

The same result that had no effect in two clients AEC as following version: implementation "dev.onvoid.webrtc:webrtc-java:0.6.0-20211130.150550-5:windows-x86_64"

[devopvoid]

Do you have an analog native c/c++ version of what you are trying to do with Java?

Maybe discuss-webrtc can help you.

[devopvoid]

Your timings with getStreamDelayMs() and setStreamDelayMs() must be right, otherwise AEC will have no effect.

[nanguantong]

Do you have an analog native c/c++ version of what you are trying to do with Java?

Maybe discuss-webrtc can help you.

Yes, I use the analog webrtc native c/c++ version (master branch) that works ok, but use your webrtc-java no effect aec.

[nanguantong]

Your timings with getStreamDelayMs() and setStreamDelayMs() must be right, otherwise AEC will have no effect.

Yes, I use the same demo code comared webrtc-java with native c++ version.

[devopvoid]

Do you have the source code somewhere online of your native version?

[nanguantong]

bool WriteFrame(FILE* file, int16_t* data, size_t samples_per_channel) {
    size_t frame_size = samples_per_channel;
    size_t write_count = fwrite(data, sizeof(int16_t), frame_size, file);
    if (write_count != frame_size) {
        return false;  // This is expected.
    }
    return true;
}

int main(int argc, char **argv) {
    int ret = 0;
    int delay_ms = 0; // 200;

    // Usage example, omitting error checking:
    rtc::scoped_refptr<webrtc::AudioProcessing> apm = webrtc::AudioProcessingBuilder().Create();

    webrtc::AudioProcessing::Config config;
    config.echo_canceller.enabled = true;
    config.echo_canceller.mobile_mode = false;

    config.gain_controller1.enabled = true;
    config.gain_controller1.mode = webrtc::AudioProcessing::Config::GainController1::kAdaptiveAnalog;

    config.gain_controller2.enabled = true;
    config.high_pass_filter.enabled = true;
    config.voice_detection.enabled = true;

    config.noise_suppression.enabled = true;
    config.noise_suppression.level = webrtc::AudioProcessing::Config::NoiseSuppression::Level::kHigh;

    config.residual_echo_detector.enabled = true;
    config.transient_suppression.enabled = true;

    apm->ApplyConfig(config);

    ret = apm->Initialize();

    webrtc::AudioFrame far_frame;
    webrtc::AudioFrame near_frame;

    float frame_step = 10; // ms
    int near_read_bytes = 0;

    far_frame.sample_rate_hz_ = 8000;
    far_frame.samples_per_channel_ = far_frame.sample_rate_hz_ * frame_step / 1000.0;
    far_frame.num_channels_ = 1;
    near_frame.sample_rate_hz_ = 8000;
    near_frame.samples_per_channel_ = near_frame.sample_rate_hz_ * frame_step / 1000.0;
    near_frame.num_channels_ = 1;

    webrtc::StreamConfig input_config(8000, 1);
    webrtc::StreamConfig output_config(8000, 1);

    size_t size = near_frame.samples_per_channel_;

        argv[3] = (char*) "xxx/test/ref8k-1ch.pcm";
        argv[4] = (char*) "xxx/test/mic8k-1ch.pcm";
        argv[5] = (char*) "xxx/test/out8k-1ch.pcm";

    FILE *far_file = fopen(argv[3], "rb");
    FILE *near_file = fopen(argv[4], "rb");
    FILE *aec_out_file = fopen(argv[5], "wb");

    int16_t far_frame_dest[AudioFrame::kMaxDataSizeSamples] = {0};
    int16_t near_frame_dest[AudioFrame::kMaxDataSizeSamples] = { 0 };

    size_t read_count = 0;
    while (true) {
            read_count = fread((void*)far_frame.data(), sizeof(int16_t), size, far_file);
            if (read_count != size) {
              //fseek(near_file, read_count * sizeof(int16_t), SEEK_CUR);
            }
            ret = apm->ProcessReverseStream(far_frame.data(), input_config, output_config, far_frame_dest);

        read_count = fread((void*)near_frame.data(), sizeof(int16_t), size, near_file);
        near_read_bytes += read_count * sizeof(int16_t);

        if (read_count != size) {
            break;  // This is expected.
        }

        AudioProcessingStats stats = apm->GetStatistics();
        if (stats.delay_ms.has_value()) {
            std::cout << "!!!!delay_ms " << stats.delay_ms.value() << " voice detected " << stats.voice_detected.value() << std::endl;
        }

        //apm->set_stream_delay_ms(delay_ms);

        ret = apm->ProcessStream(near_frame.data(), input_config, output_config, near_frame_dest);

        WriteFrame(aec_out_file, near_frame_dest, size);
    }

    fclose(far_file);
    fclose(near_file);
    fclose(aec_out_file);

    std::cout << "over.";
    return 0;
}

[nanguantong]

[devopvoid]

Thanks, I will have a look at that and will report back.

[devopvoid]

I tested AEC with 0.6.0-SNAPSHOT and to me it works. I used the reference PCM samples from the WebRTC repository (far8_stereo.pcm and near8_stereo.pcm). You can see the result below.

With the code below.

import dev.onvoid.webrtc.media.audio.AudioProcessing;
import dev.onvoid.webrtc.media.audio.AudioProcessingConfig;
import dev.onvoid.webrtc.media.audio.AudioProcessingConfig.NoiseSuppression.Level;
import dev.onvoid.webrtc.media.audio.AudioProcessingStreamConfig;

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

public class Test {

    private static void testAEC() throws IOException {
        AudioProcessingConfig config = new AudioProcessingConfig();
        config.echoCanceller.enabled = true;
        config.echoCanceller.enforceHighPassFiltering = true;

        // Optionally enable gain control.
        config.gainControl.enabled = false;
        config.gainControl.adaptiveDigital.enabled = true;

        config.highPassFilter.enabled = true;

        config.noiseSuppression.enabled = true;
        config.noiseSuppression.level = Level.MODERATE;

        AudioProcessing audioProcessing = new AudioProcessing();
        audioProcessing.applyConfig(config);

        AudioProcessingStreamConfig streamConfig = new AudioProcessingStreamConfig(8000, 2);

        InputStream farStream = Test.class.getResourceAsStream("/far8_stereo.pcm");
        InputStream nearStream = Test.class.getResourceAsStream("/near8_stereo.pcm");

        OutputStream outputStream = new FileOutputStream("output.pcm");

        int inputBufferSize = streamConfig.sampleRate / 100 * streamConfig.channels * 2;
        int outputBufferSize = audioProcessing.getTargetBufferSize(streamConfig, streamConfig);

        byte[] inputBuffer = new byte[inputBufferSize];
        byte[] outputBuffer = new byte[outputBufferSize];

        while (farStream.read(inputBuffer) > 0) {
            int result = audioProcessing.processReverseStream(inputBuffer,
                    streamConfig, streamConfig, inputBuffer);

            if (result != 0) {
                throw new IllegalStateException();
            }

            nearStream.read(inputBuffer);

            result = audioProcessing.processStream(inputBuffer, streamConfig,
                    streamConfig, outputBuffer);

            if (result != 0) {
                throw new IllegalStateException();
            }

            outputStream.write(outputBuffer);
        }

        outputStream.flush();
        outputStream.close();
    }

    public static void main(String[] args) throws Exception {
        testAEC();
    }
}

[nanguantong]

Yeh，I also tested some samples that work，but some samples that not work used same code. I'll try to sync strictly the near and far samples time in some cases, and look at the result.

Can you please feel free to update webrtc to the lastest version (e.g.` master or branch 47xx) ?

[devopvoid]

I update the branches according to this schedule.

I guess the issue is, you have an offset in your example between far and near. For this apm->set_stream_delay_ms must be set. In this case its a static delay which can be calculated.

[nanguantong]

I guess the issue is, you have an offset in your example between far and near. For this apm->set_stream_delay_ms must be set. In this case its a static delay which can be calculated.

apm->set_stream_delay_ms has been set, native and webrtc-java had the different result output, I guess the APM module has modified between the two versions.

[devopvoid]

Closing as it is working as expected.