[GST Plugin] Audio is garbled with WebRTC

[adamthesax]

I'm working on using the gstreamer + kvsplugin as a way to ingest some video streams into KVS for real-time monitoring. The gstreamer pipeline runs successfully, but if I preview the audio in the AWS web console or via the browser example, the audio is coming out scrambled/garbled.

This can be reproduced with this simple test-generator pipeline:

gst-launch-1.0 \
    videotestsrc is-live=TRUE ! queue ! video/x-raw,width=1280,height=720,framerate=30/1 ! \
    x264enc bframes=0 speed-preset=veryfast bitrate=512 byte-stream=TRUE tune=zerolatency ! \
    video/x-h264,stream-format=avc,alignment=au,profile=baseline ! \
    kvsplugin name=kvs aws-region=us-east-1 channel-name=audio-issue enable-streaming=false connect-webrtc=true log-level=3 \
    audiotestsrc is-live=TRUE ! queue ! audioconvert ! audioresample ! faac ! \
    audio/mpeg,rate=48000,channels=2 ! kvs.

I think the underlying cause is codec incompatibility between the kvsplugin sinks and the underlying KVS WebRTC service. No matter which format that is given to kvsplugin (audio/mpeg, audio/x-mulaw, or audio/x-alaw), the plugin always reports it's codec as audio/opus. So the end result is the browser is decoding AAC encoded audio using the opus decoder.

It feels if the GST plugin is meant to support both KVS Video Streams and WebRTC, then it should handle some codec conforming, so that a single audio/video input can be routed to both a Video Stream or Signaling channel. If kvsplugin is meant to support both WebRTC and VideoStreams but not in the same pipeline then it may make sense to add an audio sink that accepts audio/x-opus. What's the best path forward here?

[hassanctech]

Thank you for bringing this to our attention. It appears we have hardcoded opus here, we are working on the change properly declare support for opus/mulaw/alaw. We do not support AAC in WebRTC. We'll use the gstreamer parameter that's passed in to select which of opus/mulaw/alaw to send. You can find our supported audio/video codecs here:

https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/tree/master/src/source/Rtp/Codecs

[disa6302]

@adamthesax ,

I have PR out to fix the issue? Would you like to give it a try?

[disa6302]

@adamthesax ,

the changes are merged. Feel free to reach out if you face any issues.

[adamthesax]

Thanks for putting in a fix and sorry for the delay in getting back to you. I'm still having a tough time getting audio working cleanly. With your latest changes, I can send mulaw or alaw audio, which KVS recognizes as such (audio/PCMU), but playback is still distorted (my test tone sounds like it has shifted down in frequency). Maybe it is something with my pipeline, but it's pretty straight forward. Here's the updated pipeline which is sending mulaw:

gst-launch-1.0 \
    videotestsrc is-live=TRUE pattern=ball ! queue ! video/x-raw,width=1280,height=720,framerate=30/1 ! \
    x264enc bframes=0 speed-preset=veryfast bitrate=512 byte-stream=TRUE tune=zerolatency ! \
    video/x-h264,stream-format=avc,alignment=au,profile=baseline ! \
    kvsplugin name=kvs aws-region=us-east-1 channel-name=audio-issue enable-streaming=false connect-webrtc=true log-level=3 \
    audiotestsrc wave=8 sine-periods-per-tick=40 marker-tick-period=3 is-live=TRUE ! queue ! audioconvert ! audioresample ! mulawenc ! \
    audio/x-mulaw,rate=48000,channels=2 ! kvs.

Also there isn't a declared sink on kvsplugin for audio/x-opus so even though internally the plugin is detecting if we are sending opus, alaw or mulaw, there is no way to pipe opus audio into the plugin.

Thanks again for the help!

[disa6302]

@adamthesax ,

I tested with the pipeline you provided previously, juts replacing with mulaw/alaw and the allow sample rate (8000 or 12000) Yes, audio/x-opus was removed in previous PR since WebRTC supports opus but not producer. Will need to look into it.

In the meantime, can you try this pipeline? Your audio sample rate is 48000 which mulawenc or alawenc does not support. Check this: https://gstreamer.freedesktop.org/documentation/mulaw/mulawenc.html?gi-language=c

gst-launch-1.0 videotestsrc is-live=TRUE ! queue ! video/x-raw,width=1280,height=720,framerate=30/1 ! x264enc bframes=0 speed-preset=veryfast bitrate=512 byte-stream=TRUE tune=zerolatency ! video/x-h264,stream-format=avc,alignment=au,profile=baseline ! kvsplugin name=kvs channel-name=audio-issue enable-streaming=false connect-webrtc=true log-level=3 audiotestsrc is-live=TRUE ! queue ! audioconvert ! audioresample ! mulawenc ! audio/x-mulaw,rate=8000,channels=2 ! kvs.

[adamthesax]

@disa6302,

Thanks for getting back to me! Moving to 8000 sample rate does fix this problem, but the audio quality isn't great. From my understanding of the link you sent, mulawenc supports a range from 8000 to 192000. Consider this pipeline:

gst-launch-1.0 audiotestsrc is-live=TRUE ! queue ! audioconvert ! audioresample ! mulawenc ! audio/x-mulaw,rate=48000 ! decodebin ! autoaudiosink

mulawenc is happy to encode at various sample rates between 8kHz and 192kHz, but kvsplugin is expecting it to be at 8kHz. I looks like 8 kHz is baked into the webrtc-sdk but it's a bit unclear from the WebRTC spec if 8kHz is a hard requirements for PCM audio.

From WebRTC Audio Codec and Processing Requirements

For all cases where the endpoint is able to process audio at a sampling rate higher than 8 kHz, it is RECOMMENDED that Opus be offered before PCMA/PCMU.

I recognize the long term solution here may be to move to Opus for better quality at a lower bitrate (recognizing that there's still work to integrate opus support into kvsplugin), but in the short term, do you know if the 8kHz is an KVS/SDK limitation or an actual limitation of the WebRTC spec?

[disa6302]

@adamthesax ,

I do notice that the audio is not clear at a sampling rate higher than 8KHz. And yes, we hardcode the audio sample rate in the WebRTC C SDK. And I am assuming the limit was set based on the recommendation. It would be best to proceed with Opus for now. I have a PR sitting to include Opus. Can you try it out?

[adamthesax]

@disa6302 Yes happy to test out Opus support, just point me at the PR

[disa6302]

https://github.com/aws-samples/amazon-kinesis-video-streams-demos/pull/162

[adamthesax]

Alright I checked out your branch and built it. I think the big problem right now in terms of Opus support is there is not declared sink for audio/x-opus so GST can't build a pipeline from opusenc to kvsplugin. At a minimum I think we need to declare that kvsplugin accepts audio/x-opus here but when I add that, I get:

2021-03-29 19:59:13 ERROR   gst_kvs_plugin_change_state(): Failed to initialize track with 0x00000002
ERROR: from element /GstPipeline:pipeline0/GstKvsPlugin:kvs: Could not initialize supporting library.
Additional debug info:
/Users/211351/code/thirdparty/amazon-kinesis-video-streams-demos/gst/gst-kvs-plugin/src/GstPlugin.c(1296): gst_kvs_plugin_change_state (): /GstPipeline:pipeline0/GstKvsPlugin:kvs:
Failed to initialize with 0x00000002

I thus I think we also need to support opus in initTrackData()

[disa6302]

I think that is where the problem is. The code needs some refactoring. The issue is, the KVS producer does not support OPUS and the initTrackData uses producer to initialize. I am going to make some changes to outsource this to the plugin and then reference in WebRTC. Stay tuned...

[disa6302]

@adamthesax ,

Can you test the PR? I tested with a-law, mu-law and opus and it works. Would like to get it verified from you as well.

[adamthesax]

🙌 Yes! Opus is working for me. Ran into one little missing ;:

GstPlugin.c:1372:59: error: expected ';' after expression

After I patched that up opus is working and sounding great!

[disa6302]

@adamthesax ,

PR is merged. Feel free to open a new issue if you have any questions.