search

awslabs / amazon-kinesis-video-streams-producer-c

https://awslabs.github.io/amazon-kinesis-video-streams-producer-c/group__PublicMemberFunctions.html
Apache License 2.0
54 stars 72 forks source link

Create H264 Video + u-law Audio sample application #121

Closed dwaynehubbard closed 3 years ago

dwaynehubbard commented 3 years ago

My ultimate goal for this pull request is to use AWS Kinesis Videostreams for non-video, multi-track (stereo), PCM (u-law) audio streaming.

This pull request contributes a sample application - KvsMuLawAudioVideoStreamingSample.c - the mirrors an existing application with the exception of using u-law instead of AAC for audio. Sample audio files are provided in the samples/pcmMuLawTrack1SampleFrames directory. This pull request also modifies StreamInfoProvider (.c and .h), CMakeLists.txt, and requires some amazon-kinesis-video-streams-pic submitted in https://github.com/awslabs/amazon-kinesis-video-streams-pic/issues/86

It is unclear to me if the current amazon-kinesis-videostreams solutions requires a video track (that must be track 1) and/or if AWS KVS can currently be used for multi-track u-law audio. This pull request, when built with https://github.com/awslabs/amazon-kinesis-video-streams-pic/pull/87 and executed with:

KVS_DEBUG_DUMP_DATA_FILE_DIR=./

will generate a local .mkv file that when inspected by ffmpeg will produce the output below. I can extract the u-law audio from the mkv file and play it in audacity, but cannot play the audio using VLC, MediaPlayer, MPV, etc ... as they don't appear to recognize the audio codec. I'm not sure if I'm doing anything wrong in the creation of the .mkv or if u-law + H264 is generally not a realistic combination.

My next steps towards the ultimate goal would be to 1) try to add a second u-law audio track (as track 3), 2) remove the video track (track 1) leaving only two audio tracks.

ffmpeg version N-98199-g318a1a3 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.12) 20160609 configuration: libavutil 56. 55.100 / 56. 55.100 libavcodec 58. 92.100 / 58. 92.100 libavformat 58. 46.101 / 58. 46.101 libavdevice 58. 11.100 / 58. 11.100 libavfilter 7. 86.100 / 7. 86.100 libswscale 5. 8.100 / 5. 8.100 libswresample 3. 8.100 / 3. 8.100 Guessed Channel Layout for Input Stream #0.1 : stereo Input #0, matroska,webm, from 'UII-Videostream-Member-11_0.mkv': Metadata: title : Kinesis Video SDK encoder : Kinesis Video SDK 1.1.0 Duration: N/A, start: 0.000000, bitrate: N/A Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt470bg/smpte170m/bt709, progressive), 640x480 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1k tbn, 50 tbc (default) Metadata: title : kvs_video_track Stream #0:1(eng): Audio: pcm_s16le, 8000 Hz, stereo, s16, 256 kb/s (default) Metadata: title : kvs_audio_track_2

MushMal commented 3 years ago

It's hard to tell why vlc player is choking - I am not very familiar with how to debug it.

In general, if you are attempting to replace the AAC samples with mulaw samples then you can still use the existing sample application but just do dynamic run-time parameter selection I guess. We are trying to not proliferate a lot of samples in this repository. The main demo/samples repository is here: https://github.com/aws-samples/amazon-kinesis-video-streams-demos

dwaynehubbard commented 3 years ago

@MushMal got it. I can work this into the existing demo and update the branch. Also, my next two steps in addition to your feedback are 1) try to add a second audio track, 2) remove the video track. Do you know of any limitations that would prevent these actions?

dwaynehubbard commented 3 years ago

After adding functionality for: 1) try to add a second audio track; there definitely seems to be a limitation somewhere in the producer code:

Input #0, matroska,webm, from 'UII-Videostream-Member-11_0.mkv': Metadata: title : Kinesis Video SDK encoder : Kinesis Video SDK 1.1.0 Duration: N/A, start: 0.000000, bitrate: N/A Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt470bg/smpte170m/bt709, progressive), 640x480 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1k tbn, 50 tbc (default) Metadata: title : kvs_video_track Stream #0:1(eng): Audio: pcm_s16le, 8000 Hz, stereo, s16, 256 kb/s (default) Metadata: title : kvs_audio_track_2 Stream #0:2(eng): Audio: pcm_s16le, 0 channels (default) Metadata: title : kvs_audio_track_3

hassanctech commented 3 years ago

@dwaynehubbard Is there still an issue here? We definitely support two audio tracks so there shouldn't be an issue there. If you're hitting a specific SDK issue can you please provide logs indicating a potential SDK bug you are hitting?

sharadrj commented 3 years ago

@dwaynehubbard, to be clear on limitations with multi-track streams:

dwaynehubbard commented 3 years ago

Thank you guys, I will investigate on my side and update this ticket ASAP. I appreciate your time and responses.

MushMal commented 3 years ago

Any update on this issue? Is there any specifics that you would like to demo in the sample? From what I see, there is nothing new but rather a few changes in the StreamInfo. These can be rather documented than a new sample produced. We are trying to strike a balance between the ease of getting started and feature demonstration. I am not sure whether this cuts it.

MushMal commented 3 years ago

Resolving