search

shaka-project / shaka-packager

A media packaging and development framework for VOD and Live DASH and HLS applications, supporting Common Encryption for Widevine and other DRM Systems.
https://shaka-project.github.io/shaka-packager/
Other
1.98k stars 507 forks source link

Assigning audio group in multibitrate HLS #899

Open vasudevavarun opened 3 years ago

vasudevavarun commented 3 years ago

System info

Operating System: Centos 7.4(64bit) Shaka Packager Version: 2.4.3

Issue and steps to reproduce the problem

Packager Command:

./packager-linux-2.4.3 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_2000.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/2000-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/2000.m3u8' 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_700.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/700-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/700.m3u8' 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_250.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/250-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/250.m3u8' 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_2000.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/128-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/128.m3u8,hls_group_id=audio' 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_250.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/32-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/32.m3u8,hls_group_id=audio' 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_700.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/64-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/64.m3u8,hls_group_id=audio' 'in=/tmp/Subtitle\ 1.vtt,stream=text,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/$Number$.vtt,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/main.m3u8,hls_group_id=text,hls_name=ENGLISH' --hls_master_playlist_output /opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/a.m3u8

What is the expected result?

EXTM3U

Generated with https://github.com/google/shaka-packager version v2.4.3-dd98700-release

EXT-X-MEDIA:TYPE=AUDIO,URI="128.m3u8",GROUP-ID="audio",LANGUAGE="en",NAME="stream_1",AUTOSELECT=YES,CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="64.m3u8",GROUP-ID="audio",LANGUAGE="en",NAME="stream_5",CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="32.m3u8",GROUP-ID="audio",LANGUAGE="en",NAME="stream_3",CHANNELS="2"

EXT-X-MEDIA:TYPE=SUBTITLES,URI="main.m3u8",GROUP-ID="text",NAME="ENGLISH",AUTOSELECT=YES

EXT-X-STREAM-INF:BANDWIDTH=2264710,AVERAGE-BANDWIDTH=2189543,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=768x576,AUDIO="audio_128",SUBTITLES="text"

2000.m3u8

EXT-X-STREAM-INF:BANDWIDTH=893814,AVERAGE-BANDWIDTH=865367,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=640x480,AUDIO="audio_64",SUBTITLES="text"

700.m3u8

EXT-X-STREAM-INF:BANDWIDTH=421558,AVERAGE-BANDWIDTH=408952,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=384x288,AUDIO="audio_32",SUBTITLES="text"

250.m3u8

What happens instead?

EXTM3U

Generated with https://github.com/google/shaka-packager version v2.4.3-dd98700-release

EXT-X-MEDIA:TYPE=AUDIO,URI="128.m3u8",GROUP-ID="audio",LANGUAGE="en",NAME="stream_1",AUTOSELECT=YES,CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="64.m3u8",GROUP-ID="audio",LANGUAGE="en",NAME="stream_5",CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="32.m3u8",GROUP-ID="audio",LANGUAGE="en",NAME="stream_3",CHANNELS="2"

EXT-X-MEDIA:TYPE=SUBTITLES,URI="main.m3u8",GROUP-ID="text",NAME="ENGLISH",AUTOSELECT=YES

EXT-X-STREAM-INF:BANDWIDTH=2264710,AVERAGE-BANDWIDTH=2189543,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=768x576,AUDIO="audio",SUBTITLES="text"

2000.m3u8

EXT-X-STREAM-INF:BANDWIDTH=893814,AVERAGE-BANDWIDTH=865367,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=640x480,AUDIO="audio",SUBTITLES="text"

700.m3u8

EXT-X-STREAM-INF:BANDWIDTH=421558,AVERAGE-BANDWIDTH=408952,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=384x288,AUDIO="audio",SUBTITLES="text"

250.m3u8

I want to associate audio group with desired bitrate. Thanks

kqyang commented 3 years ago

@vasudevavarun You can specify the audio group using hls_group_id stream descriptor: https://google.github.io/shaka-packager/html/tutorials/hls.html#hls-specific-stream-descriptor-fields, i.e.

./packager-linux-2.4.3  \
  'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_2000.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/2000-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/2000.m3u8'  \
  'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_700.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/700-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/700.m3u8'  \ 
  'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_250.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/250-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/250.m3u8'  \ 
  'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_2000.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/128-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/128.m3u8,hls_group_id=audio_128'  \ 
  'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_250.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/32-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/32.m3u8,hls_group_id=audio_32'  \ 
  'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_700.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/64-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/64.m3u8,hls_group_id=audio_64'  \
  'in=/tmp/Subtitle\ 1.vtt,stream=text,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/$Number$.vtt,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/main.m3u8,hls_group_id=text,hls_name=ENGLISH'  \
  --hls_master_playlist_output /opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/a.m3u8

I am closing this issue. Feel free to reply if you have more questions.

vasudevavarun commented 3 years ago

Hi,

Command:

./packager-linux-2.4.3 \ 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_2000.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/2000-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/2000.m3u8,hls_group_id=audio_128' \ 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_700.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/700-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/700.m3u8,hls_group_id=audio_64' \ 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_250.mp4,stream=video,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/250-$Number$.ts,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/250.m3u8,hls_group_id=audio_32' \ 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_2000.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/128-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/128.m3u8,hls_group_id=audio_128' \ 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_250.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/32-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/32.m3u8,hls_group_id=audio_32' \ 'in=/opt/video/data/amritatv_ott615/bulk_video/dest/mp4/a_2_700.mp4,stream=audio,segment_template=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/64-$Number$.aac,playlist_name=/opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/64.m3u8,hls_group_id=audio_64' \ --hls_master_playlist_output /opt/video/data/amritatv_ott615/bulk_video/dest/hls/a_2/a.m3u8

Expected Behavior:

EXTM3U

Generated with https://github.com/google/shaka-packager version v2.4.3-dd98700-release

EXT-X-MEDIA:TYPE=AUDIO,URI="128.m3u8",GROUP-ID="audio_128",LANGUAGE="en",NAME="stream_0",AUTOSELECT=YES,CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="32.m3u8",GROUP-ID="audio_32",LANGUAGE="en",NAME="stream_2",AUTOSELECT=YES,CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="64.m3u8",GROUP-ID="audio_64",LANGUAGE="en",NAME="stream_4",AUTOSELECT=YES,CHANNELS="2"

EXT-X-STREAM-INF:BANDWIDTH=421558,AVERAGE-BANDWIDTH=408952,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=384x288,AUDIO="audio_32"

250.m3u8

EXT-X-STREAM-INF:BANDWIDTH=893814,AVERAGE-BANDWIDTH=865367,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=640x480,AUDIO="audio_64"

700.m3u8

EXT-X-STREAM-INF:BANDWIDTH=2264710,AVERAGE-BANDWIDTH=2189543,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=768x576,AUDIO="audio_128"

2000.m3u8

Result:

EXTM3U

Generated with https://github.com/google/shaka-packager version v2.4.3-dd98700-release

EXT-X-MEDIA:TYPE=AUDIO,URI="128.m3u8",GROUP-ID="audio_128",LANGUAGE="en",NAME="stream_0",AUTOSELECT=YES,CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="32.m3u8",GROUP-ID="audio_32",LANGUAGE="en",NAME="stream_2",AUTOSELECT=YES,CHANNELS="2"

EXT-X-MEDIA:TYPE=AUDIO,URI="64.m3u8",GROUP-ID="audio_64",LANGUAGE="en",NAME="stream_4",AUTOSELECT=YES,CHANNELS="2"

EXT-X-STREAM-INF:BANDWIDTH=421558,AVERAGE-BANDWIDTH=408952,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=384x288,AUDIO="audio_128"

250.m3u8

EXT-X-STREAM-INF:BANDWIDTH=893814,AVERAGE-BANDWIDTH=865367,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=640x480,AUDIO="audio_128"

700.m3u8

EXT-X-STREAM-INF:BANDWIDTH=2264710,AVERAGE-BANDWIDTH=2189543,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=768x576,AUDIO="audio_128"

2000.m3u8

EXT-X-STREAM-INF:BANDWIDTH=324107,AVERAGE-BANDWIDTH=312862,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=384x288,AUDIO="audio_32"

250.m3u8

EXT-X-STREAM-INF:BANDWIDTH=796363,AVERAGE-BANDWIDTH=769277,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=640x480,AUDIO="audio_32"

700.m3u8

EXT-X-STREAM-INF:BANDWIDTH=2167259,AVERAGE-BANDWIDTH=2093453,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=768x576,AUDIO="audio_32"

2000.m3u8

EXT-X-STREAM-INF:BANDWIDTH=356779,AVERAGE-BANDWIDTH=344903,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=384x288,AUDIO="audio_64"

250.m3u8

EXT-X-STREAM-INF:BANDWIDTH=829035,AVERAGE-BANDWIDTH=801318,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=640x480,AUDIO="audio_64"

700.m3u8

EXT-X-STREAM-INF:BANDWIDTH=2199931,AVERAGE-BANDWIDTH=2125494,CODECS="avc1.4d4029,mp4a.40.2",RESOLUTION=768x576,AUDIO="audio_64"

2000.m3u8

The audio should be mapped once to the desired video bitrate.

kqyang commented 3 years ago

I see. Shaka packager does not support fixing the audio group for the variant streams right now.

It is possible to add support for it with some simple modifications. Let us know if you are interested in helping us implement it.

vasudevavarun commented 3 years ago

I see. Shaka packager does not support fixing the audio group for the variant streams right now.

It is possible to add support for it with some simple modifications. Let us know if you are interested in helping us implement it.

Sure, Please let me know how can I help.

kqyang commented 3 years ago

It is trivial if we just want to support fixing the audio group id for the variant stream. Here is a sample change to do that:

https://github.com/google/shaka-packager/blob/master/packager/hls/base/master_playlist.cc#L435

  for (const auto& variant : variants) {
    if (video_playlists.empty())
      break;
    content->append("\n");
    for (const auto& playlist : video_playlists) {
      if (!playlist.group_id().empty() && variant.audio_group_id && 
          playlist.group_id() != *variant.audio_group_id) {
        continue;
      }
      BuildStreamInfTag(*playlist, variant, base_url, content);
    }
  }

The above change supports having one audio group id per variant stream only. We may want to support more than one audio group ids per variant stream.

We should also support text group ids. Ideally there should be separate stream descriptors for audio and text, but that requires more changes.

One possible option is to reuse the same stream descriptor hls_group_id, but modify it to support both audio and text, and optionally more than one audios and texts, e.g. use ; to separate the groups and use + to separate audio and text, something like: hls_group_id=audio1;audio2+text1;text2. You may use base::SplitString to split the strings.

shaka-bot commented 3 years ago

@vasudevavarun Does this answer all your questions? Can we close the issue?

kqyang commented 3 years ago

@vasudevavarun Please ignore the shaka-bot message above. The issue was not correctly labeled which confused the bot.

vasudevavarun commented 3 years ago

Hi @kqyang, I've applied the above patch but receiving the following error on ninja build

ninja: Entering directoryout/Release' [2/297] CXX obj/hls/base/hls_builder.master_playlist.o FAILED: obj/hls/base/hls_builder.master_playlist.o ../../packager/third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/hls/base/hls_builder.master_playlist.o.d -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DCHROMIUM_BUILD -DCR_CLANG_REVISION=317263-2 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DENABLE_PEPPER_CDMS -DENABLE_NOTIFICATIONS -DUSE_UDEV -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DGFLAGS_DLL_DECL= -DGFLAGS_DLL_DECLARE_FLAG= -DGFLAGS_DLL_DEFINE_FLAG= -DUSE_LIBPCI=1 -DUSE_NSS_CERTS=1 -DSTDC_CONSTANT_MACROS -DSTDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -Igen -I../../packager -I../.. -Igen/protoc_out -I../../packager/third_party/protobuf/src -I../../packager/third_party/gflags/gen/posix/include -I../../packager/third_party/gflags/src/src -fstack-protector --param=ssp-buffer-size=4 -Werror -pthread -fno-strict-aliasing -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -fcolor-diagnostics -Wheader-hygiene -Wstring-conversion -Wno-deprecated-register -Wno-inconsistent-missing-override -Wno-shift-negative-value -Wno-undefined-var-template -Wno-nonportable-include-path -Wimplicit-fallthrough -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -c ../../packager/hls/base/master_playlist.cc -o obj/hls/base/hls_builder.master_playlist.o ../../packager/hls/base/master_playlist.cc:440:15: error: member reference type 'const shaka::hls::MediaPlaylist *const' is a pointer; did you mean to use '->'? if (!playlist.group_id().empty() && variant.audio_group_id && 


                     ->
../../packager/hls/base/master_playlist.cc:441:19: error: member reference type 'const shaka::hls::MediaPlaylist *const' is a pointer; did you mean to use '->'?
          playlist.group_id() != *variant.audio_group_id) {
          ~~~~~~~~^
                  ->
2 errors generated.
[3/297] LINK mpd_unittest
ninja: build stopped: subcommand failed.
`
kqyang commented 3 years ago

Sorry for the typo. Yes, it should be ->.

vasudevavarun commented 3 years ago

Hey, Replacing with '->' solved the compilation issue, By assigning the hls_group_id to different bitrate is working. I've got the following output m3u8.

image

On the other hand, This modification is generating error when I'm parsing vtt subtitles in the command which was working before, I'm having the following error when parsing subttitles.

terminate called after throwing an instance of 'std::regex_error' what(): regex_error Aborted (core dumped)

kqyang commented 3 years ago

On the other hand, This modification is generating error when I'm parsing vtt subtitles in the command which was working before, I'm having the following error when parsing subttitles.

Can you double check that it works before the change? It seems like a different issue.

vasudevavarun commented 3 years ago

Hi @kqyang

I realized that the command with vtt subtitles are working with per-compiled binaries that you guys have provided but it is throwing this exception only on my compiled version, That is why it is only giving error there.

I came across this ticket while searching for it and it is exactly what I'm facing. https://github.com/google/shaka-packager/issues/865

kqyang commented 3 years ago

@vasudevavarun What is your platform and gcc/clang version?

vasudevavarun commented 3 years ago

OS: CentOS Linux release 7.8.2003 GCC: gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)

Although, I've also tried with gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) using "scl enable devtoolset-8" But it is still same.

kqyang commented 3 years ago

Can you comment on #865 with the information above and also the VTT file you use? It can help us re-produce the issue. Thanks.