search

bentasker / HLS-Stream-Creator

Simple Bash Script to take a media file, segment it and create an M3U8 playlist for serving using HLS
BSD 3-Clause "New" or "Revised" License
273 stars 101 forks source link

encrypted mp4 not playing in vlc or safari #34

Closed hardevine closed 4 years ago

hardevine commented 4 years ago

created encrypted parts using

hls-stream-creater.sh -i ./small.mp4 -e -s 10 ./

it created successfully but now its not playing.

m3u8 file contents:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:YES
#EXT-X-KEY:METHOD=AES-128,URI=small.mp4.key
#EXT-X-TARGETDURATION:6
#EXTINF:5.600000,
small.mp4_00000.ts
#EXT-X-ENDLIST
bentasker commented 4 years ago

Odd.

Did it create the key (i.e. does small.mp4.key exist)?

What do you get if you run ffprobe -i small.mp4_00000.ts?

Could you post your manifest, key and segment somewhere so I can have a closer look? I'm wondering if perhaps the encryption didn't happen for some reason (or maybe the IV's wrong etc)

hardevine commented 4 years ago

yes there is the key file on running ffprobe this is the output:

$ ffprobe -i ./small.mp4_00000.ts
ffprobe version git-2019-11-14-73ee53f Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 9.2.1 (GCC) 20191010
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 35.101 / 56. 35.101
  libavcodec     58. 62.100 / 58. 62.100
  libavformat    58. 35.100 / 58. 35.100
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 66.100 /  7. 66.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
./small.mp4_00000.ts: Invalid data found when processing input

check https://we.tl/t-ddElWGfPA0

bentasker commented 4 years ago

Thanks

Key-length is correct

ben@milleniumfalcon:~/tmp/HLS$ wc -c small.mp4.key 
16 small.mp4.key

Segment decrypts happily enough using the key

ben@milleniumfalcon:~/tmp/HLS$ cp small.mp4_00000.ts test.ts
ben@milleniumfalcon:~/tmp/HLS$ IV=$(printf '%032x' 00000)
ben@milleniumfalcon:~/tmp/HLS$ KEY=$(cat small.mp4.key | hexdump -e '16/1 "%02x"')
ben@milleniumfalcon:~/tmp/HLS$ openssl aes-128-cbc -d -in test.ts -out test.dc.ts -K $KEY -iv $IV
ben@milleniumfalcon:~/tmp/HLS$ ffprobe -hide_banner -i test.dc.ts
Input #0, mpegts, from 'test.dc.ts':
Duration: 00:00:05.53, start: 1.466667, bitrate: 350 kb/s
Program 1 
    Metadata:
    service_name    : Service01
    service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 560x320, 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, mono, fltp, 72 kb/s

and actually, VLC playback works fine for me:

ben@milleniumfalcon:~/tmp/HLS$ cvlc small.mp4.m3u8 
VLC media player 2.2.2 Weatherwax (revision 2.2.2-0-g6259d80)
[0000000000fc11d8] dummy interface: using the dummy interface module...
[00007fb888c01518] httplive stream: HTTP Live Streaming (/home/ben/tmp/HLS/small.mp4.m3u8)
[00007fb888c01fc8] ts demux: MPEG-4 descriptor not found for pid 0x101 type 0xf
[00007fb888ddbb78] packetizer_mpeg4audio packetizer: AAC channels: 1 samplerate: 48000
[00007fb888c84718] avcodec decoder: Using NVIDIA VDPAU Driver Shared Library  340.104  Thu Sep 14 16:45:03 PDT 2017 for hardware decoding.

Could you try playing using cvlc so that errors get logged to console and provide the output? I don't think it will, but does anything hit the Safari Javascript console when you attempt playback?

hardevine commented 4 years ago

Safari gives "Failed to load resource: Plug-in handled load" How to play using cvlc, im on Macosx ?

hardevine commented 4 years ago

this is what is get from vlc 

main debug: VLC media player - 3.0.8 Vetinari
main debug: Copyright © 1996-2019 the VideoLAN team
main debug: revision 3.0.8-0-gf350b6b5a7
main debug: configured with /Users/d-fu/vlc-3.0/extras/package/macosx/../../../configure  '--prefix=/Users/d-fu/vlc-3.0/build/vlc_install_dir' '--enable-macosx' '--enable-merge-ffmpeg' '--enable-osx-notifications' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-shout' '--enable-ncurses' '--enable-twolame' '--enable-realrtsp' '--enable-libass' '--enable-macosx-qtkit' '--enable-macosx-avfoundation' '--disable-skins2' '--disable-xcb' '--disable-caca' '--disable-pulse' '--disable-sdl-image' '--disable-vnc' '--build=x86_64-apple-darwin17' '--host=x86_64-apple-darwin17' '--with-macosx-version-min=10.7' '--with-macosx-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk' '--with-breakpad=https://mac.crashes.videolan.org' 'build_alias=x86_64-apple-darwin17' 'host_alias=x86_64-apple-darwin17' 'CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang' 'CFLAGS=-g' 'LDFLAGS=' 'CXX=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++' 'CXXFLAGS=-g' 'OBJC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang' 'OBJCFLAGS=-g'
main debug: Creating an input for preparsing 'small.mp4.m3u8'
main debug: processing request item: small.mp4.m3u8, node: Playlist, skip: 0
main debug: rebuilding array of current - root Playlist
main debug: rebuild done - 1 items, index 0
main debug: starting playback of new item
main debug: resyncing on small.mp4.m3u8
main debug: small.mp4.m3u8 is at 0
main debug: creating new input thread
macosx debug: Enabling media key support
macosx warning: Failed to enable media key support, likely app needs to be whitelisted in Security Settings.
main debug: looking for meta fetcher module matching "any": 1 candidates
main debug: Creating an input for 'small.mp4.m3u8'
lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/fetcher
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/fetcher
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/fetcher
main debug: no meta fetcher modules matched
main debug: looking for art finder module matching "any": 2 candidates
lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/art
main debug: requesting art for new input thread
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/art
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/art
main debug: using timeshift granularity of 50 MiB
main debug: using default timeshift path
main debug: `file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8' gives access `file' demux `any' path `/Users/Admin/Desktop/vid_hls/small.mp4.m3u8'
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac
main debug: creating demux: access='file' demux='any' location='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' file='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8'
main debug: looking for access_demux module matching "file": 14 candidates
main debug: no access_demux modules matched
main debug: creating access: file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8
main debug:  (path: /Users/Admin/Desktop/vid_hls/small.mp4.m3u8)
main debug: looking for access module matching "file": 23 candidates
main debug: using access module "filesystem"
main debug: looking for stream_filter module matching "prefetch,cache_read": 26 candidates
cache_read debug: Using stream method for AStream*
cache_read debug: starting pre-buffering
cache_read debug: received first data after 0 ms
cache_read debug: pre-buffering done 192 bytes in 0s - 2314 KiB/s
main debug: using stream_filter module "cache_read"
main debug: looking for stream_filter module matching "any": 26 candidates
lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/playlist
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/playlist
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/playlist
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/anevia_streams.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/anevia_xml.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/appletrailers.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/bbc_co_uk.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/cue.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/dailymotion.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac
main debug: no art finder modules matched
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/jamendo.luac
main debug: looking for meta fetcher module matching "any": 1 candidates
lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/fetcher
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/fetcher
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/fetcher
main debug: no meta fetcher modules matched
main debug: looking for art finder module matching "any": 2 candidates
lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/art
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/art
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/art
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/koreus.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/liveleak.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/newgrounds.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/rockbox_fm_presets.luac
lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac
main debug: no art finder modules matched
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/soundcloud.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/twitch.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/vimeo.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/vocaroo.luac
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/youtube.luac
main debug: no stream_filter modules matched
main debug: looking for stream_directory module matching "any": 1 candidates
main debug: no stream_directory modules matched
main debug: attachment of directory-extractor failed for file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8
main debug: looking for stream_filter module matching "record": 26 candidates
main debug: using stream_filter module "record"
main debug: creating demux: access='file' demux='any' location='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' file='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8'
main debug: looking for demux module matching "any": 55 candidates
adaptive debug: Retrieving file:///Users/Admin/Desktop/vid_hls/mall.mp4.ke @0
main debug: creating access: file:///Users/Admin/Desktop/vid_hls/mall.mp4.ke
main debug:  (path: /Users/Admin/Desktop/vid_hls/mall.mp4.ke)
main debug: looking for access module matching "file": 23 candidates
filesystem error: cannot open file /Users/Admin/Desktop/vid_hls/mall.mp4.ke (No such file or directory)
main debug: no access modules matched
main error: no suitable access module for `file:///Users/Admin/Desktop/vid_hls/mall.mp4.ke'
adaptive debug: Period
adaptive debug:  BaseAdaptationSet default_id#0
adaptive debug:   Representation file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8
adaptive debug:    Segment #1 url=file:///Users/Admin/Desktop/vid_hls/small.mp4_00000.ts duration 560
adaptive debug: opening playlist file (/Users/Admin/Desktop/vid_hls/small.mp4.m3u8)
main debug: using demux module "adaptive"
main debug: looking for a subtitle file in /Users/Admin/Desktop/vid_hls/
main debug: looking for meta reader module matching "any": 2 candidates
adaptive error: Failed to create demuxer 0x0 Unknown
lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/reader
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/reader
lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/reader
lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/reader/filename.luac
main debug: no meta reader modules matched
main debug: `file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8' successfully opened
main debug: EOF reached
main debug: removing module "adaptive"
main debug: removing module "record"
main debug: removing module "cache_read"
main debug: removing module "filesystem"
main debug: dead input
main debug: changing item without a request (current 0/1)
main debug: nothing to play
bentasker commented 4 years ago

Hmmm, that's a fairly useless message :( Looks like it can apply to a range of things

Just to rule out the first google hit though, What are you hosting the files to test from?

For avoidance of doubt, can you check if it plays via my infra - https://filedrop.bentasker.co.uk/5/t1rLz_W-n2HBYPHQn0qvuacA=:NtgjYczwBXOdWd7RvXl9v9cZdZv7NTnU6BSib4G@YMMTKNmpLfgzFqQLbRKQ7xohU4P@aqUKeDoIFZ9aa14UU6SZTriiK8GX-mC:dhi7oIFTAJq@AIBYkEe68X=5/HLS/small.mp4.m3u8

For cvlc you'll need to launch a terminal, and then it should (hopefully) already be installed.

Failing that, in VLC if you attempt playback and then go to Tools (actually, might be "Window" on Mac) and then messages

bentasker commented 4 years ago

Sorry, just saw your latest. Bear with me

bentasker commented 4 years ago

I've managed to find an old macbook and there's something very weird going on here (VLC version is latest Vetinari).

It's truncating letters off the key-filename:

no suitable access module for 'http://scratch.holly.home:80/HLS/mall.mp4.key'

I noticed truncation in your output but assumed it might've been copy paste.

Verified on my webserver that it is requesting the truncated version

192.168.1.187   -       -       [16/Nov/2019:12:05:10 +0000]    "GET /HLS/mall.mp4.ke HTTP/1.0" 404     168     "-"     "VLC/3.0.8 LibVLC/3.0.8"        "-"     "scratch.holly.home"    CACHE_- 0.002   -

Dumping the file out as hex it looks ok

ben@milleniumfalcon:~/tmp/HLS$ xxd small.mp4.m3u8 
00000000: 2345 5854 4d33 550a 2345 5854 2d58 2d56  #EXTM3U.#EXT-X-V
00000010: 4552 5349 4f4e 3a33 0a23 4558 542d 582d  ERSION:3.#EXT-X-
00000020: 4d45 4449 412d 5345 5155 454e 4345 3a30  MEDIA-SEQUENCE:0
00000030: 0a23 4558 542d 582d 414c 4c4f 572d 4341  .#EXT-X-ALLOW-CA
00000040: 4348 453a 5945 530a 2345 5854 2d58 2d4b  CHE:YES.#EXT-X-K
00000050: 4559 3a4d 4554 484f 443d 4145 532d 3132  EY:METHOD=AES-12
00000060: 382c 5552 493d 736d 616c 6c2e 6d70 342e  8,URI=small.mp4.
00000070: 6b65 790a 2345 5854 2d58 2d54 4152 4745  key.#EXT-X-TARGE
00000080: 5444 5552 4154 494f 4e3a 360a 2345 5854  TDURATION:6.#EXT
00000090: 494e 463a 352e 3630 3030 3030 2c0a 736d  INF:5.600000,.sm
000000a0: 616c 6c2e 6d70 345f 3030 3030 302e 7473  all.mp4_00000.ts
000000b0: 0a23 4558 542d 582d 454e 444c 4953 540a  .#EXT-X-ENDLIST.

But, if we take the relevant line and try and convert it back the issue becomes apparent

ben@milleniumfalcon:~/tmp/HLS$ echo "382c 5552 493d 736d 616c 6c2e 6d70 342e 6b65 790a" | xxd -r
URI=small.mp4.ke

The end of that looks ok, hex 79 is dec 121 ("y" in ascii) and 0a is a line break. Let's iterate over the hex and convert to dec then

ben@milleniumfalcon:~/tmp/HLS$ echo "382c 5552 493d 736d 616c 6c2e 6d70 342e 6b65 790a"  | tr -d ' ' | python3 -c 'import sys,re; p=re.findall("..",sys.stdin.read()); [print(int(x,16))for x in p]'
56
44
85
82
73
61
115
109
97
108
108
46
109
112
52
46
107
101
121
10

Dumping those to chars

ben@milleniumfalcon:~/tmp/HLS$ echo "382c 5552 493d 736d 616c 6c2e 6d70 342e 6b65 790a"  | tr -d ' ' | python3 -c 'import sys,re; p=re.findall("..",sys.stdin.read()); [print(chr(int(x,16)))for x in p]'
8
,
U
R
I
=
s
m
a
l
l
.
m
p
4
.
k
e
y

Looks ok...

Converting one way and back works fine

ben@milleniumfalcon:~/tmp/HLS$ cat small.mp4.m3u8 | xxd > hex
ben@milleniumfalcon:~/tmp/HLS$ cat hex | xxd -r
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:YES
#EXT-X-KEY:METHOD=AES-128,URI=small.mp4.key
#EXT-X-TARGETDURATION:6
#EXTINF:5.600000,
small.mp4_00000.ts
#EXT-X-ENDLIST

Perhaps the earlier test was a false positive then?

This is ridiculous. I hate macs.

Going back to basics, what if we quote the keyfile name

ben@milleniumfalcon:~/tmp/HLS$ cat ~/scratch/HLS/small.mp4.m3u8 
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:YES
#EXT-X-KEY:METHOD=AES-128,URI="small.mp4.key"
#EXT-X-TARGETDURATION:6
#EXTINF:5.600000,
small.mp4_00000.ts
#EXT-X-ENDLIST

VLC works. And so does Safari....

FFS...

OK, for now, please edit your manifest and quote the keyfile name - I'll make a code change in a minute to have that happen automatically...

bentasker commented 4 years ago

OK, if you pull the latest master and try again the issue should now be fixed by 6fb42e9 (which stupidly referred to the wrong issue reference... happy saturday)

hardevine commented 4 years ago

OMG amazing? 😆 its working on safari and in vlc. yeah macs are weird.

bentasker commented 4 years ago

Yeah what's particularly weird is that VLC handles it just fine on Linux.

Suggests on a Mac it may be passing the manifest off to some OS handler (or maybe running OS specific code against it).

I think it's probably OS specific code though rather than an OS level handler. If I repro in Safari I don't actually see a key request at all (unlike VLC which makes one but has chopped the first and last char of the filename off).

Looking at RFC8216 though, it does say URI is a quoted-string, so this is definitely a bug in HLS-Stream-Creator even if the downstream behaviour is somewhat weird.

bentasker commented 4 years ago

HLS-35 Refers