search

video-dev / hls.js

HLS.js is a JavaScript library that plays HLS in browsers with support for MSE.
https://hlsjs.video-dev.org/demo
Other
14.93k stars 2.59k forks source link

Feature request: fMP4/Sample-AES decryption support #1491

Closed awakenedguy closed 6 years ago

awakenedguy commented 6 years ago
Steps to reproduce

Playback Sample-AES encrypted streams

Expected behavior

Plays fine, just as unencrypted version

Actual behavior

Playback of the Bento4 packaged Sample-AES version fails with an error code: The video could not be loaded, either because the server or network failed or because the format is not supported - CHUNK_DEMUXER_ERROR_APPEND_FAILED: Append: stream parsing failed. Data size=847 append_window_start=0 append_window_end=inf

Playback of the Sample-AES version packaged with Shaka packager fails with an error code: The video could not be loaded, either because the server or network failed or because the format is not supported - CHUNK_DEMUXER_ERROR_APPEND_FAILED: Append: stream parsing failed. Data size=952 append_window_start=0 append_window_end=inf

It seems that Sample-AES encrypted streams produced by Shaka Packager play fine in other players, as confirmed here.

Apparently hls.js seems to successfully read Sample-AES encrypted content produced by Kaltura NGINX-based MP4 Repackager, though I did not try this packager, as apparently it is not available as a standalone packager.

Until this is fixed, previously declared Sample-AES support on the main page should normally be declared partial.

mangui commented 6 years ago

indeed there might be an issue there, pinging @erankor

erankor commented 6 years ago

From what I see, both unplayable streams use fMP4 container, SAMPLE-AES in hls.js is supported only for MPEG2-TS container. While this can technically be added, supporting fMP4/SAMPLE-AES with cbcs scheme, is not a small task, and will make the library much larger. On the server side, it took me 1k lines of code to do it for AVC (https://github.com/kaltura/nginx-vod-module/blob/master/vod/avc_parser.c), and 2k lines to do it for HEVC (https://github.com/kaltura/nginx-vod-module/blob/master/vod/hevc_parser.c)

awakenedguy commented 6 years ago

It is fmp4 indeed. For me library size increase would not be a problem whatsoever, though I don't know it for others. E.g. there is a web player which takes 1.8M of uncompressed js source and around 0.9M minified, still it is fully usable. On the other hand, a separate hls.js bundle without Sample-AES fmp4 functionality could also be provided for those who don't need it.

erankor commented 6 years ago

Well, several months ago I submitted a pull for adding AC-3 support and it was not merged since the feature wasn't high enough priority for paying for the extra size... Either way, it's a tedious task to do it, those 3k lines I have in the server will need to be converted to JS for it to work... and unfortunately, I don't have the time to handle it myself

mangui commented 6 years ago

if it is SAMPLE-AES / fmp4 then couldn't we just use EME / ClearKey ? @tchakabam ?

kanongil commented 6 years ago

As far as I know, this is a matter of detecting the encrypted tracks in mp4demuxer.js, and hooking it into the sample-aes decryptor. Or am I missing something?

erankor commented 6 years ago

No, it's much more complicated... The encryption scheme that is used with HLS/fMP4 is normally cbcs (AFAIK, this is the only scheme Apple devices support). In this scheme, encryption of AVC/HEVC starts after the slice_header of the codec, beyond this point, every 10th block is encrypted. The hard part is figuring out the size of the slice header, that is the sole purpose of the ~3k lines of code I mentioned above... MPEG2-TS/SAMPLE-AES uses the same 10% encryption pattern, but there encryption starts at a fixed offset - after 32 bytes, that's the reason it's much easier. Btw, supporting cbcs for audio is easy, but doesn't have much value by itself IMHO...

awakenedguy commented 6 years ago

Right, it is exactly cbcs, here is how one call bento4 packager to produce hls fmp4 content encrypted with clear key Sample-AES: mp4dash --hls --encryption-cenc-scheme=cbcs --encryption-key=KID:KEY:IV audio.mp4 video.mp4

It is also written in the hls pantos spec:

An encryption method of SAMPLE-AES means that the Media Segments contain media samples, such as audio or video, that are encrypted using the Advanced Encryption Standard. How these media streams are encrypted and encapsulated in a segment depends on the media encoding and the media format of the segment. fMP4 Media Segments are encrypted using the 'cbcs' scheme of Common Encryption.

kanongil commented 6 years ago

@erankor Wow, you are right. Alternatively, we can require the input to have properly specified Sample Auxiliary Information, and use the BytesOfClearData to find the starting point. This should be significantly simpler to extract.

erankor commented 6 years ago

Right, that can work, however I'm not sure whether there's any packager at the moment that outputs this info with cbcs. For example, Apple's packager (mediafilesegmenter) which I consider as the 'source of truth' for HLS, does not. But maybe we should first ask - why is this feature important? why can't MPEG-TS/SAMPLE-AES or fMP4/AES-128 be used instead? The reason I implemented support for cbcs in the server was -

  1. FairPlay requires SAMPLE-AES
  2. HEVC requires fMP4

So if I want to play HEVC with FairPlay, I have to support cbcs. But this reason does not apply here, since hls.js isn't going to play FairPlay content anyway...

awakenedguy commented 6 years ago

The reasons are simple - compared to AES-128, Sample-AES decreases the complexity of the decryption process, so fewer CPU cycles are needed, mobile devices need less power consumption (longer battery life), which becomes more important on higher bitrates.

MPEG-TS is fundamentally inefficient and sub-optimal for delivery of media to mobile devices, particularly at high bitrates (this container format includes insufficient metadata, so the client has to dig around in the raw media stream to get at what it needs), also its high overhead is significant for any content producer, especially for shorter videos (lots of unnecessary padding), and ts segments are exactly short (I am using 2 seconds segments for optimal viewing experience). Compared to fMP4, MPEG-TS adds around 10-20% of free overhead (storage+traffic). It also depends on the ts muxer of course. E.g. ffmpeg's one is not efficient at all in this sense - the resulting overhead is really high.

erankor commented 6 years ago

I see, these are all valid points, a couple of comments from my side -

  1. For us, supporting the widest range of devices is probably the most important factor, and IMHO MPEG-TS is still on the lead in this aspect
  2. Regarding muxing overhead, that is correct, ffmpeg is very inefficient, but an efficient packager (e.g. Apple's or our on-the-fly packager) can reduce it to ~5%.
  3. Regarding the use of short segments, we are using this pattern for our segments - 2,2,2,4,10,10... this has the benefit of allowing the player to converge faster on the optimal bitrate, without having the cost of a large manifest + many http requests for the rest of the video.
awakenedguy commented 6 years ago

So for the moment I'll just stick with fMP4/AES-128, if fMP4/SAMPLE-AES is unlikely to be added. If it will be added and will work just fine, then i'll just repackage/reencrypt everything, as advantage is obvious.

I am only using hls.js based player and fMP4 works flawlessly, despite being declared beta, until now I did not see any device compatibility issues, it just works wherever MPEG-TS works. Third point is interesting, but seeking will suffer... fast, almost instantaneous, "offline-like" seeking is important to me. Many http requests and large manifests are the price to pay :)

On-the-fly packager is really nice, but currently I am sticking with a simple CDN and prepackaged approach, so unhappily not an option for me, though of course on-the-fly packaging has big advantages.

tchakabam commented 6 years ago

Can't you use the ClearKey CDM scheme to do this, since it is a CENC standardized format?

Then you can just pass through the fragments as they are to the browser, and pass the key via EME.

We are supporting EME API with Widevine content currently.

It wouldn't be hard to have support for ClearKey, as long as the application passed the key somehow :)

tchakabam commented 6 years ago

didn't see @mangui 's comment before. yes, you can :)

awakenedguy commented 6 years ago

https://github.com/gpac/mp4box.js/blob/master/src/parsing/saiz.js

This seems to do what is required. Shame that I am not proficient enough in js yet for such a task, otherwise I would have do it myself.

erankor commented 6 years ago

This parses a saiz atom, won't help you with cbcs, since it doesn't have this atom (unless you create some non-standard form of cbcs, as suggested above by @kanongil) Btw, what I meant earlier regarding support for devices was, for example, native players in smart TVs/STBs, I'm guessing that many of these still do not support fMP4 and SAMPLE-AES in general. If you always use hls.js, then I agree it doesn't matter much to you...

tchakabam commented 6 years ago

@awakenedguy If you are interested in having this feature, we can implement it for you on devices that support EME API via ClearKey scheme (Chrome, Firefox, Edge/IE11, Safari). Hls.js has already support for Widevine CENC scheme, adding ClearKey is a fairly easily reachable goal. Please contact me via email :)

kanongil commented 6 years ago

@tchakabam Afaik, ClearKey also requires a wellformed saiz box, and cbcs scheme support in the browser. Last I checked, Chrome has the code for it, but it is not enabled for PC / Mac builds.

tchakabam commented 6 years ago

Ok I see :) So this particular case might be unsupported by the EME implementation.

Thanks for enlightening me about the fact this isn't of course as simple. And apologies for my too quick general affirmation on this.

But more generally, Shaka-player/packager have example streams of ClearKey protected media, they actually play with Shaka. And that definitely relies on sample-level encryption.

So there are cases of fMP4 sample-encryption we can cover with ClearKey/EME :)

Maybe the interested party can also move it's payload to a sample-encrypted format-flavor which we can support?

It probably makes sense looking at the MP4 fragments generated by Shaka-packager for ClearKey operations.

@erankor How about Kaltura, do you have any working end-to-end solution using ClearKey with any kind of player/format?

awakenedguy commented 6 years ago

Thanks but for the moment I'll just stick to simple fMP4/AES-128, as the demand to fMP4/Sample-AES seems low.

erankor commented 6 years ago

@tchakabam, regarding your question above - our packager supports these forms of clear key -

  1. HLS/AES-128 (MPEG-TS/fMP4)
  2. HLS/SAMPLE-AES (MPEG-TS/fMP4)
  3. DASH/CENC At the moment, only HLS/AES-128 is actually being used on production. In many cases when a customer demands encryption, we go with full DRM (Widevine/Playready/Fairplay).

Other than that, one small comment regarding "there are cases of fMP4 sample-encryption we can cover with ClearKey/EME" - I assume you were referring to cenc scheme. While this scheme is also sample-encryption, the vast majority of the file ends up encrypted - everything after the AVC/HEVC NALU header is encrypted (there is no support for a 10% encryption ratio as in cbcs AFAIK) @awakenedguy wrote that one of his main reasons for asking for cbcs was to make the decryption lighter (less CPU usage/longer battery) and in this sense, I don't think cenc will provide any noticeable improvement compared to plain HLS/AES-128 encryption.

sumitdipsite2005 commented 6 years ago

Does this package support HLS/SAMPLE-AES decryption? I am trying to run the video with the m3u8: https://watch.spuul.com/a825389a-f801-44d0-a184-31d7e25b2b96/fp/1512117773/1080p/rendition.m3u8

But no success. Here is the code i been using:

<script src="//cdn.jsdelivr.net/hls.js/latest/hls.min.js"></script>
<video id="video" controls></video>
<script type="text/javascript">
      var video = document.getElementById("video");
      var videoSrcHls = "https://watch.spuul.com/a825389a-f801-44d0-a184-31d7e25b2b96/fp/1512117773/1080p/rendition.m3u8";

      if(Hls.isSupported()) {
        var hls = new Hls();
        hls.loadSource(videoSrcHls);
        hls.attachMedia(video);
        hls.on(Hls.Events.MANIFEST_PARSED,function() {
          video.play();
        });
      }

  function addSourceToVideo(element, src, type) {
    var source = document.createElement('source');
    source.src = src;
    source.type = type;
    element.appendChild(source);
  }  
       </script>

What am i doing wrong here?

erankor commented 6 years ago

This stream is FairPlay DRM protected (as can be seen by KEYFORMAT="com.apple.streamingkeydelivery"), will only play on Apple devices

sumitdipsite2005 commented 6 years ago

Thanks for the reply. Well i have subscription and can run the video on the website but unable to download it to my pc/mac. Any way to do so? Or is it a no go and i should stop wasting my effort?

erankor commented 6 years ago

In general, DRM can support offline playback, but if the publisher didn't set that up, it's probably a no go

sumitdipsite2005 commented 6 years ago

Yes the offline playback is available. I am able to download the file offline to my iphone. The file has .mp4 extension. What now? Can i play it i.e. remove the drm now?

awakenedguy commented 6 years ago

You probably misunderstood the intent of this library. hls.js is a javascript library for Apple HLS playback on the web. It is merely a playback engine, not a DRM remover :) Apple Fairplay DRM is an industrial-grade technology for copyright protection, which is expensive enough for a content producer to implement. It involves a lot of complex stuff in order to make it work, such as sophisticated license exchange and policy management.

You can't remove it like that against content producer's will, so... simply assume it impossible.

dannyfinks commented 6 years ago

Does anybody know who maintains the Bento4 demo examples? They are no longer available.