Safari Fairplay MediaKeys Polyfill

[sanbornhilland]

In issue #279 it was mentioned that "The status of Fairplay through EME is unknown at this time." Can you elaborate on what you mean by this? We have some apple sample code that shows Fairplay playback in Safari. However trying to play the same content through Shaka with an HLS Parser plugin is not going so smoothly. I'm not sure the MediaKeys polyfill is working correctly on Safari. Have you tested in Safari?

I can get more detailed logs and specific failings in the next few days but I wanted to initially enquire about the compatibility of the MediaKeys polyfill and the Safari (9.1) EME implementation.

[joeyparrish]

What I meant by that comment is that we have never seen any Fairplay-compatible DASH content we could try in Shaka, nor any EME-based Fairplay demos using HLS. So we can only assume Apple's EME works. Can you share a link to something of that sort?

A quick check shows that Safari 9's EME is the webkit-prefixed v0.1b API. The polyfill for that was originally developed against Chrome, but Chrome has long-since implemented the unprefixed working-draft EME API. If we need to modify the polyfill to better support Safari, I have no problems with that. Safari is, at the moment, the only known consumer of it.

[sanbornhilland]

We have a server issue to resolve but then I should be able to provide you with a working Safari sample with HLS Fairplay content.

[sanbornhilland]

I'm still working on getting you some HLS + Fairplay samples to give you. But there is sample client code from apple here: https://developer.apple.com/streaming/fps/ if you download the FairPlay Streaming Server SDK. If you haven't looked at this it would be useful for comparing to the v01b polyfill.

For instance, for FPS on Safari the init data is formed by concatenating the initData from the needkey event with a contentId and an Apple App Cert. I believe this is different from Chrome and other browsers that just require the init data? It looks to me like currently Shaka and the MediaKeys polyfill assumes the needkey event.initData is sufficient.

After that it looks like there are some other differences. The concatenated init data is passed in the createSession call and then there is no generateRequest call at all. This also appears to be different than how Shaka is handling it where the createSession call does not take any arguments and there is a generateRequest call that takes initData.

It appears to me that the flow on Safari is a bit different than on Chrome. I have tested that Apple sample code with HLS + Fairplay content and it does work in Safari.

I'm also having the following issue in Safari: In shaka.polyfill.PatchedMediaKeys.v01b.MediaKeys.prototype.onWebkitNeedKey_ it throws an error when trying to dispatch the shaka FakeEvent: UNSPECIFIED_EVENT_TYPE_ERR: DOM Events Exception 0: The Event's type was not specified by initializing the event before the method was called. The only way I can see to avoid that is to create a custom event instead like this:

var event2 = new CustomEvent('encrypted');
event2.initDataType = 'webm';
event2.initData = event.initData;

I'm not sure you'd want to handle it like this just for consistency reasons but this was how I could get around that error just for POC purposes.

[sarge]

@joeyparrish would it help if I provide some test content to run against?

[joeyparrish]

@sarge, it couldn't hurt! Thanks!

[boredom2]

Hi there - we are now also finally under Pressure to run a DRM Solution for Safari asap. I am really a little confused :) We have Assets in HLS and with FairPlay Protection ready. Initially, I thought, that Safari will handle the Process itself, but looks, like if we have to do something here :) So Apple DRM also works on MSE Base? And if yes, I suggest, this will only work with HLS (DASH with FairPlay is afaik not possible to create or at least never done and for sure not possible to create on MS Azure). So can you maybe give me a little Hint on how to make this work? Did someone ever made HLS+FairPlay+MSE work at all? (and if you know that - how does this work on iOS, if this does not support MSE?)

[sarge]

@joeyparrish I have uploaded a working sample of the fairplay drm. https://s3.amazonaws.com/shift72-temp/hls_fps_bento4_sintel/master.m3u8

This file contains reference to the "Apple Developer Program License Agreement", so you can choose not to view it. Not sure what to do about worlds colliding here (shrug) https://s3.amazonaws.com/shift72-temp/hls_fps_bento4_sintel/safari_hls.html

@boredom2 HLS + Fairplay works on Safari Desktop. But does not work on Safari iOS, an app is still required. Dash + Fairplay + (Widevine + Playready) + MSE appears possible, implementations are rare from what I have seen. An additional pssh box with the fairplay id and some work generating the initData in the format [initData] = [initData] + [4 byte: idLength] + [idLength byte: id] + [4 byte:certLength] + [certLength byte: cert] should work.

[boredom2]

Thanks for that Information - I will try once SHAKA can handle the slightly "crippled" HLS Manifest, that we get from MS Azure CDN. I will update then here :)

[joeyparrish]

Here's an update on our progress.

The polyfill part is not that difficult. I have written a polyfill that builds a compliant MediaKeys API on top of Safari 10's WebKitMediaKeys.

Parsing the FairPlay tag in HLS to feed EME is trivial.

The challenge is that the key system com.apple.fps.1_0 seems to be usable only with video.src=foo.m3u8. When used with MediaSource, createSession() causes the page to crash and reload. And Shaka Player currently only works with MediaSource.

Judging from a WebKit commit log, they introduced com.apple.fps.2_0 for MediaSource:

(WebCore::keySystemIsSupported): Use "com.apple.fps.2_0" to distinguish from the not-media-source scheme.

I have not been able to find any working examples of that key system in use.

Creating a session with fps.2_0 results in a message that contains only the text "certificate". Looking at WebKit sources, this is unconditional and we are expected to call update() with the certificate after that.

Dealing with this certificate request is enough to do, and we can even hide it in the MediaKeys polyfill. However, the message we get after that is not a license request, either. It's completely empty.

WebKit sources also show support for com.apple.fps.3_0, but that also has no documentation and does not get us any further than fps.2_0 did.

[sarge]

Thanks Joey for taking a look at this. You got further than I did. Would you mind pushing a branch?

fps.2_0 being for MSE is a good find - the apple forums note that NetFlix uses this scheme.

When you say after the certificate request you get a message that is empty do you mean that get a call to webkitkeymessage with no initdata?

[joeyparrish]

I mean that the second webkitkeymessage event has message.length of zero.

[joeyparrish]

I can't push anything until it's been through code review by the team, but I'll try to clean up what I've done so far and make it available in some form.

[avelad]

Safari 11, now it's in beta with macOS High Sierra. Are there any change on eme-fairplay?

Shaka Player support.html reports:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13) AppleWebKit/604.1.21 (KHTML, like Gecko) Version/11.0 Safari/604.1.21 v2.1.3

{ "manifest": { "application/dash+xml": true, "application/x-mpegurl": true, "application/vnd.apple.mpegurl": true, "application/x-offline-manifest": true, "mpd": true, "m3u8": true, "application/vnd.ms-sstr+xml": false, "ism": false }, "media": { "video/mp4; codecs=\"avc1.42E01E\"": true, "video/mp4": true, "video/mp4; codecs=\"avc3.42E01E\"": true, "video/mp4; codecs=\"hvc1.1.6.L93.90\"": true, "audio/mp4; codecs=\"mp4a.40.2\"": true, "audio/mp4": true, "audio/mp4; codecs=\"ac-3\"": true, "audio/mp4; codecs=\"ec-3\"": true, "video/webm; codecs=\"vp8\"": false, "video/webm": false, "video/webm; codecs=\"vp9\"": false, "video/webm; codecs=\"av1\"": false, "audio/webm; codecs=\"vorbis\"": false, "audio/webm": false, "audio/webm; codecs=\"opus\"": false, "video/mp2t; codecs=\"avc1.42E01E\"": true, "video/mp2t": true, "video/mp2t; codecs=\"avc3.42E01E\"": true, "video/mp2t; codecs=\"hvc1.1.6.L93.90\"": true, "video/mp2t; codecs=\"mp4a.40.2\"": true, "video/mp2t; codecs=\"ac-3\"": true, "video/mp2t; codecs=\"ec-3\"": true, "text/vtt": true, "application/mp4; codecs=\"wvtt\"": true, "application/mp4": true, "application/ttml+xml": true, "application/mp4; codecs=\"stpp\"": true }, "drm": { "org.w3.clearkey": null, "com.widevine.alpha": null, "com.microsoft.playready": null, "com.apple.fps.2_0": null, "com.apple.fps.1_0": null, "com.apple.fps": null, "com.adobe.primetime": null }, "offline": true }

In MAC OSX El capitan reports:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7 v2.1.3

{ "manifest": { "application/dash+xml": true, "application/x-mpegurl": true, "application/vnd.apple.mpegurl": true, "application/x-offline-manifest": true, "mpd": true, "m3u8": true, "application/vnd.ms-sstr+xml": false, "ism": false }, "media": { "video/mp4; codecs=\"avc1.42E01E\"": true, "video/mp4": true, "video/mp4; codecs=\"avc3.42E01E\"": false, "video/mp4; codecs=\"hvc1.1.6.L93.90\"": false, "audio/mp4; codecs=\"mp4a.40.2\"": true, "audio/mp4": true, "audio/mp4; codecs=\"ac-3\"": true, "audio/mp4; codecs=\"ec-3\"": true, "video/webm; codecs=\"vp8\"": false, "video/webm": false, "video/webm; codecs=\"vp9\"": false, "video/webm; codecs=\"av1\"": false, "audio/webm; codecs=\"vorbis\"": false, "audio/webm": false, "audio/webm; codecs=\"opus\"": false, "video/mp2t; codecs=\"avc1.42E01E\"": true, "video/mp2t": true, "video/mp2t; codecs=\"avc3.42E01E\"": false, "video/mp2t; codecs=\"hvc1.1.6.L93.90\"": false, "video/mp2t; codecs=\"mp4a.40.2\"": true, "video/mp2t; codecs=\"ac-3\"": true, "video/mp2t; codecs=\"ec-3\"": true, "text/vtt": true, "application/mp4; codecs=\"wvtt\"": true, "application/mp4": true, "application/ttml+xml": true, "application/mp4; codecs=\"stpp\"": true }, "drm": { "com.apple.fps.1_0": { "persistentState": true }, "com.apple.fps": { "persistentState": true }, "org.w3.clearkey": null, "com.widevine.alpha": null, "com.microsoft.playready": null, "com.apple.fps.2_0": null, "com.adobe.primetime": null }, "offline": true }

[joeyparrish]

At a glance, the EME API has not changed much in Safari 11 Tech Preview. I have not yet investigated deeper to see if FairPlay is any easier to use.

[sarge]

I have had a dig into how other well know players appear to work. These are the functions and events I have seen. I am a little unclear on the role of webkitkeymessage at this stage.

[Info] EME LOGGER: WebKitMediaKeys.isTypeSupported (2) (inject.js, line 54)
"com.apple.fps.3_0"
"video/mp4"

// get the init data from the content
[Log] video.webkitneedkey called – Uint8Array (253) (inject.js, line 85)
Uint8Array (253)

// create the session with the initdata 
[Info] EME LOGGER: WebKitMediaKeys.create session (2) (inject.js, line 69)
"video/mp4"
Uint8Array (251)

// 
[Log] session.webkitkeymessage called – Uint8Array (11) (inject.js, line 73)
Uint8Array (11) 

// certificate being added
[Info] EME LOGGER: session.update called – Uint8Array (1246) (inject.js, line 61)
Uint8Array (1246)
[Info] EME LOGGER: session.update – "MIIE2jCCA8KgAwIBAgIIBRGnbPd8z1YwDQYJKoZIhvcNAQEFBQAwfzELMAkGA1UEBhMCVVMxEzARBgNVBAoMCkFwcGxlIEluYy4xJjAkBgNVBAsMHUFwcGxlIENlcnRpZmljYXRpb24g…" (inject.js, line 62)

// license being added?
[Log] session.webkitkeymessage called – Uint8Array (6576) (inject.js, line 73)
Uint8Array (6576)
[Info] EME LOGGER: sessionupdate – Uint8Array (1052) (inject.js, line 61)
Uint8Array (1052)
[Info] EME LOGGER: session.update – "AAAAAQAAAAAeuevhIffYAy83tkB6vtdqAAAEAPX5+hQW6VnoZjC6GVznn3zbfn86yM1UM22rXjEfriDHzw9lDiGPzGLYGFtMB6nT1kAwYeSvNZJ9CHxIqTZWf4VTt3Q1AGTZkgieM7Rs…" (inject.js, line 62)

// playback succeeds

[joeyparrish]

That looks a lot like what I was able to achieve, except that when I get the 11-byte certificate message and pass my certificate back to update(), the next message I get is empty. I'm not sure if my cert is bad or if the original init data was to blame. (Presumably the real license request based on init data was deferred during cert request, so it could be the init data's fault.)

[sarge]

So I reached out to Apple for support on this issue and the feedback is that this is not currently part of their public api, so would not offer any support in this matter. I filed a feature request as suggested. It could be any number of issues at this point, I would not rule out a whitelist on the certificate or signing authority restricting this feature.

[joeyparrish]

In that case, there is probably nothing that we can do to complete this work. If Apple actively does not want us to do it, there may be no point in continuing. I will still endeavor to clean up and publish my work-in-progress FairPlay polyfill, in case it is useful to anyone in the future.

@sarge, is your support conversation with Apple public? If so, could you provide a link to help document the current situation?

[sarge]

@joeyparrish Unfortunately all done comms were over email, also the feature request does not have an external link either. I will update this thread if I hear back.

I agree that this issue can be closed until we hear more.

Thanks for your efforts:)

[joeyparrish]

Hi all,

I just published a new branch called fairplay including just one unique commit with what I was able to accomplish for FairPlay in Safari with MediaSource.

I hope this is a useful starting point for someone to help us figure out the missing pieces, assuming it is possible at all. For now, I am moving this issue back to the backlog. Please let us know if you have any insights or if you can contribute.

Thanks, Joey

[chrisfillmore]

I was on a call today with Apple where they assured me HLS-FP playback was possible using MSE and EME in Safari, and that solutions existed in market which used these API's. Hopefully we can get some support from them, so I'll send you an email.

[bbert]

Hi, Is it possible to share information you got from Apple to make HLS+FP playback working? Guidelines from WWDC15's presentation (https://developer.apple.com/videos/play/wwdc2015/502/) seems not to be enough. Especially if a certificate is required.

[chrisfillmore]

@joeyparrish I’m in touch with Apple about getting a license challenge message in Safari using MSE. You mention calling MediaKeySession#update with the certificate after you receive the phony “certificate” keymessage. I’m working on a demo to send to Apple but I’m not seeing the same behaviour you are, the empty keymessage event. I just get a webkitkeyerror from the session.

Have I understood your post correctly? I’ll take a closer look at your fairplay branch to see if I’ve missed anything. I’ve also sent you the demo app I’m working on so far, in case you want to look at it (you should receive an invite).

[chrisfillmore]

Just to add more detail, this is the callback I have attached to webkitkeymessage:

function onKeyMessage (e) {
  console.debug('webkitkeymessage was fired', e);
  var strMessage = String.fromCharCode.apply(undefined, e.message);

  if (strMessage === 'certificate') {
    console.debug('cdm requested the certificate');
    keySession.update(new Uint8Array(certificate)); // Do I call update like this??
  } else {
    console.debug('license challenge available');
  }
}

It's possible I have some other issue. I know the certificate works in general, as I've played HLS-FP content with it under other circumstances. Also, I am using com.apple.fps.2_0.

[chrisfillmore]

@joeyparrish I've learned it's not necessary to concatenate the initData with the certificate when using com.apple.fps.2_0. The initData should be passed to createSession() as-is.

[avelad]

@chrisfillmore with this change, can you use com.apple.fps.2_0 without problems?

[chrisfillmore]

@avelad I can't comment on playback, because the test app I wrote doesn't get that far. But I can produce a Fairplay license challenge.

[sandeep-mirpuri]

@joeyparrish @sarge any updates on this?

[joeyparrish]

Hi all,

We haven't had time to look at this in the last couple months. It will be a priority for v2.3, which we will be starting soon. I apologize for the delay.

[avelad]

Hi @joeyparrish , can you schedule it in v2.3 milestone?

[avelad]

@joeyparrish any updates on this?

[chrisfillmore]

I wanted to add: I don't think it's necessary to call DrmEngine#formatFairPlayRequest_. Right now we're doing HLS-FP playback in Safari via src= and I don't do any manipulation on the message from the CDM, I pass it on directly to the server. This may depend on how the license server is configured?

[joeyparrish]

Sorry for the lack of updates. We haven't had time to work on this lately, but we hope to get back to it soon. This may be obviated by #997, at which point we don't need to do quite so much work to make FairPlay + MediaSource work.

[chrisfillmore]

Safari 11 is out: https://developer.apple.com/library/content/releasenotes/General/WhatsNewInSafari/Safari_11_0/Safari_11_0.html

I notice that requestMediaKeySystemAccess is once again undefined. :(

[ghost]

@chrisfillmore Any reason why you need to use com.apple.fps.2_0 rather than com.apple.fps.1_0.

The Apple FairPlay example app uses the latter keysystem - however I do notice that most of the functionality seems to be embedded underneath the video tag (Not exposed via MSE)

[chrisfillmore]

com.apple.fps.1_0 is for streaming via videoElement.src = playlist.m3u8. This is the method of playback documented in the Fairplay SDK demo.

com.apple.fps.2_0 is for playback via MSE.

[barbarosalp]

Hi @joeyparrish, Could you guys have a time to work on this lately? Thanks.

[chrisfillmore]

@joeyparrish I am tinkering with the fairplay branch to see if I can get it working. Do you want to assign to me? It won't get done overnight, but I do have time to work on it.

[barbarosalp]

Hi @chrisfillmore, there is an Apple provided code example for fairplay. If you don't have it, I can send them.

[chrisfillmore]

Thanks @barbarosalp, I have it.

[chrisfillmore]

@joeyparrish Could someone update fairplay with the latest from master?

I am hoping I can submit a PR against fairplay with this commit:

https://github.com/chrisfillmore/shaka-player/commit/8a057d7a9b7491dc79625923dd413c396b028af2

I haven't achieved playback yet but I can make a successful license request. I'm trying to play some TS content but it's not working, I'll need to do more investigation. I'm hoping in the meantime we can review the changes I've made.

[KarlGallagher]

@chrisfillmore I took the liberty of briefly reviewing this change. The only comment I have (for now) is that there seems to be some application specific logic embedded within the generic library implementation. This is concerning license acquisition from a remote KSM and subsequent response parsing. For other DRM systems the application developer is simply handed off raw bytes by Shaka, which you can then chose to prepare yourself via a network 'hook' - e.g. the developer is free to chose their own request format (URI encode, HTTP header..etc)

The same applies for response parsing - Shaka expects an ArrayBuffer to be returned, but the application is free to use a custom response format (and add the appropriate parsing hook)

[chrisfillmore]

Hi @karlg-arris, this implementation would still allow custom license requests and responses via the networking engine. Nothing has changed in that layer.

I think the only app-specific code which I neglected to remove was in drm_engine:

request.headers['Content-Type'] = 'application/octet-stream';

[KarlGallagher]

Sorry @chrisfillmore - thanks for clarifying !

[chrisfillmore]

Beep beep Shaka team

I'd like to submit a PR against fairplay so that we can advance the state of FP support in Shaka, even if it's not all the way to playback. Can you advise on next steps @joeyparrish ?

[joeyparrish]

Based on feedback from our survey, this is at the top of our list for v2.4. I'll try to get the branch updated against master soon.

[joeyparrish]

I pushed updates to the fairplay branch. I hope this helps.

[chrisfillmore]

Thanks Joey, sorry I haven't looked at this again yet. I will submit a PR early next week so I don't fall too far behind.

[joeyparrish]

Sounds good. We are no rush at the moment. It's something we plan to give more attention to for v2.4 in January.

[chrisfillmore]

I invite anyone that has HLS-FP content that they can test with to give this commit a shot. I will need to integrate this into my custom app to do more thorough testing. That probably won't happen till January. Merry Christmas everyone!