Incorrently add BMFF for PlayReady tests and 500 license error

[calvaris]

Answering https://github.com/youtube/js_mse_eme/pull/17#issuecomment-255856808 :

The initData should have the keys as expected for clearKey regardless if we need them or not. This points to a problem with the initData the UA is providing. I suspect that you are using a different EME version other than 0.1b. That would explain why your initData is not the expected initData.

The story begins when I am fixing a problem with PlayReadyH264Video EME test as we were selecting the wrong decryptor. This is irrelevant for now as when I encountered this problem I stashed that code and just forced the selection of the PlayReady decryptor when I hit the JS issue and later the 500. Let's forget about this now then.

I am using v0.1b here, I am not building anything else.

How is the initData going to have the keys expected for ClearKey when we are providing PlayReady initData? FTR: The initData, the are providing for the PR engine, the generated key request and the server answer can be found at http://pastebin.com/neLhSDr4 . Of course this is after applying the fix I provided at https://github.com/youtube/js_mse_eme/pull/17, otherwise I would get no generateKeyRequest call.

I also was not able to reproduce the 500s on the license server. Can you provide more information with how you are running the tests or if you have made any modifications.

You have the info about the 500 on the pastebin link above.

I am running the following link for the tests http://mylocalipaddress/mstest/2016.html?enablewebm=false&command=run&test_type=encryptedmedia-test&tests=16 which contains changes at http://pastebin.com/w5tSPvsm . As you can see it is only the fix and some useful debugging (part of it was submitted, some was rejected and some accepted and reverted later, but I still find it useful).

I would be happy to look into this further but I need some more information on the failure. A complete stacktrace would be helpful, information on the UA EME version, and information about the 500 on the license server.

Info about the 500, you have it already. For UA EME version we run WebKitForWayland on a Raspberry Pi 2. I don't know what you mean by stacktrace? JS stacktrace? It is difficult to get one cause of having an embedded device but I can tell you that the get onneedkey event handler, then generateKeyRequest is called, that triggers onkeymessage handler that calls licenseManager.acquireLicense that ends up calling licenseManager.requestLicense that answers that I told you and that ends up calling the callback that calls video.addKey with the 500 answer.

At the moment I don't have an immediate issue with the fix you are proposing but I think it is a red herring for the problem you are experiencing. I'm closing this request as a result. Please feel free to file a bug report so we can analyze this problem further.

I opened this issue then. Let's see if it is a red herring or not by analyzing the problem further :)

[tdedecko]

We haven't selected a key system until generateKeyRequest in this case. Do you have multiple needkey events getting called for each of the pssh's? Is the event.keySystem set for the needkey event? Is clearkey supported on this device?

Thanks for providing the body of the request and response. Can you provide the request and response headers as well? If you are getting a 500 we should be able to reproduce outside of the device.

[tdedecko]

Do you have a user agent string to help identify the 500s you are experiencing?

[calvaris]

I am planning to add multiple needkey events. I have the code but I am not running it now in case it could interfere with any of the other errors.

We are setting the event key system to com.microsoft.playready.

ClearKey is supported on the device. I have other tests running it. Not the ones I talk about https://github.com/youtube/js_mse_eme/issues/12 because of the issue we discuss there but the others are running fine.

I cannot give you more info now, will try to get it tomorrow. Hope this helps

Just a question, what does ClearKey have to do with this test?

[tdedecko]

ClearKey doesn't have anything specific to do with the test. My concern is with a UA that does not return all psshs in the initData. This is required for YouTube. I agree that the code could be simplified to remove checking for a clearkey pssh since we aren't testing clearkey. Though, this test as a side effect is uncovering initData that is structured in an unexpected way. So, I think this is working as intended. I'll add a test to assert on the structure of the initData more explicitly in the future.

Probably not relevant but YouTube doesn't support com.microsoft.playready. We support com.youtube.playready.

As for the 500, I'm very curious to dig deeper into this. Let's work together to get this resolved.

[calvaris]

Here you have the headers http://pastebin.com/VnmDgpi9 .

About including all initData structures, it is very unclear at https://dvcs.w3.org/hg/html-media/raw-file/eme-v0.1b/encrypted-media/encrypted-media.html#algorithms-enrypted-block . According to you I have to fire a needkey event with all the found initDatas one after another? We consider finding a pssh box an "Encrypted Block Encountered" event, so we run the algorithm and fire a needkey event for every pssh box with the corresponding initData in each one.

You also asked if we were setting key system at the event. I guess, taking the spec and considering that we don't have a key system (actually you say that "We haven't selected a key system until generateKeyRequest in this case" at that point we should set it to null. If we have to add all initDatas in the same needkey event we definitely have to set the key system to null, if we fire one per pssh box, we can set it based on the UUID that we get.

Guessing the key system from the UUID is also complicated in this case as 9a04f079-9840-4286-ab92-e65be0885f95 stands for "Microsoft PlayReady" at http://dashif.org/identifiers/protection/ and we support both, but if I have to select one based on the UUID we go for Microsoft. There is no way to do otherwise unless I inspect the init data searching for YouTube or whatever. Anyway, this seems to be irrelevant as you don't use what is inside the event.

I hope this info helps with the 500.

I think I'll focus more on V2, the not promise based object oriented version that you support in 2016.

[tdedecko]

Clarification on the needkey event. There should be a needkey event called with the event having initData with all the psshs.

My reading of the spec is that if an encrypted block of media data is encountered then run the algorithm. This is media that needs a key to decrypt it. A pssh is not an encrypted block.

Our 2016 requirements ask the UA to implement specifically 0.1b since the not-promised-based object orientated version doesn't have a spec that documents the implementation. We also don't have tests against it.

Thanks for the information. Not sure if the user agent string you are using is going to be unique enough to find the error but I'll see what I can do. It would help a lot to have a user-agent string that is unique and that follows the description outlined in the Technical Requirements documentation (Section 13.1). If you can reproduce this issue with an updated user-agent string that would help a lot.

[calvaris]

I got you a build with a more specific UA http://pastebin.com/bwJh8kVK . The WPE was already there but I appended "calvaris PlayReadyH264Video". I guess it shouldn't be too difficult to find in logs.

I'll work on the initData stuff but currently I got any higher prio task, so it will take some days. I'd appreciate though if you can check the 500. Please, do not hesitate to ask me more info if needed.

[tdedecko]

Thanks for the more specific UA string. I'll update you soon.

[tdedecko]

We just had a new release. Can you try again. If you have same problem please post the headers again and provide a datetime for when you got the 500.

[tdedecko]

Any updates from your side?

[calvaris]

Sorry, I couldn't work on this, higher prio task. When I get back to this, I'll let you know.

[calvaris]

Back with this.

I updated the repo, still using the code that I submitted to be included, the one that prevents the JS issue because of not being a ClearKey request cause I couldn't deal with the initData issue yet. This is, we are sending only the corresponding initData for PlayReady for current EME test 17.

The answer is still 500, you can get the headers at http://pastebin.com/ZsmBiBw1, UA should be easily "greppable" and test was run at 15:52CET on Nov 10th, just a few moments ago.

[tdedecko]

We are having some trouble finding your request in our logs. We are still not sure what is causing this at the moment. We'll continue to look into this.

I noticed that you are running these tests on your own server. Do you still get a 500 if you run the test on our public website?

[calvaris]

I will run the test on your public webside as soon as I manage to get rid of the init data issue interfearing with the JS.

[calvaris]

A side comment, it is really (not) funny that when you sent back the initDatas back to generateKeyRequest, you send the whole lot instead of sending only the needed one.

I run the tests in Chrome 52 and they don't work.

[tdedecko]

Chrome 52 does not support EME 0.1b. generateKeyRequest shouldn't be supported. There is a similar test for 2017 and tip for EME Spec Version: 04 February 2016. This test passes for Chrome.

So, this is expected.

[calvaris]

http://yt-dash-mse-test.commondatastorage.googleapis.com/unit-tests/2017.html?command=run&test_type=encryptedmedia-test&timestamp=1479289934125 in Google Chrome 52 shows failures in test from 5 to 7 (included). This is the log http://pastebin.com/qqrZtsev.

[calvaris]

Got this https://github.com/youtube/js_mse_eme/pull/20 for you. It is harmless and can help debug.

[calvaris]

Ok, I managed to sort out the initData issue and I am sending back the three pssh atoms. This makes the new test 1 pass so that part should be ok.

Then I run http://yt-dash-mse-test.commondatastorage.googleapis.com/unit-tests/2016.html?enablewebm=false&command=run&test_type=encrypted media-test&tests=17 and still get a 500 even when the key generation seems to be right.

Here you have the HTTP headers of running that URL on Nov 16th 2016 at 13:32CET (of course, UA included in the headers): http://pastebin.com/SDCqxiS4

[tdedecko]

5 through 7 are expected to fail for Chrome 52. Chrome does not support PlayReady. For test 5, Chrome 52 does not support subsample encryption. I believe Chrome 55 will have that support. Please check the Canary if you are interested.

[tdedecko]

Thanks for running the tests on the public website. I'll look into this more. I'm still not sure why we haven't been able to find this error.

[calvaris]

About the 500, the sooner you figure out what is going on, the better cause it is preventing us from ensuring things work.

[tdedecko]

We are going to try to replay the request. Can you provide the request body as well?

[tdedecko]

Thanks for sharing the body in http://pastebin.com/tCLV28Wu

The body looks truncated. Are you setting the Content-Length in the request or sending a chunked request?

[calvaris]

You were quick :) I deleted de message cause I realized that I was printing it as a regular string and as it contains null characters the string was truncated during printing. I am working on getting a proper body.

I tried now at 10:41CET with the following results http://pastebin.com/p4cjMEUD . I had to try with my local test repo cause currently I'm a train with limited connectivity and for some reason the online version is failing

[calvaris]

I got an example against the upstream URL http://yt-dash-mse-test.commondatastorage.googleapis.com/unit-tests/2016.html?enablewebm=false&command=run&test_type=encrypted media-test&tests=17 run a few moments ago at 15:57CET: http://pastebin.com/ZQYJ88ca .

[tdedecko]

Looks like there are several characters after \ and before \ which are causing problems.

[tdedecko]

500 is the correct response because the license request is invalid. I'm going to close this since everything seems to be working as intended.

[calvaris]

Could you please point which characters are invalid?

[tdedecko]

lines 49 to 52 in the pastebin link.

[calvaris]

Thx, I'll investigate this.

[calvaris]

We were sending what was coming from the PlayReady engine. Now I sanitzed that to remove those characters you mentioned and replayed at 13:16CET with another 500 http://pastebin.com/i8w6UHQ5 .