search

w3c / encrypted-media

Encrypted Media Extensions
https://w3c.github.io/encrypted-media/
Other
180 stars 80 forks source link

Should Wait for Key set readyState to HAVE_CURRENT_DATA when playing? #338

Closed ddorwin closed 7 years ago

ddorwin commented 7 years ago

The Wait for Key algorithm currently says to set readyState to HAVE_METADATA. This makes sense if no frames have been decoded, such as when playback starts or a seek occurs. However, what is the correct value when playback encounters block it cannot decrypt?

This appears to depend on whether "the immediate current playback position" is the frame for which we do not have the key or the frame before it. In other words, did the "current playback position" advance before the Encrypted Block Encountered algorithm was called.

If so, HAVE_METADATA is likely correct. Otherwise, readyState should probably be HAVE_CURRENT_DATA.

A note in the HTML spec says the following about the difference between the two states:

In practice, the difference between HAVE_METADATA and HAVE_CURRENT_DATA is negligible. Really the only time the difference is relevant is when painting a video element onto a canvas, where it distinguishes the case where something will be drawn (HAVE_CURRENT_DATA or greater) from the case where nothing is drawn (HAVE_METADATA or less).

readyState changes were discussed in #129, but I don't believe this specific case was discussed.

@foolip and @cpearce for comments

ddorwin commented 7 years ago

The Attempt to Resume Playback If Necessary algorithm includes the step "If the user agent can advance the current playback position in the direction of playback."

This could indicate that the current playback position was decryptable and advancing was blocked. Alternatively, advancing could mean attempt to decrypt the current frame, which will allow the time to be advanced.

cpearce commented 7 years ago

The crux of this issue is whether we want readyState to reflect whether data is buffered or whether data is renderable at and after the current playback position.

In the past month or so Firefox 52 changed so that our readyState calculation for HAVE_CURRENT_DATA and HAVE_FUTURE_DATA is now basically a function of how much data is buffered after the currentTime is in the buffered ranges, i.e. basically what @foolip described here: https://github.com/w3c/encrypted-media/issues/129#issuecomment-159632283

I was under the impression other browsers implemented readyState this way, so we changed.

If we want readyState to take into account whether content at the currentTime can be rendered, then it makes sense for readyState to be HAVE_CURRENT_DATA if we'd been advancing playback over a decryptable range and then we find we need a key.

foolip commented 7 years ago

@ddorwin @wolenetz, do you know the details of how readyState is calculated in Chrome? Is it just a bunch of call sites changing it back and forth so that it can't really be said to follow any principle at all?

ddorwin commented 7 years ago

@foolip, yes, it appears to be five sites that change it in response to actions or events. See the call sites for SetReadyState.

The current readyState reduction in Chrome occurs due to underflow (like it does for non-encrypted content). This is also the cause of the issue mentioned in #339. The good thing is that the EME behavior is consistent with normal behavior, which i believe was the intent. Unfortunately, that does not match the EME spec.

cpearce commented 7 years ago

@ddorwin:

Unfortunately, that does not match the EME spec.

By not matching the EME spec, do you mean you'll fallback to HAVE_CURRENT_DATA rather than HAVE_METADATA when you're waiting for a key?

ddorwin commented 7 years ago

Yes, I believe we currently report HAVE_CURRENT_DATA (after three seconds - see #339). More broadly, we do not currently have specific behavior for EME. Returning to my statement, just being consistent with unencrypted behavior is not currently EME spec compliant - the EME spec is currently very prescriptive.

Adding to my preceding comment, I'm not sure which frame the "current playback position" refers to in any of these cases in Chrome. That is possibly the most important question in deciding between these two values.

cpearce commented 7 years ago

The Firefox media playback team discussed this issue, and we think it makes sense for the readyState to transition to HAVE_CURRENT_DATA when we were playing but then we start waiting for key, and to transition to HAVE_METADATA if we hit waiting for key before we've played anything.

If we can't advance the current playback position, but can still render the frame at the current time, we think it makes sense to be in readyState HAVE_CURRENT_DATA.

ddorwin commented 7 years ago

@cpearce, I agree. Do you have proposed language? Could you make a PR to fix the spec?

I like the language in https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0042.html ("the last frame we decrypted and decoded is at the current playback position and is currently being rendered"), but I don't know how easy that is to turn into spec text.