DRM performance on various devices

[kgrevehagen]

[REQUIRED] Searched documentation and issues

ExoPlayer Javadoc and code, exoplayer.dev, issue tracker, stack overflow.

[REQUIRED] Question

When implementing DRM support in ExoPlayer I feel it is not so straight forward as the documentation states. The implementation itself is fine and works as expected on "good" devices, but when you start shipping this to the masses with x number of different "not-so-good" devices(and lower api levels) we start seeing problems. I want to address some of these here to try to clear up what to expect, what workarounds can be made etc.

It all boils down to the native call MediaDrm.provideKeyResponse(...) which on these slow devices I have seen take up to 2500 ms, compared to below 100 ms on faster devices. From my understanding this is called on a single threaded trusted execution environment(TEE). provideKeyResponse of course needs to run in this TEE, and since the media being played is using DRM, decrypting also have to happen on this same thread within the TEE!? This means that for these devices it will either slow down startup time with 2500 ms or it will stall the playback for up to 2500 ms(depending on buffer size) if you are using clear lead. Both of these will also mean that gapless playback in a ConcatenatingMediaSource will not work.

In some issues you mention that there has been done improvements on Android 10 and concurrency(https://github.com/google/ExoPlayer/issues/6344#issuecomment-532084342 and https://github.com/google/ExoPlayer/issues/6751#issuecomment-579417711). What exactly are those improvements? Is that something that is added in Android 10 that ExoPlayer can utilize or did ExoPlayer only add it for Android 10? Do you have a commit/PR I can look into to see what this change is, so that I can better understand what is happening and/or try to make other improvements myself? Is it somehow possible to apply these improvements(or other similar workarounds) for earlier versions? Are all Android 10 guaranteed to not cause stalls or longer startup time due to this improvement? Does it exist a multi threaded trusted execution environment?

Since many Android 9 versions actually are fast enough I still want the to be able to use drm, are there any way we can figure out which devices this is working on?

You also mentioned in one of the issues(https://github.com/google/ExoPlayer/issues/6751#issuecomment-579517982), that you wondered if this happens on Android 8.1 or above? What was the reason for that? Is there anything that should've changed for that version? Are there any differences of this on 8.1 and below? In that case, what are the differences between 8.1 and Android 10?

There is also mentioning about "other DRM improvements"(https://github.com/google/ExoPlayer/issues/4133#issuecomment-527394423), can you elaborate on what they are and when they are expected to be released?

What about the bufferSize in DefaultAudioSink? Are there any guides on how big this can be? If we succeed to make it a certain size, will it ever cause problems as long as the AudioTrack gets initialized? As you wrote (https://github.com/google/ExoPlayer/issues/6751#issuecomment-571145807), we could try again with a smaller value if it fails to initialize(but then we're at square one either way).

Has any of this anything to do with audio offloading(mentioned in the bottom here: https://exoplayer.dev/battery-consumption.html) or improved playback rendering(https://medium.com/google-exoplayer/improved-rendering-performance-operating-mediacodec-in-asynchronous-mode-and-asynchronous-buffer-3026207850b2)?

Sorry for all the various questions, but they all relate to each other, I'm just trying to find out how I can best implement DRM with ExoPlayer without degrading performance compared to non-DRM.

I completely understand that you cannot do anything about the slow call and the single threaded TEE, but since you support DRM, it would be nice to know where we can use it and how it affects performance on devices that are not super fast. It would be OK for us to only use this on Android 10, but then we need to be confident that it will actually work on ALL android 10 devices.

Hope you can answer some of these questions as I think they are crucial to actually applying DRM on a release build without the user noticing any bad performance.

Thanks!

A full bug report captured from the device

N/A

Link to test content

N/A

[ojw28]

@jt64 - Please take a look.

[kgrevehagen]

Hey. I don't want to nag you, but is it possible to get some feedback on this? Really looking forward to hearing your views on this topic :) Thanks in advance!

[kgrevehagen]

Hi again! After some more digging I found out that there are more calls that take a significant time before they return. All of these happen in FrameworkMediaDrm.java:

1.

this.mediaDrm = new MediaDrm(adjustUuid(uuid));

Pixel 3: about 10-30 ms LG G3: about 15-30 ms

2.

byte[] bytes = mediaDrm.openSession();

Pixel 3: about 200-350 ms LG G3: about 100 ms

3.

MediaDrm.KeyRequest request =
        mediaDrm.getKeyRequest(
            scope, initData, mimeType, keyType, optionalParameters);

Pixel 3: about 90 ms LG G3: about 2000 ms

There might be more, but these are the ones I have seen so far.

All of the above is called before playback starts, so we're not getting a glitch, but increased startup time. As you can see, for the LG G3(old device) it takes an average of about 5 seconds in total, which is pretty unusable. But even for a newer and more performant device, the total extra averages around 500 ms, which is quite a big increase compared to non-drm.

After modifying the code a bit to not call openSession() and getKeyRequest(), I can perfectly fine start playback. Why isn't these long running calls part of the playClearSamplesWithoutKeys logic? However, if they were, I'm afraid it will add a longer glitch(I have yet to test this, though), so that brings us back to the original question regarding the threading issues with these calls.

@ojw28 or @jt64: Any tips on how to work around this? Any tips or guidance would be very highly appreciated. I can't be the only one noticing this or are concerned about this? That makes me think I might be doing something wrong :/

Thanks again!

[kgrevehagen]

Hi again.

I did even more research around this topic and I am pretty sure I have found the root cause of this. There's mainly two issues I see with the current DRM implementation when using clear lead.

1. Slower startup time: As briefly mentioned in the previous commit, we prepare, acquire and start fetching keys(mediaDrm.getKeyRequest) before starting playback. This is not necessary, because playback can perfectly fine start without doing any drm related things just yet.

2. Audible glitch when fetching keys: mediaDrm calls are blocking, so when these take time to run, they will simply block the execution in MediaCodecRenderer.render() because we are using a codec with a crypto. But, when starting playback on a non-crypto codec we can do all the longrunning mediaDrm calls without them blocking probably because the codec does not talk with the same mediaDrm instance that are being blocked. However, since some of these longrunning mediaDrm calls are made on the playback thread, it will still block, so it will play until the buffer is done, then it will block on codec.queueInputBuffer, and then we will have the glitch and playback will again continue when the block is released. This is simply fixed by putting these mediaDrm calls on another thread in DefaultDrmSession.java.

I'm sorry for the bad explanation, as I tried to keep it short and sweet, but I can explain in more detail if that is needed.

Does this seem correct and like a reasonable fix? I made those fixes(along with some other minor issues), but then I encountered another problem. That is basically that we do get a very tiny(but still audible) glitch when switching from non-secure codec to a secure one. I will make another ticket to track that issue: #7613

If the codec switching issue is something we can work out, I'm happy to contribute with these changes if that is of interest!

Thanks again!

[ojw28]

@jt64 - Could you comment on the slow DRM calls?

[rrfrias]

We are looking at performance data from devices and discussing options to improve this situation. Will update when we have some information to share.

[ziad-halabi9]

@rrfrias Thank you for your comment. Is there any update regarding this? Or at least, is there any foreseeable timeline for when a solution will be provided? Thanks!

[jt64]

While we are looking into the slow drm calls across different devices and can improve this in future releases, realistically we won't be able to update all existing devices with any changes. Your suggestions about changing the threading model by, as you said "putting these mediaDrm calls on another thread in DefaultDrmSession.java", seem to be the most likely way to resolve this in the short term since it would be an update to the application and can be released soon.

@ojw28 can you look into whether this is feasible?