Blocked UI when using large playlists

[juliankotrba]

Issue

We are experiencing UI lags when using big playlists containing podcast urls. Specifically, we are using a MediaController where we usesetMediaItems to pass around 600 MediaItems. The Items are presented in a RecyclerView where users can select the desired item. After selecting an item by click, we call MediaController#seekToDefaultPosition. This call clearly freezes the UI and also makes everything unresponsive for a few milliseconds. We have not checked if the problems also occurs with big playlists of local audio files.

Reproduce

We successfully reproduced the issue in your session demo project by slightly modifying the following line

https://github.com/androidx/media/blob/72a4fb082d9e68ecd7db3ff87b19f4232a5991d2/demos/session/src/main/java/androidx/media3/demo/session/PlayableFolderActivity.kt#L161

with

subItemMediaList.addAll(List(500) { children.get(0) })

As you can imagine, this code change only provokes a large playlist.

First select any audio file in the album, artist or genre folder. For example: Artist Folder -> The Kyoto Connection -> click on Intro - The Way ..

You should see the UI lag when you try to scroll in the list immediately after clicking an item in the bottom list view in the PlayerActivity.

We also found out that switching between two MediaItems which have been played before does not produces any lags. So maybe the UI issues are related to ongoing network calls.

For us, the UI lags are visible on the following phones, however, the bug is "better" visible on the Samsung S9:

• Samsung S9 (Android 11)
• OnePlus 6 (Android 11)

media3 Version: 1.0.0-alpha03

Please let us know if you have any further questions or if there is anything we can do to support you in finding the cause. Thanks for building media3!

Julian

[juliankotrba]

Maybe related to https://github.com/androidx/media/issues/57

[marcbaechinger]

Thanks for reporting!

As you discovered this is a known issue. The time isn't spend with networking I think but with bundling/unbundling the MediaItems when they are sent over the binder to the service and then back to the activity.

As a first step I'd recommend to only send MediaItems that contain the mediaId only from the activity. The service can then load the data for the whole Mediaitem and set it to the player. Please note that we just introduced an API change that makes it possible to load the data for all the media items asynchronously before setting the playlist and preparing the player. I recommend to wait until we released the 1.0.0-beta001 that will be released within the next two weeks.

I think you will run into similar problems with 1.0.0-beta01 though. So if you want to work around the blocked UI you need, as of now, probably to add the media items in batches - like 50 at a time, wait for the onTimelineChange() event and then add the next 50 media items.

We will try to improve this at some point and we acknowledge that this is slow currently when adding 100+ items.

I wonder, as a last thing, if this is a regression and how the performance for these operations was with androix.media (MediaControllerCompat). Did this work smoothly with 600 items in the legacy MediaSessionCompat world?

[juliankotrba]

Thanks for the reply and the implementation tipps to work around the performance problem @marcbaechinger!

We are planning to implement the workaround, however, I am not exactly sure if I understand correctly how to implement it. Let's say we have 500 playlist items. The user clicks on the first item in the recycler view and then scrolls all to the bottom of the list to plays the last playlist item. Do we have to add the 9 batches (450 items) after the onTimelineChange() was called from the first already playing item? If so, would this fix the performance problem because the lag always happens when changing playlist items and at this point we would also have all 500 items in the playlist.

Thanks for your time and also thanks for the information about the first beta release, really looking forward to it.

PS: As far as I can tell, the legacy implementation works super smooth with the same big playlist we are currently having "trouble" with.

[marcbaechinger]

Yeah, you are right. I played around with such use cases some more yesterday and can confirm your findings.

My comment above was more about the addition of 500 media items to the playlist at once. But I see and agree to your point that this happens not only when adding items but also when selecting another current media item. This triggers sending over a new PlayerInfo object from the service to the activity that includes the timeline with 500 items. So my advice from above is not helpful for that case and it seems it also happens when just sending the PlayerInfo from the service to the controller after selecting a new current item.

On a Pixel 6 I can see the lagging UI in some cases, but in some cases it is very smooth also. I see that the phone unbundles PlayerInfo object with 500 items in around 60 ms. This is IMO not enough to cause the lagging UI that I'm seeing that lasts for probably 2 seconds until it gets smooth again. For the case of the demo app some improvements in the UI may help a bit.

So, probably my comment above was a bit a shot from the hip. Sorry for that. I'm currently profiling this a bit to see where the time is actually spent. Will comment here once I found something useful.

(thanks for the confirming it is smooth with the legacy implementation)

[juliankotrba]

No worries and thanks for your support! I will also let you know if we find out anything more.

[XuQK]

This problem does exist, it's been bothering me for two days, I set 1500 mediaItems, when I seekToPreviousItem, the androidx.media3.session.MediaSessionImpl.dispatchOnPlayerInfoChanged() was called a dozen times, The more time-consuming over 400ms, and the less is tens of milliseconds, block UI about 2-3s，always.

[rohitjoins]

Hi All,

We have reduced the time required to bundle and unbundle MediaItems, which has reduced the latency to perform actions on large playlists.

These updates have already been published to the main branch and will be included in the release branch during our upcoming release cycle. Please let us know if you continue to experience significant UI latency when using large playlists.

[AfigAliyev]

@rohitjoins,

Hi, Rohit. I confirm that the issue is still present in the stable version.

[rohitjoins]

@MaximillianLeonov Could you provide further details regarding your situation that would assist us in replicating the problem on our end? For instance, could you inform us about the number of items present in the playlist initially and the delay you're encountering while attempting to play or scroll through the playlist?

[AfigAliyev]

@rohitjoins,

You can find the repository for reproducing the issue here.

I encountered this problem with 500-1000 songs. Although the issue is less noticeable in this repository with only 500-1000 songs, so I set 10,000 songs to make the problem more clearly visible.

Reproduction video

[rohitjoins]

@MaximillianLeonov,

To clarify our optimisations, we have reduced the time it takes to bundle and unbundle playlists, ensuring that the user interface (UI) remains responsive during common use cases, typically a few hundred to thousand items. However, the latency still depends on the length of the playlist and the capabilities of the phone being used.

I conducted a test on a Pixel 4A by playing a playlist containing 1000 items, and it performed smoothly without any issues. Even for larger playlists with 10,000 items, the lag was significantly reduced, and the UI did not freeze. However, if you continue to increase the playlist size to 100,000 items, it can still cause the UI to freeze. On a more advanced phone, such as the Pixel 6 Pro, the experience is smooth even when dealing with playlists of 10,000 items.

Can you please share the device you're using to experience the lag with 500-1000 songs?

[AfigAliyev]

@rohitjoins,

Regarding the devices I experienced lag on, they are the Samsung Galaxy A23 and Xiaomi Redmi Note 8.

In addition, I have noticed that the UI freezes specifically when the mediaController.prepare() command is executed. If I remove this line of code, the UI no longer freezes, but the media does not start playing (intended behavior).

[rohitjoins]

@MaximillianLeonov,

mediaController.prepare() is called to update PlayerInfo and is a necessary call when mediaItem is changed.

Although I did not have access to the mentioned devices for testing, I tested the demo session app on a Samsung Galaxy A51 and Xiaomi Redmi Note 9 Pro. During the testing, I did not observe any freezing of the UI or noticeable delays when the playlist contained ~1000 items. However, when the playlist size was increased to 10000 items, substantial UI lags were encountered.

[svenoaks]

Since I was rewriting a large portion of my old media playing app to reduce laggy UI and ANR's, I have not been too thrilled to discover that Media3 introduces some new problems when using large playlists.

I'm wondering if there was a way for the library to detect if the MediaController(s) is in-process and avoid the binder calls if that is the case? That would completely eliminate the lag I believe. Something like how there was a LocalBroadcastManager to prevent the same kind of thing with BroadcastReceivers back in the day.

[tonihei]

@svenoaks Local binder calls should not be a bottleneck at all because they automatically avoid the inter-process communication protocol (=what I think you mean by binder calls) and instead call the receiving method directly. That is, the data being sent is not going through any process or threading jumps and doesn't need to be marshalled and unmarshalled to/from a shared memory space (see first bullet point in this guide: https://developer.android.com/guide/components/aidl)

[svenoaks]

@tonihei Thanks for the clarification. Since I'm using SimpleBasePlayer, would calling the constructor with something other than Looper.getMainLooper() effectively solve this for me by moving the bundling / unbundling off the main thread?

[tonihei]

@rohitjoins probably knows this a bit better, but if my understanding is correct, the blocking happens on the reading side (=the MediaController) while it's waiting for all the data to be transferred. So I think using a different thread for your player on the MediaSession process won't help?

[naveensingh]

@marcbaechinger @rohitjoins @tonihei

Hi!

I hope this gets your attention.

Context: I have been working on rewriting https://github.com/SimpleMobileTools/Simple-Music-Player to media3 to take advantage of the features of media3 and delegate some of the responsibilities to the library instead of doing everything ourselves. The app initially used android MediaPlayer in a background service and intents for all UI<->Service communication.

Original MediaPlayer performance:

I did some tests using 10000 items and putting aside the time it takes to initially prepare the data and player (usually 0-10 seconds), the player itself was superfast. No UI lags (obviously because all potentially time-consuming operations were done on a background thread), No delays, nada.

Media3 performance: It works smoothly up to a certain number of media items but after that, media3 is just unusable. I'm facing significant UI lags, ANRs, and unacceptable delays in player responsiveness. I'm already using ExoPlayer.setLooper() and MediaController.setLooper() to avoid overloading the main thread.

Initially, I thought this issue was an edge case, and not many people have that many media items but looking at the reports from Google Play and on GitHub, the number of users with large playlists turned out to be common enough to warrant a proper solution.

As a first step I'd recommend to only send MediaItems that contain the mediaId only from the activity. The service can then load the data for the whole Mediaitem and set it to the player.

@marcbaechinger this really helped reduce UI lags, however, I'm still getting ANRs with large playlists (only when the media controller is involved). I do not know how things work internally but why can't we move this media controller work to a background thread? The existing MediaController.setLooper() does not seem to help offload this work to the background.

To clarify our optimisations, we have reduced the time it takes to bundle and unbundle playlists, ensuring that the user interface (UI) remains responsive during common use cases, typically a few hundred to thousand items. However, if you continue to increase the playlist size to 100,000 items, it can still cause the UI to freeze.

This bundling/unbundling process is still slow and it blocks the main thread enough to cause a crash or ANR. Doing this on the main thread is a bad idea/design.

However, when the playlist size was increased to 10,000 items, substantial UI lags were encountered

@rohitjoins even after applying the @marcbaechinger's media ID hack (let's call it), the player is slow enough to not be usable and this has nothing to do with overloaded main thread. How to reproduce:

1. Prepare the player with ~10,000 media items on any device with reasonable specs.
2. Start playing.
3. Hit next/previous a couple of times (doesn't matter if you do it through player notification, player UI or using android auto)
4. Wait.

It takes like 30-60 seconds before the player resumes playing again. The observed delay is directly proportional to the number of media items. There's a significant delay even with 5000 media items. I believe this is the same issue as https://github.com/androidx/media/issues/592#issuecomment-1688376202 and https://github.com/androidx/media/issues/402

I tried playing around with the DefaultLoadControl parameters thinking unnecessary buffering is the issue but I did not see any improvements.

To summarise:

1. Please make MediaController do all blocking operations on a background thread (because the current behavior leads to inevitable crashes and ANRs on many devices and the media ID hack is not enough to fix it)
2. Is there any way to customize ExoPlayer so it doesn't do whatever unnecessary things it's doing that's causing this delay during the next/previous/play/pause commands? If ExoPlayer was not intended to be used with a large number of media items, can we please get an alternative Player implementation that is simpler and designed or at least optimized for audio playback?

thanks!

[tonihei]

Thanks for the thorough reply - that's useful to hear and we definitely know that large playlists continue to be problematic.

Let me summarize some of things we've done, discussed and are planning to do.

There are two main data flows to look at:

1. Setting a large number of new items on a MediaController (this is technically tracked by #592). This involves (a) App code to generate these items (b) MediaController to MediaSession Binder communication (c) Player code to handle and prepare the playlist (e.g. ExoPlayer)
2. Sharing the current playlist with controllers (tracked by this issue, #81). This involves: (d) MediaSession to MediaController Binder communication (e) MediaSession to legacy MediaSessionCompat compatibility layer

Note that all these steps need to handle the large number of items somehow and to improve the performance, there are two options: optimize the operation or avoid the operation as much as possible.

What we've done so far

• Hugely optimized the time for (b) and (d) by reducing the Bundle operations required for individual items. (see this comment: https://github.com/androidx/media/issues/81#issuecomment-1378747452)
• Avoided sending updated for (d) when nothing changed.
• Highlighted efficient app setup pattern for large playlists (step (b)), in particular optimizing the calls by only setting mediaId values and avoiding calls by suggesting to incrementally send updates instead of all at once. (see #592)
• Further internal analysis on how the Binder communication influences the performance. Part of this highlighted that local Binder calls in particular are not as optimized as they should be (see my comment here, that is not entirely true in hindsight: https://github.com/androidx/media/issues/81#issuecomment-1671822737)

What we are planning to do next

• Some local binder improvements for (b) and (d). This only helps for controller<->session communication within the same process, but should noticeably improve some of the cases where you use a controller in the same app (that is likely to run in the same process). This will be part of release 1.2.0-rc01.

For later releases:

• A new setting that restricts the number of items shared with legacy session (e) and MediaController instances in other processed (d). This avoids the issue by simply shortening the playlist. This is also a loss of functionality for remote Media3 controllers. For legacy controllers it never worked anyway.
• Send incremental updates to Media3 controllers (d) to re-add the functionality we may remove with the step above.
• a "paging" Player wrapper implementation that let's you handle large playlists in the session and UI, but only sets a subset on the actual player. This helps with step (c) for all types of players (not just ExoPlayer).

What we are not planning to do

• Any improvements for (a). This outside of our control and needs to be handled by your app directly. You can use similar ideas of course: load items in batches, generate incremental updates where needed, etc.
• Use a new background thread for any processing. This may sound counter-intuitive, but it really wouldn't help.
  • Most of the operations that may take time in MediaSession, MediaController and ExoPlayer are already on a background thread anyway.
  • Moving further updates like the Binder communication to a background thread would likely only help for local in-process calls, which are already optimized by the planned steps above. (The remote cross-process calls are already on background threads)
  • Another reason not to do this is that it won't help with the user-visible delay in an app. Even if you use another background thread to transmit/receive data, the user still needs to wait until this is finished to see the effect in the UI. It also hugely complicates the correctness and ordering of other operations happening around the same time.
  • It's likely more beneficial to work on reduced updates or incremental updates as explained above, as these can be done quickly and result in a responsive UI.

---

Some comments on your concrete proposals from the comment above:

Hit next/previous a couple of times (doesn't matter if you do it through player notification, player UI or using android auto)... It takes like 30-60 seconds before the player resumes playing again.

This sounds strange, but may be related to a huge backlog of work generated by these actions. We'll have a look at this case to see if there is something else going on.

Please make MediaController do all blocking operations on a background thread because the current behavior leads to inevitable crashes and ANRs on many devices

ANRs can be caused by any of the steps mentioned above if they run within one main thread iteration. We'll submit some improvements, and you can also check in your app code that it is as efficient as possible and not handling too many items at once if it can be avoided. The code in MediaController in particular is not the slow part (and also not blocking). And as explained above, we are also not planning to consider any further background threads because we don't see a case where they would help further.

Is there any way to customize ExoPlayer so it doesn't do whatever unnecessary things it's doing that's causing this delay during the next/previous/play/pause commands?

The delay is likely not ExoPlayer related, but in the session/controller communication as said above. We'll check your reproduction steps though, just in case there is some additional problem here.

Original MediaPlayer performance

This comparison is not really useful because this setup neither has a player that supports playlists nor a session that knows and shares all the items. So it's fast by removing all the functionality and avoiding all the work.

[naveensingh]

@tonihei thanks for the much-needed clarifications!

Blocked UI when using large playlists (pretty much resolved)

Highlighted efficient app setup pattern for large playlists (step (b)), in particular optimizing the calls by only setting mediaId values and avoiding calls by suggesting to incrementally send updates instead of all at once. (see https://github.com/androidx/media/issues/592)

Only setting the mediaId definitely helped reduce UI lag significantly, before this the app was almost unusable with a large number of items. I should mention that once the media items are set and the player is prepared/playing, I'm still getting some slowness on the UI side (not to mention ANRs) but I'm not yet sure about the exact cause (no issues if the media items aren't set).

I did some tests with ~10000 items and after calling MediaController.setMediaItems(), most of the time MediaSession's onSetMediaItems() is called within 5-20 seconds (guess it's acceptable(?) for such playlists).

Could you please provide some guidance on how to send incremental updates using a media controller? A simple mediaItems.chunked(500).forEach { controller.addMediaItems(it) } doesn't seem to work for me. Are we supposed to set up a Player.Listener (on the controller side) and send each chunk only after onTimelineChanged() is called?

I'm curious, does sending incremental updates help ExoPlayer prepare faster too or is it just for fast MediaController=>MediaSession communication?

Unresponsive/slow player

Hit next/previous a couple of times (doesn't matter if you do it through player notification, player UI or using android auto)... It >>takes like 30-60 seconds before the player resumes playing again.

This sounds strange, but may be related to a huge backlog of work generated by these actions. We'll have a look at this case to >see if there is something else going on.

The delay is likely not ExoPlayer related, but in the session/controller communication as said above. We'll check your reproduction steps though, just in case there is some additional problem here.

When this happens MediaSession => MediaController updates are delayed as well. The Player.Listener callbacks on the MediaSession side are also delayed so it's probably caused by the player taking too long to prepare with large playlists. I hope this will be fixed by the planned improvements.

Not sure if it's relevant but I was looking at the memory profiler to reduce memory usage and I noticed that when I do next or previous multiple times, the memory usage spikes up and even leads to an OOM if I keep doing it:

My current workaround is to throttle the seekToNext()/seekToPrevious() calls in a forwarding player but it doesn't work very reliably since the player is slow in general when the playlist is huge.

A "paging" Player wrapper (deserves to be highlighted)

"paging" Player wrapper implementation that let's you handle large playlists in the session and UI, but only sets a subset on the actual player. This helps with step (c) for all types of players (not just ExoPlayer).

That is a great idea. If implemented, I think it will solve most if not all of these problems (other optimizations are nice too).

[tonihei]

A simple mediaItems.chunked(500).forEach { controller.addMediaItems(it) } doesn't seem to work for me. Are we supposed to set up a Player.Listener (on the controller side) and send each chunk only after onTimelineChanged() is called?

Yes, simply calling it multiple times in a loop likely makes no big difference I'd say. You'd need to wait with a timed delay or for the callback as you suggested. This isn't super easy to achieve of course and only helps with setting new items from your code (data flow 1 in my post above). It doesn't help with data flow 2 (sending updates to controllers) as this needs to be solved inside our library.

I'm curious, does sending incremental updates help ExoPlayer prepare faster too or is it just for fast MediaController=>MediaSession communication?

Should also help with the player preparation (see "paging" player idea that relies on a similar pattern)

that when I do next or previous multiple times, the memory usage spikes up

Probably caused by the pending tasks kept in memory. This should resolve itself by reducing overall delays (as we then don't keep a long lists of pending tasks anymore).

[naveensingh]

OK, thanks for the help.

Just in case you guys need it, I'm attaching two large CPU traces (compressed) below to demonstrate the main thread getting blocked by the media controller's setMediaItems() for longer durations (leading to ANRs). Recorded when only mediaId is set on the media items but no incremental updates are done.

Since Github doesn't allow large files, here's a Google Drive link: https://drive.google.com/file/d/1XUdtA5usxPSkhW5HFZZQec-3L5zdvQMd/view?usp=sharing

For anyone interested in the 'slow next/previous' workaround, click to expand

```kotlin /** * This is here so the player can quickly seek next/previous without doing too much work. * It probably won't be needed once https://github.com/androidx/media/issues/81 is resolved. */ private fun seekWithDelay() { seekJob?.cancel() seekJob = scope.launch { delay(timeMillis = 400) val seekCount = seekToNextCount - seekToPreviousCount if (seekCount != 0) { seekByCount(seekCount) } } } private fun seekByCount(seekCount: Int) { runOnPlayerThread { if (currentMediaItem == null) { return@runOnPlayerThread } val currentIndex = currentMediaItemIndex val seekIndex = if (shuffleModeEnabled) { val shuffledIndex = shuffledMediaItemsIndices.indexOf(currentIndex) val seekIndex = rotateIndex(shuffledIndex + seekCount) shuffledMediaItemsIndices.getOrNull(seekIndex) ?: return@runOnPlayerThread } else { rotateIndex(currentIndex + seekCount) } seekTo(seekIndex, 0) seekToNextCount = 0 seekToPreviousCount = 0 } } private fun rotateIndex(index: Int): Int { val count = mediaItemCount return (index % count + count) % count } ```

[svenoaks]

I am commenting here because large playlists are still a major concern using 1.1.1 or 1.2.0-rc01. I have rolled out to 5% of my user base with no compensation (besides only setting the mediaId for MediaItems) for the fact that media3 library is still badly lagged with large playlists and have already received a few negative reviews mentioning it.

I can reproduce all of the things mentioned in this issue, probably the lagging prev/next buttons are the worst, as there can be a huge backlog of events that can take minutes to play out, but even some kind of debouncing of them would not fix all that lagging and blocking of the main thread with large playlists. (Edit: most of the next/prev lag was due to app code)

For now the only solution for me seems to be limit the size of the playing queue to 1000 items or so unless the user specifically adds more. I hope that that the "paging" player idea can be implemented quickly on the library side as this seems like the only real good solution.

[svenoaks]

I ending up implementing a solution where my SimpleBasePlayer implementation only ever sets a playlist size of up to 10 items, and my app handles most manipulation of the actual playlist internally, overrides like handleSetMediaItems are no-ops now. The only flaw I noticed with this setup in my case was a small flickering of the track title when the SimpleBasePlayer predicated state does not line up with the actual result state, a rare occurrence.

As mentioned in the previous edit, the real major lags with prev/next buttons was caused by app code. However, with the large playlists set on the player many mediaController events caused noticeable jank, even where the playlist did not change. With the limiting of the playlist that the player to 10, all the jank has been removed.

[tonihei]

Thanks for sharing! The pending improvements I mentioned above should result in a similar behavior out of the box so that no special workaround is required on app side.