Support for thumbnail scrubbing in 1.3.0 ?

[sobbe81]

Hi

Release note for 1.3.0-beta01 suggests that there are support for DASH thumbnails.

Image:
Add support for DASH thumbnails. Grid images are cropped and individual thumbnails are provided to ImageOutput close to their presentation times.

Have I understood it correctly that you set an ImageOutput using exoplayer.setImageOutput(ImageOutput imageOutput) and that I would get callbacks to onImageAvailable(long presentationTimeUs, Bitmap bitmap) during playback?

I tried this but I do not get any callbacks and from what I could tell the ImageRenderer is not enabled. I probably have done something wrong in my setup. Do I need to do some track selection in our application to enable it? Do you have any sample code on how to do this?

[microkatz]

@sobbe81

Thank you for posting your question! Yes, 1.3.0 provided support for DASH thumbnails. If you provide an ImageOutput class then you should get callbacks to onImageAvailable during playback.

It would help if you provided the example content you are trying to play. However, if I am to guess, perhaps the media content you are trying to play has a video track as well? ExoPlayer currently will not select both a video and image track at the same time. If you want the player to select and play the image track over the video track, then you need to use the isPrioritizeImageOverVideoEnabled set through TrackSelectionParameters(https://github.com/androidx/media/blob/1.3.0-beta01/libraries/common/src/main/java/androidx/media3/common/TrackSelectionParameters.java).

[sobbe81]

Hi,

Yes, you are correct. The content has both a video and an image track. The use case we are trying to solve is to support is to show a thumbnail in our custom player ui when scrubbing.

Sounds like a lot of underlying support is already implemented to support DASH thumbnails. How difficult would it be to expose an api that returns an image / url for a given time similar to Shaka player's getThumbnails(trackId, time)?

[microkatz]

@sobbe81

Great! Then the TrackSelectionParameters#isPrioritizeImageOverVideoEnabled API should work for you.

Would you be able to elaborate on the expected callbacks or functionality you are expecting? I'm not sure how that is different to setting a thumbnail content url on the player and then seeking to the time that you want?

[sobbe81]

Hi,

If I set TrackSelectionParameters#isPrioritizeImageOverVideoEnabled I do get a callback to onImageAvailable(). However, as I think you pointed out earlier you will not get any video rendered instead.

What I'm looking for is a way to do this

So, I would like to be able to play the video and when the user moves the scrub head I would also like present a thumbnail image that corresponds to the position of the scrub head.

Basically, I would like to ask Exoplayer for an image (or potentially a url to an image) for a given time and then display that in my Player UI.

Even if onImageAvailable() would provide callbacks with the video playing it would not really help me since I would get callbacks for the playback position and not for the position of the scrub head.

[sobbe81]

Due to DRM/Encryption the screenshot displays a black image. In reality, the actual video is playing.

[microkatz]

Would it work for your application if you were to use two Players? One for the video playback and then one for scrubbing?

[sobbe81]

Potentially as a test, it's feels like a hack rather than a stable and maintainable solution.

Say that we would go with a specific player for the images. As I understand it, it's design to use to render an image that corresponds to the playback position. In my case, I would like to get an image for any given position. How would that player know for which timestamp to emit a callback for?

[microkatz]

Actually I would say it would be the preferred solution. If you had a single player playing both the video, audio for the main streaming and also the scrubbing thumbnails then how would you receive thumbnails from a timestamp different to the current playback position?

If you want to get an image for a given position then for the thumbnail image player you just seek to the position you want(ex: the timestamp the user is scrubbing over) and you'll get a callback for the position and image that you want.

If you are trying to play both the thumbnail player and playback player in sync, then just set the thumbnail player's clock to the playback player's and they should play in sync. There are a number of Github threads about multi-player syncing as well.

[sobbe81]

Ok, we'll give it a try.

[microkatz]

Great! Hope that works for you. I'm going to close this issue. Please reopen or file a new post if you have any additional problems or questions.

[prashantchothani]

@microkatz We are facing the same issue even in the non DRM videos. Only the Image Track is playing and Video Track is not visible. Would it be possible if there could be some more documentation or ideally a basic example in Kotlin for us to understand how to get this working ?

[hammar83]

@microkatz We have tested creating two separate players now, one solely for handling the image track and one for video. It works to the extent that we can display thumbnails for the current position, but as soon as you scrub a bit faster it becomes inefficient and the images do not load in time. It feels like some sort of image prefetching would be nice here - but I can't think of a good way to do it with this setup. Do you have any thoughts on how to make image loading more efficient and smooth?

[prashantchothani]

@hammar83 Is it possible to share the code ?

[hammar83]

@prashantchothani This is basically what we have done now during testing, we created another ExoPlayer instance with setPrioritizeImageOverVideoEnabled set to true, something like this:

val thumbPlayer = ExoPlayer.Builder(context)
            .setTrackSelector(
                DefaultTrackSelector(context).apply {
                    setParameters(
                        buildUponParameters()
                            .setPrioritizeImageOverVideoEnabled(true)
                            .build()
                    )
                }
            )
            ...

And on the original player we have a scrub listener that, onScrub, make the thumbPlayer seek to a new position: override fun scrubTo(position: Long) { thumbPlayer.seekTo(position) }

And finally we set the image output listener to receive the loaded bitmap and display it on screen: thumbPlayer.setImageOutput(imageOutput)

[prashantchothani]

@hammar83 Is there no other way than creating another ExoPlayer instance ?

[hammar83]

@prashantchothani It's according to @microkatz recommendation/suggestion above.

[prashantchothani]

@hammar83 I think @microkatz is suggesting that if 2 player solution would work. But I guess this is not the right approach. Before media3 version 1.3 support for thumbnails, we were using as mentioned in this example : https://github.com/kaltura/kaltura-player-android-samples/tree/master/AdvancedSamples/MediaPreviewSample Using 2 players is an overkill and not advised in my view. @microkatz can you please help with an example with a single exoplayer ?

[tonihei]

But I guess this is not the right approach

I'm curious why you think it's not the right approach. Could you let us know what your main worry is?

Our motivation to recommend a second player is that showing thumbnails coming from a DASH manifest requires a lot of the same code paths as normal playback (media loading, manifest parsing, image extraction etc). Usually, you also want to enable the user to scrub independently of your main playback, so it's not easy to use the same player either because you'd have to present two tracks at two different positions. So our conclusion was that using a second player that only plays the thumbnail images and be controlled independently is the right approach.

[prashantchothani]

@tonihei handling both player instances with their events, mounting and disposing off, memory allocations, etc. would be a pain. Syncing both the players is also another problem. It would be better to have the scrubbing load the image track above the video track just like we can see in Youtube or any other player. I have shared an example of the Kaltura player in the above link. On scrubbing the video track is paused and Image track moves forward and on rlease, the player starts from there.

Ideally the media3 UI controller should have this feature available out of the box, hopefully in coming versions where it can have 3 ways to handle this: None (no image track), Single View (this will display a single image over the seekbar of that location), Track View (this will show the the multiple images). This can be used on the TV to move forward and backward, and the focus can be on the centre image. In mobile as well it will look like Youtube Player where the images are shown below the track bar. It can also have a property of placement : above or below the seekbar

[tonihei]

Thanks for sharing your thoughts! This is useful for us to understand how we can integrate better support in the future and which areas need better documentation.

Ideally the media3 UI controller should have this feature available out of the box

We definitely want to get there, we just need more infrastructure on the player side to make it more easily configurable.

handling both player instances with their events, mounting and disposing off, memory allocations, etc. would be a pain.

• The memory allocation would be the same overall since you need to load both video and thumbnails in any case. The only thing that may be loaded twice is the DASH manifest, and that could be avoided by using a CacheDataSource if needed.
• Creating and releasing a player can presumably happen in the same places where you create and release your main player?
• Event handling is interesting. I assume if you are not interested in events, you could just not add a listener. But it's more likely you actually want a separate listener so you can react to specific changes in the thumbnail track without affecting your main playback? Acknowledged that this may cause more overhead in your app.

On scrubbing the video track is paused and Image track moves forward and on release, the player starts from there.

If you never want to enable scrubbing while the main playback is ongoing, there is an alternative usage model with a single player only:

• Once scrubbing starts, pause the main player, remove the main surface from the player, change track selection to prefer image tracks and show the thumbnail output. Any future seek output will appear in the thumbnail view only.
• Once you are done scrubbing, hide the thumbnail view, change track selection to prefer video again and re-add the main surface to the player. Then continue playback.

One downside of this model is that you can't easily independently preload the thumbnail track into memory. (You could download the track in advance though).

[prashantchothani]

Hi @tonihei, thanks for explaining so much in detail. Instead of having to use another player instance, can we instead build a custom PlayerView and pass the Image Output to it ? This would help us to customize the way we want and be more efficient ? Possible to try this ?

[hammar83]

@tonihei Do you have an example of how preloading the thumbnail track could easily be done?

[tonihei]

can we instead build a custom PlayerView and pass the Image Output to it

We are intending to provide support for image output in PlayerView out of the box, so there won't be a need to customize this.

Do you have an example of how preloading the thumbnail track could easily be done?

When you set up a separate player, you can configure it to have a large forward and back buffer in DefaultLoadControl to keep all the thumbnails in memory. As soon as you prepare the player, it will preload the data, which can be done at any point before playback if needed. Alternatively, you can also look into caching and downloading functionality to preload the thumbnails to disk.

[prashantchothani]

We are intending to provide support for image output in PlayerView out of the box, so there won't be a need to customize this.

Could you please estimage as when do you think this would be available @tonihei ?

[tonihei]

We generally don't provide estimates for features to avoid disappointment when we have to move them around at short notice. That said, we have the intention to work on this within this quarter, but I'm not sure yet whether it will be in Media3 1.4.0 or later.