[Cast] Convert MediaItem.playbackProperties.subtitles to MediaTracks

[gsavvid]

Previously, I would set the metadata like this:

val movieMetadata = MediaMetadata(MediaMetadata.MEDIA_TYPE_MOVIE)
movieMetadata.putString(MediaMetadata.KEY_TITLE, title)
movieMetadata.putString(MediaMetadata.KEY_SUBTITLE, subtitle)
movieMetadata.addImage(WebImage(Uri.parse(posterImage)))

val mediaInfoBuilder = MediaInfo.Builder(media.mediaUrl)
       .setStreamType(MediaInfo.STREAM_TYPE_BUFFERED)
       .setMetadata(movieMetadata)

player.loadItem(MediaQueueItem.Builder(mediaInfoBuilder.build()).build(), TimeUnit.SECONDS.toMillis(position))

Where MediaMetadata in the above example is com.google.android.gms.cast.MediaMetadata. Now that CastPlayer.loadItem() is deprecated, I want to replace it with Player.setMediaItem() but I can't find how to pass the same metadata to a MediaItem. I know that it accepts com.google.android.exoplayer2.MediaMetadata but this currently only has a title property.

So my question is how can I set the title, subtitle, and image to be displayed in the cast receiver app using a MediaItem?

[krocard]

Looking into the cast player extension, the non deprecated method setMediaItem converts the MediaItem to MediaInfo internally with with: https://github.com/google/ExoPlayer/blob/b1000940eaec9e1202d9abf341a48a58b728053f/extensions/cast/src/main/java/com/google/android/exoplayer2/ext/cast/DefaultMediaItemConverter.java#L59-L70

This seems to imply that all other metadata than the title are dropped during the conversion. @marcbaechinger, could you confirm? If so, should @gsavvid continue to use loadItem even if deprecated?

@Samrobbo, are the new metadata fields planned contain subtitle, MEDIA_TYPE and image ? (Side note, having a MEDIA_TYPE metadata is awfully similar to AudioAttribute's contentType, we might consider merging them).

[Samrobbo]

The new metadata fields will cover a large selection of things yes. The initial change will be fairly minimal, but it will be quick and easy to expand it, and fields like those listed are common ones that will no doubt be implemented early on.

[krocard]

Thanks Sam. @gsavvid, until the new Metadata field are added, please continue to use the deprecated player.loadItem. I'm afraid your use-case was missed when we deprecated loadItem.

[marcbaechinger]

I don't think that use cases were missed, but as a library we can not know how an app want to pass arbitrary metadata. An app can add this behaviour though.

With new MediaItem.Builder().setTag(customObject) and app can set an arbitrary object as the tag. This could be a data object that includes metadata as strings or bitmaps or whatever is needed for your use case.

The CastPlayer then has a constructor that allows injection of a custom MediaItemConverter. You app can then convert that mediaItem with the custom tag into a MediaQueueItem from the cast API.

It's true that the MediaItem of ExoPlayer does not support meta data yet, and even if it does once, developers probably want or doesn't want to map this to the Cast item or override the metadata in the media with something else. For this purpose the converter approach has a default that just does the minimum but allows apps to customize this conversion.

Does this cover you use case @gsavvid?

[gsavvid]

It's true that you can't know how an app wants to pass arbitrary metadata but I'd expect that the metadata used by the Styled Media Receiver app would be supported.

Nevertheless, I ended up using the solution with the custom MediaItemConverter mentioned above by @marcbaechinger for another reason: the DefaultMediaItemConverter ignores the subtitles defined in the MediaItem so I wrote an implementation that takes care of that, too.

[marcbaechinger]

Thanks @gsavvid! Marking as an enhancement for now.

I agree we could probably support the styled media receiver (context: https://developers.google.com/cast/docs/styled_receiver) better than we currently do. I think it uses MediaMetadata to get the data to display. We provide what we have like mediaItem.mediaMetadata.title to mediaQueueItem.metadata.putString(KEY_TITLE, title). That's not much, I agree, but that's already opinionated, and not useful for all apps. Because, we need to decide at the very begin what type of media it is. The current default converter currently blindly assumes it is of MEDIA_TYPE_MOVIE.

MediaMetadata metadata = new MediaMetadata(MediaMetadata.MEDIA_TYPE_MOVIE);

Just some notes around what was the thinking when basic support was added for this with the converter: apart from the title, doing it right could turn out more complex than it seems. That's the reason why it's currently only having very rudimentary support (only the title).

The problem to solve is for example, how to map metadata keys from MediaMediadataCompat to ExoPlayer's MediaItem to Cast's MediaQueueItem.MediaInfo. This is not clear just per see (do we also support BOOK_TITLE? just as a starter :).

Further, for a really nice solution we probably want to include in-band metadata that is for instance ID3 tags read from the media and also the duration which can only be properly determined after the media stream is read. We need to take into account that some of this data is available only when the media is prepared or even the media period is created for that source - so some time after the media item is built and added to the playlist. Like at the moment the user adds a list of MediaItems to the ExoPlayer, the data is not complete but we want to populate the Cast queue already. So at any given time we may want to merge app provided data with the data read from the media. For the case of artworks, even app provided data is provided in two steps. String like the title is available after the app read if from eg. a database, but artworks need to be loaded asynchronously and we currently do not have a way to update the media item once it is passed to the player. And then later when the media is prepared and played we probably also receive in-band metadata from ID3 tags.

We also probably want to have this working in both directions like #8212 suggests. So when the item is added to the cast queue on the cast device by another cast sender, it should be properly land in Androids media session like as if the user had added it to the ExoPlayer.

Marked as an enhancement for now.

[marcbaechinger]

@gsavvid beside my general comment above

the DefaultMediaItemConverter ignores the subtitles defined in the MediaItem

Do you mean the title or the subtitle? We only have a title currently which we support here:

https://github.com/google/ExoPlayer/blob/release-v2/extensions/cast/src/main/java/com/google/android/exoplayer2/ext/cast/DefaultMediaItemConverter.java#L61

What d o you refer to when mentioning the subtitle?

[gsavvid]

Oh yes, sorry I should have been more specific regarding the subtitles. I was referring to the subtitle tracks. These are not added by default to the MediaQueueItem. Here's my implementation that adds the subtitle tracks:

internal class EnhancedMediaItemConverter(private val defaultConverter: DefaultMediaItemConverter = DefaultMediaItemConverter()) : MediaItemConverter {

    override fun toMediaItem(item: MediaQueueItem): MediaItem {
        return defaultConverter.toMediaItem(item)
    }

    override fun toMediaQueueItem(item: MediaItem): MediaQueueItem {
        val movieMetadata = MediaMetadata(MediaMetadata.MEDIA_TYPE_MOVIE)
        movieMetadata.putString(MediaMetadata.KEY_TITLE, item.mediaMetadata.title)

        val mediaInfoBuilder = MediaInfo.Builder(item.playbackProperties!!.uri.toString())
            .setStreamType(MediaInfo.STREAM_TYPE_BUFFERED)
            .setContentType(item.playbackProperties!!.mimeType)
            .setMetadata(movieMetadata)

        item.playbackProperties?.subtitles?.let { subtitles ->
            mediaInfoBuilder.setMediaTracks(createSubtitleMediaTracks(subtitles))
        }

        return MediaQueueItem.Builder(mediaInfoBuilder.build()).build()
    }

    companion object {
        private fun createSubtitleMediaTracks(subtitles: List<MediaItem.Subtitle>): ArrayList<MediaTrack> {
            val subtitleMediaTracks = ArrayList<MediaTrack>()

            subtitles.forEachIndexed { index, subtitle ->
                val subtitleTrack = MediaTrack.Builder(index.toLong(), MediaTrack.TYPE_TEXT)
                    .setName(subtitle.label)
                    .setSubtype(MediaTrack.SUBTYPE_SUBTITLES)
                    .setContentId(subtitle.uri.toString())
                    .setLanguage(subtitle.label)
                    .build()

                subtitleMediaTracks.add(subtitleTrack)
            }

            return subtitleMediaTracks
        }
    }
}

[marcbaechinger]

Ha! After starring at the metadata keys for a while I got confused by MediaMetadata#KEY_SUBTITLE that the Styled Cast receiver probably is using. Didn't think about timed text subtitles :)

@gsavvid! I think subtitles should be the first thing we add to the DefaultMediaItemConverter. We need to figure out whether cast supports all types of subtitles that we do support (and vice versa). Thanks!

https://developers.google.com/cast/docs/android_sender/media_tracks

[ahadEW]

Any update? I implemented the EnhancedItemConverter still not able to see subtitles on castplayer.

[shooter7]

@marcbaechinger any update? I use 2.18.2 exoplayer version and still no subtitle in cast

[marcbaechinger]

No updates I'm afraid. You need to create your own MediaItemConverter to support subtitles.

[shooter7]

No updates I'm afraid. You need to create your own MediaItemConverter to support subtitles.

if you build MediaItemConverter support Subtitle can you share it?

I build one but still no show subtitle in cast MyMediaItemConverter.txt

[marcbaechinger]

I haven't tested this but did you look into the version posted above? https://github.com/google/ExoPlayer/issues/8669#issuecomment-792719685

[silverAndroid]

For anyone who stumbles on to this issue (since DefaultMediaItemConverter now expects custom data to be set), I created a Gist with the code on how i got subtitles to work (just a heads up, there's 2 files in that Gist)