CastPlayer: Update local MediaItem with data from MediaQueueItem (e.g. how to support custom fields sent from receiver)

[andybdahl]

We are a national streaming business trying to rewrite our ExoPlayer and Cast sender SDK specific implementations of playback and move to using the Media3 APIs instead to allow for more consistent code and test code for cast and local playback streaming.

Our setup is as follows:

• We have a custom cast receiver
• We have an android app with a custom cast controller (both for extended and minimized versions) using the RemoteMediaClient and CastContext APIs to listen for playback changes on the cast device.
• We are a flow TV streaming business and thus support streaming of programs currently playing on flow TV on local channels, and we have support for starting these streams over and little features like only allowing seeking to previously played positions in streams etc. This means we have some custom properties that the receiver sends to the cast sender app.
• We have multiple platform apps (iOS, Android, AndroidTV, AppleTV, Browser etc.), and thus our cast receiver is doing some logic like getting the stream URI, the drm license Uri and other things and broadcasting those to the apps to make sure we do not need to do these implementations on all our platforms and instead let the cast device do them. This is done by the receiver putting json data inside the customData field, and then our old implementation uses the RemoteMediaClient.Callback.onMetadataUpdated to read the values when they change during the stream.

My question is, is there a way for us to receive the customData that the Cast receiver sends out using only the Media3 APIs? inside the CastPlayer class the callback method we use is just empty. I have tried the different callback functions in the Player.Listener interface but it seems none of them exposes the data the cast device puts in the customData field.

[tonihei]

Thanks for the request. Could you clarify which customData field you mean exactly? I can see multiple different ones in the Cast SDK classes. If this is about the MediaQueueItem.customData, then you can already inject a custom MediaItemConverter in the CastPlayer constructor, which allows you to pass in custom data or read it back as needed. If it's about a different type of custom data, we may be able to provide similar injection points that let you customize the conversion to your needs.

[andybdahl]

Hi @tonihei, and thanks for the quick answer.

We are using the Cast SDKs playerManager.setMediaInformation function and setting the customData on the MediaInformation object.

This data does not seem to propagate anywhere in the Media3 APIs in my testing yesterday.

In my test I tried the MediaItemConverter, but it seems almost nothing of the data on the mediaInformation object in the playermanager makes its way to the MediaItemConverter, neither the title, subtitle, artwork or anything else. In our setup the mobile app launches the load request using only the ID for the media that should play, and then the Cast device itself fetches the metadata from our servers and puts it into the MediaInformation.metadata field, but it seemed in my test none of this data is propagated through to the MediaItemConverter when it changes neither.

The cast device changes the metadata to include some properties that are based on the position, so the data changes every couple seconds, but the CastPlayer.Listener interface methods are never called except when a MediaItem is set from the app, the things set on the cast device never seems to trigger any events in the CastPlayer, rendering us unable to use the API for anything other than starting the stream itself, and not for any UI updates etc. to keep the user informed on the position, playbackState etc.

[tonihei]

Thanks for the additional details. I'm not super familiar with the Cast SDK and which callbacks are meant to be triggered in this case, but I wonder if this callback is called when you change the media information on the Cast receiver. Do you have a local checkout of Media3 or debuggable version to verify this (or whether any of the callbacks are received when the change happens)? If there is a callback, and we just don't handle it, I guess it should be solvable by adding updateTimelineAndNotifyIfChanged(); to the callback implementation. If none of the callbacks are called, then this may be an issue of the Cast receiver or SDK implementation and you'd to check with them for more details.

[andybdahl]

Hi @tonihei , The callback you mentioned is the one we are currently using in our implementation before migration to media3, so that does work yes, and I was also surprised to see that that callback is not even used in the current media3 implementation, when I looked into why the media3 library did not propagate information from the receiver. I think your suggestion is worth trying out, although I cannot test it out as the only thing I can do locally as CastPlayer is a final class is copying the entire code including all the private classes used in the code.

Do you wanna try it out in a branch, or should I create a PR I can try out for myself instead?

[tonihei]

Great, that sounds promising. Creating a PR to try it out sounds useful as we can the merge it directly as well. Alternatively, if you just want to play around with the code, you can depend on Media3 locally as described here: https://github.com/androidx/media?tab=readme-ov-file#locally

[andybdahl]

I will get back to you when I have tried it, thanks for the quick reply's

[andybdahl]

I have set up the media library locally and have been debugging and trying the suggestion you had. It does not seem to work. The issue is that the MediaItem class from Media3 does not seem to contain any information from the customData field in the MediaInfo class from the gms.cast library, so none of that information is getting parsed into the MediaItem class used locally inside the castplayer.

I might want to parse the information and add it to the MediaItem.extras field perhaps, but I don't know if that is how the MediaItem is supposed to be used or not.

Do you have any tips or other suggestions?

[tonihei]

The issue is that the MediaItem class from Media3 does not seem to contain any information from the customData field in the MediaInfo class from the gms.cast library, so none of that information is getting parsed into the MediaItem class used locally inside the castplayer.

The information should be processed by the MediaItemConverter class I suggested initially. The MediaQueueItem from Cast should ideally contain any custom data you set in the Cast receiver and then the converter can put it into the matching field in MediaItem (I guess most of them can go into MediaItem.mediaMetadata). Sorry if this isn't very helpful because I have to guess around a lot without having the same setup to test it out... If you can confirm the RemoteMediaClient.Callback.onMetadataUpdated() is called, then calling updateTimelineAndNotifyIfChanged() should trigger all the conversion from MediaQueueItem with the MediaItemConverter. You probably need to step through the code with a debugger to confirm this is actually happening (e.g. set a breakpoint in DefaultMediaItemConverter and check if the data you are looking for is in the MediaQueueItem)

[andybdahl]

That would be ideal yes, but I can confirm that when the customData field changes, th onMetadataUpdated callback is invoked but calling theupdateInternalStateAndNotifyIfChanged after that does not trigger a call to the MediaItemConverters toMediaItem function unfortunately, it seems the library only fetches the mediaItem from the Timeline on the cast device and not the mediaItem from the remoteMediaClient.

[tonihei]

Thanks for highlighting this. It looks like we use a previously set MediaItem here without attempting to convert it again. Also note the mediaItemsByContentId that stores these previously set MediaItem instances.

The solution is probably to run the MediaItem conversion anyway, but merge the resulting MediaItem with the existing one if needed. I think this may actually be a duplicate of https://github.com/google/ExoPlayer/issues/8212 too, I'll keep this issue open as an enhancement to port it over to the Media3 issue tracker. See also the existing discussion on the other issue that highlights some reliability issues with this metadata that seemed to have prevented a proper implementation for now.