search

androidx / media

Jetpack Media3 support libraries for media use cases, including ExoPlayer, an extensible media player for Android
https://developer.android.com/media/media3
Apache License 2.0
1.62k stars 385 forks source link

Can't interact with ExoPlayer using Jetpack Compose on Android TV #460

Open taschmidt opened 1 year ago

taschmidt commented 1 year ago

Media3 Version

ExoPlayer 2.18.7

Devices that reproduce the issue

Emulator running Android TV API 31

Devices that do not reproduce the issue

None

Reproducible in the demo app?

Yes

Reproduction steps

Using androidx.tv version 1.0.0-alpha06 with the following composable (adding focusable and focusRequester were my attempts to fix it):

@OptIn(ExperimentalComposeUiApi::class)
@Composable
fun VideoPlayer(mediaItem: VideoApi.MediaItem) {
    val (playerFocus) = remember { FocusRequester.createRefs() }
    val context = LocalContext.current

    val exoPlayer = remember {
        ExoPlayer.Builder(context)
            .build()
            .apply {
                addMediaItem(
                    MediaItem.Builder()
                        .setUri(mediaItem.hlsCaptions)
                        .setMimeType(MimeTypes.APPLICATION_M3U8)
                        .build()
                )
                prepare()
                playWhenReady = true
            }
    }

    LaunchedEffect(Unit) {
        playerFocus.requestFocus()
    }

    DisposableEffect(
        AndroidView(
            modifier = Modifier
                .focusable(true)
                .focusRequester(playerFocus),
            factory = {
                StyledPlayerView(context).apply {
                    player = exoPlayer
                    useController = true
                    controllerAutoShow = true
                    setShowBuffering(StyledPlayerView.SHOW_BUFFERING_ALWAYS)
                }
            }
        )
    ) {
        onDispose {
            exoPlayer.release()
        }
    }
}

Expected result

Should be able to interact, i.e. D-pad should bring up video HUD controls.

Actual result

Any key presses (besides back) do nothing.

Media

Any media should show it. I'm using an HLS stream. For example: https://m.wsj.net/video/20230531/eaa54145-d07a-4100-8153-2eaf8d671921/1/hls/manifest-hd-wifi.m3u8

Bug Report

marcbaechinger commented 1 year ago

I'm not a Compose expert tbh. Without Compose on TV, the key events coming from the remote need to arrive here in dispatchKeyEvent for that to work. In the main demo app we delegate the calls to dispatchKeyEvents of the PlayerActivity to the PlayerView.

I believe we mainly do that for TV to have the DPad working.

How are key events delegated from your activity through the Compose stack to do what the demo app does without compose?

taschmidt commented 1 year ago

I'm still digging. What I've tried since:

taschmidt commented 1 year ago

Ok, possible breakthrough. I think my onKeyEvent was working after all, it's just that Android Studio was being weird. Also, I think I had to add the focusable modifier to my AndroidView to make it work. Once I figured all that out, I followed your helpful tip @marcbaechinger. I had to move my StyledPlayerView to state so I could call directly into it on key events but it seems to be working. Here's what I ended up with:

@Composable
fun VideoPlayer(mediaItem: VideoApi.MediaItem,
                modifier: Modifier = Modifier
) {
    val context = LocalContext.current

    val exoPlayer = remember {
        ExoPlayer.Builder(context)
            .build()
    }

    val playerView = remember {
        StyledPlayerView(context).apply {
            player = exoPlayer
            useController = true
            controllerAutoShow = true
            setShowBuffering(StyledPlayerView.SHOW_BUFFERING_ALWAYS)

            exoPlayer.setMediaItem(
                MediaItem.Builder()
                    .setUri(mediaItem.hlsCaptions)
                    .setMimeType(MimeTypes.APPLICATION_M3U8)
                    .build()
            )
            exoPlayer.prepare()
            exoPlayer.playWhenReady = true
        }
    }

    DisposableEffect(
        AndroidView(
            modifier = modifier.focusable()
                .onKeyEvent { playerView.dispatchKeyEvent(it.nativeKeyEvent) },
            factory = { playerView }
        )
    ) {
        onDispose {
            exoPlayer.release()
        }
    }
}
marcbaechinger commented 1 year ago

Noice! Glad it's working. Many thanks for the explanation!

I tagged this issue as documentation candidate. Guess this would be useful in a short section of the developer guide.

The code above or some other code involved in the problem you just solved, isn't related to androidx.tv. The used APIs are common Compose or Media3 classes only. Is this correct?

taschmidt commented 1 year ago

Correct, AFAIK there's nothing TV specific.

taschmidt commented 1 year ago

So here's the only minor nit I have. I can now push up or down and the controls show and I can interact with the play of the video. However, for usability I would love for my users to be able to just push play/pause/rew/ff with a single click rather than multiple clicks to bring up the controls, navigate to the correct one, then click. However, these clicks don't seem to reach my onKeyEvent and I'm not sure who's intercepting them. On that same AndroidView I also tried using onPreviewKeyEvent but I never got anything. Any ideas there?

taschmidt commented 1 year ago

Following up here since we're getting close to app launch. Anyone have any ideas on how I can react to rew/ff key events?

marcbaechinger commented 1 year ago

for my users to be able to just push play/pause/rew/ff with a single click

I looked into this with a TV emulator. I want to share my technical observations in case this is useful, without giving any recommendations aside from saying there is no API for this right now.

Technically there are no play/pause/rew/ff emitted by a D-pad control. Both, the TV controls on the emulator as well as the HW control of a Chromecast with Google TV do have a D-pad control without specific buttons for playback commands (as for instance media button events).

The events emitted are directional. See KEYCODE_DPAD_FOO.

Media3 UI handles D-pad events but assigns playback semantics only when the timebar is focused. In such a case KEYCODE_DPAD_LEFT/RIGHT are interpreted indirectly as ffw/rwd by moving the timebar knob to left/right (see DefaultTimeBar). In all other cases the event are used to navigate the focus from control to control and to execute the focused action. The behavior you describe above.

If the controls are not visible, then pressing any of the D-pad buttons result in just making the controls appear. This is implemented in PlayerView.dispatchKeyEvent().

However, these clicks don't seem to reach my onKeyEvent and I'm not sure who's intercepting them

Due to my lack of familiarity, I cant give you advise regarding Compose I'm afraid. When with the demo app, I intercept the call in PlayerActivity.dispatchKeyEvent() I see these D-pad keve events. I can for instance just do nothing than returning true, then nothing happens when clicking the D-pad buttons. However, I can from PlayerActivity.dispatchKeyEvent(...) implement a different behaviour:

@Override
public boolean dispatchKeyEvent(KeyEvent event) {
  if (!playerView.isControllerFullyVisible()) {
    playerView.showController();
    if (event.getKeyCode() == KeyEvent.KEYCODE_DPAD_RIGHT) {
      playerView.findViewById(R.id.exo_ffwd_with_amount).requestFocus();
    } else if (event.getKeyCode() == KeyEvent.KEYCODE_DPAD_LEFT) {
      playerView.findViewById(R.id.exo_rew_with_amount).requestFocus();
    }
    return true;
  }
  return playerView.dispatchKeyEvent(event);
}

That technically works, but at least my naive implementation above doesn't give a consistent UX, because only the first DPAD_RIGHT is interpreted as ffwd. Afterwards, they need to be used for navigation or the UI can't be used properly anymore. From a library perspective I think the current interpretation of D-pad events is how it should be by default with the given API.

The IDs of the views are defined but not documented. I don't see why we would want to change them, but you would have to expect that these IDs may change at some point in the future even if there is no indication that this will happen any time soon. Doing a test case for this would be recommended if you go down such a path.

don't seem to reach my onKeyEvent

I'm not sure why you don't see this event I'm afraid. I'd start at the most top component for which you can override dispatchKeyEvent like the activity. The PlayerView having focus may be important too in case you have a more complex UI. For PlayerActivitvy I found that not overriding super.dispatchKeyEvent worked as well when in focus, so the event needs to be intercepted somewhere in between the framework and the PlayerView if that's not working.

taschmidt commented 1 year ago

@marcbaechinger you're correct, sorry if I was misleading. I do get the D-pad events. It's specifically the rew, ff, play/pause key events. From looking at past code of mine, I believe the first two are KEYCODE_MEDIA_REWIND and KEYCODE_MEDIA_FAST_FORWARD. I understand that these keys aren't standard on the emulator or Google TV devices, but if you're a frequent Fire TV user (as I am) the inability to control full screen video with those keys feel very strange and annoying.

The fact that I don't get those key codes is odd and I'm not sure where to go with this.

UPDATE: via the debugger I do see the media key events if I put a breakpoint way up in ViewGroup.dispatchKeyEvent. I'm trying to dig into where/how this gets handed off to Jetpack Compose to see why they aren't getting to my code.

UPDATE 2: I see the events reach the compose hierarchy here but I'm not sure where to go from here.

taschmidt commented 1 year ago

Interesting, definitely some Amazon special sauce here. Seeing this in the logs:

2023-07-24 14:25:30.054   616-660   AmazonProfileService    pid-616                              I  Reporting top activity: ComponentInfo{com.dowjones.wsj.androidtv/com.dowjones.wsj.MainActivity}
2023-07-24 14:25:30.055   616-660   AmazonProfileService    pid-616                              I  Event is not filtered because it is not in filter key list.
2023-07-24 14:25:30.055   616-660   FireTVKeyPolicyManager  pid-616                              I  Dynamic key mapping has been enabled
2023-07-24 14:25:30.055   616-660   KeyMapManagerUtil       pid-616                              I  Unknown remote sku id 0. Fetch remote sku id from system property
2023-07-24 14:25:30.055   616-660   FireTVKeyPolicyManager  pid-616                              I  Dynamic key mapping key intercepted result 0
2023-07-24 14:25:30.056   616-660   AmazonProfileService    pid-616                              I  Reporting top activity: ComponentInfo{com.dowjones.wsj.androidtv/com.dowjones.wsj.MainActivity}
2023-07-24 14:25:30.056   616-660   AmazonProfileService    pid-616                              I  Event is not filtered because it is not in filter key list.
2023-07-24 14:25:30.056  1383-10394 OZ-DCS:KeyPressReceiver pid-1383                             I  Received keyCode: KEYCODE_MEDIA_PLAY_PAUSE
2023-07-24 14:25:30.056   616-660   FireTVKeyPolicyManager  pid-616                              I  Dynamic key mapping has been enabled
2023-07-24 14:25:30.057   616-660   KeyMapManagerUtil       pid-616                              I  Unknown remote sku id 0. Fetch remote sku id from system property
2023-07-24 14:25:30.057   616-660   FireTVKeyPolicyManager  pid-616                              I  Dynamic key mapping key intercepted result 0
2023-07-24 14:25:30.057  1383-10394 OZ-DCS:Cal...r_KeyPress pid-1383                             I  notify callbacks: KeyEvent { action=ACTION_DOWN, keyCode=KEYCODE_MEDIA_PLAY_PAUSE, scanCode=164, metaState=0, flags=0x8, repeatCount=0, eventTime=609805827, downTime=609805827, deviceId=27, source=0x301 }
2023-07-24 14:25:30.060  1383-10394 OZ-DCS:KeyPressReceiver pid-1383                             I  Remote button pressed. keyCode: 85, keyEvent: KeyEvent { action=ACTION_DOWN, keyCode=KEYCODE_MEDIA_PLAY_PAUSE, scanCode=164, metaState=0, flags=0x8, repeatCount=0, eventTime=609805827, downTime=609805827, deviceId=27, source=0x301 }
2023-07-24 14:25:30.061  1383-1397  OZ-DCS:KeyPressReceiver pid-1383                             I  Received keyCode: KEYCODE_MEDIA_PLAY_PAUSE
2023-07-24 14:25:30.062  1383-1397  OZ-DCS:Cal...r_KeyPress pid-1383                             I  notify callbacks: KeyEvent { action=ACTION_DOWN, keyCode=KEYCODE_MEDIA_PLAY_PAUSE, scanCode=164, metaState=0, flags=0x8, repeatCount=0, eventTime=609805827, downTime=609805827, deviceId=27, source=0x301 }
2023-07-24 14:25:30.063  1383-1397  OZ-DCS:KeyPressReceiver pid-1383                             I  Remote button pressed. keyCode: 85, keyEvent: KeyEvent { action=ACTION_DOWN, keyCode=KEYCODE_MEDIA_PLAY_PAUSE, scanCode=164, metaState=0, flags=0x8, repeatCount=0, eventTime=609805827, downTime=609805827, deviceId=27, source=0x301 }
2023-07-24 14:25:30.070 20005-20005 AmazonVideo             pid-20005                            I  Receiving MediaButtonEventReceiver intent
2023-07-24 14:25:30.076 20005-20005 AmazonVideo             pid-20005                            I  Receiving MediaButtonEventReceiver intent
taschmidt commented 1 year ago

One more update, I can actually get the media events way up in my Activity's onKeyDown (for Compose, you still have a top-level Activity instance) but I can't really think of a clean way to get that down to the active video player... argh...

mindwalkr-ttd commented 1 year ago

@taschmidt Stream them in a Flow. Inject the flow into the Activity and also in the ViewModel/whatever constructor.

DPAD means you only have a limited set of inputs "events", which makes it hacky, but "reasonable" (I claim) to pass them down from the Activity if Compose cannot receive them (eg: system keys that come into the app via an Intent instead of a KeyEvent).

taschmidt commented 1 year ago

@mindwalkr-ttd great suggestion and it appears to be working perfectly!

For anyone else, I added this to my Activity:

    private val _keyDownEvents = MutableSharedFlow<KeyEvent>()
    val keyDownEvents = _keyDownEvents.asSharedFlow()

    ...

    override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {
        event?.let {
            when (it.keyCode) {
                KeyEvent.KEYCODE_MEDIA_PLAY_PAUSE,
                KeyEvent.KEYCODE_MEDIA_PLAY,
                KeyEvent.KEYCODE_MEDIA_PAUSE,
                KeyEvent.KEYCODE_MEDIA_REWIND,
                KeyEvent.KEYCODE_MEDIA_FAST_FORWARD -> {
                    lifecycleScope.launch { _keyDownEvents.emit(it) }
                    return true
                }
                else -> {}
            }
        }

        return super.onKeyDown(keyCode, event)
    }

Then in my VideoPlayer composable (inside a LaunchedEffect I already had):

            lifecycle.coroutineScope.launch {
                activity?.keyDownEvents?.collect { value ->
                    playerView.dispatchMediaKeyEvent(value)
                }
            }
mindwalkr-ttd commented 1 year ago

Consider trapping onKeyUp instead of onKeyDown, since the User can hold down a button resulting in multiple calls to onKeyDown for the same button.

taschmidt commented 1 year ago

That didn't seem to work. It looks like PlayerControlView is only interested in key down events: https://github.com/androidx/media/blob/5328d6464acb077a7e8cba61b8cac1973c4943d7/libraries/ui/src/main/java/androidx/media3/ui/PlayerControlView.java#L1464