No video played when `setVideoEffects` called on NVIDIA Shield

damontecres commented 3 months ago

Version

Media3 1.4.0

More version details

Calling player.setVideoEffects() with any effects or even an empty list results in no video (just black screen).

I reviewed the requirements for this to work and they all seem met:

Not passing a surface manually, using a androidx. media3.ui.PlayerView
media-effect lib is added
Checking player.getRenderer and the only video one is MediaCodecVideoRenderer
Effects do not edit timestamps
No DRM
Function is called before prepare()

I guess it must be a hardware decoding issue, but I'm not sure how to validate. There's nothing obviously erroring in the logs.

Bare-bones code that causes the issue:

val player = ExoPlayer.Builder(this).build()
playerView.player = player
player.setVideoEffects(listOf()) // Commenting this out works
player.setMediaItem(MediaItem.fromUri("https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"))
player.prepare()
player.play()

Devices that reproduce the issue

NVIDIA Shield (2017 Pro & 2019 Pro models)

Devices that do not reproduce the issue

Emulator running Android TV API 34
Fire TV Stick 4k Max 2nd Gen

Reproducible in the demo app?

Not tested

Reproduction steps

Call setVideoEffects on the ExoPlayer object (even with an empty list)
Play a video

Expected result

Video plays successfully with effects applied or if no effects added at least the video plays normally

Actual result

Video playback is a black screen.

Audio plays as expected.

Media

https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4

Bug Report

[X] You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

damontecres commented 3 months ago

I didn't see an example in the demo apps that used setVideoEffects, so I made a test one https://github.com/damontecres/media3-issue if that helps. Thanks!

claincly commented 3 months ago

I was wondering what output is connected to this? Is it a PlayerView?

Can you help me double check a few things, adding logs to show the variables related to resolutions at

https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/video/MediaCodecVideoRenderer.java#L863

and

https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/libraries/effect/src/main/java/androidx/media3/effect/FinalShaderProgramWrapper.java#L366

And log if this line is executed:

https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/libraries/effect/src/main/java/androidx/media3/effect/FinalShaderProgramWrapper.java#L409

damontecres commented 3 months ago

Hi @claincly!

I was wondering what output is connected to this? Is it a PlayerView?

Yes, defined very simply:

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
  android:layout_width="match_parent"
  android:layout_height="match_parent">
  <androidx.media3.ui.PlayerView
    android:id="@+id/player_view"
    android:layout_width="match_parent"
    android:layout_height="match_parent"/>
</FrameLayout>

Can you help me double check a few things, adding logs to show the variables related to resolutions at

Here are the logs when executing with setVideoEffects called:

2024-08-14 11:53:55.657  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=0x0
2024-08-14 11:53:55.657  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:53:55.657  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = false
2024-08-14 11:53:55.771  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1920x984
2024-08-14 11:53:55.771  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:53:55.771  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true
2024-08-14 11:53:56.514  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-14 11:53:56.514  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-14 11:53:56.567  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface EGL14.eglSwapBuffers was called
2024-08-14 11:53:56.592  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-14 11:53:56.592  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-14 11:53:56.603  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface EGL14.eglSwapBuffers was called
2024-08-14 11:53:56.613  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-14 11:53:56.613  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-14 11:53:56.639  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface EGL14.eglSwapBuffers was called

Here are the logs when executing without setVideoEffects called:

2024-08-14 11:54:51.660  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=0x0
2024-08-14 11:54:51.660  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:54:51.661  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = false
2024-08-14 11:54:51.772  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1920x984
2024-08-14 11:54:51.772  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:54:51.772  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true
2024-08-14 11:54:52.389  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1749x984
2024-08-14 11:54:52.389  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:54:52.389  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true

damontecres commented 3 months ago

@claincly, is there any other information I can provide to help debug this?

Any ideas on if there is a work around? I tried sending the MSG_SET_VIDEO_OUTPUT_RESOLUTION message manually a few different ways without success.

claincly commented 2 months ago

I wonder if you could log on this line?

https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/libraries/effect/src/main/java/androidx/media3/effect/FinalShaderProgramWrapper.java#L409

The resolution logs look correct. You might noticed some discrepancies between with/without video effects, but that's expected.

damontecres commented 2 months ago

Hi @claincly, here are some logs from in that method. Thoughts?

2024-08-27 19:56:19.353 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=0x0
2024-08-27 19:56:19.353 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-27 19:56:19.353 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = false
2024-08-27 19:56:19.452 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1920x1080
2024-08-27 19:56:19.452 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-27 19:56:19.452 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true
2024-08-27 19:56:20.181 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.181 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.202 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.202 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000000000
2024-08-27 19:56:20.202 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=-1
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.218 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.244 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.244 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.249 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.249 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000041666
2024-08-27 19:56:20.249 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=440391116378964
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.254 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.263 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.263 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.265 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.265 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000083333
2024-08-27 19:56:20.265 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=440391166428963
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.269 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000125000
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=440391199795629
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.297 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.345 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.345 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720

claincly commented 2 months ago

Hmmm that is indeed very strange - I mean if eglSwapBuffers() is called, then we have already send the frame to display on the screen.

I also noticed you mentioned you didn't try reproducing using the demo app, could you try using the demo app and see it reproduces? I just wanted to rule out any unrelated reasons.

damontecres commented 2 months ago

The no video issue is reproducible in the basic demo app with a change to import the lib-effect module (it crashes when calling setVideoEffects otherwise).

I tried on another NVIDIA shield device (2017 model) and the issue is reproducible there as well.

I also tried the demo-transformer app on the same device and set some random RGB video effects. It produced a correctly transformed video including playback during export in the "debug preview". It looks like this debug preview is a Surface but not from/connected to a PlayerView if I'm reading the code correctly. Maybe that is a clue?

Here is the patch just in case:

diff --git a/demos/main/build.gradle b/demos/main/build.gradle
index 6a49adf169..78a43f180b 100644
--- a/demos/main/build.gradle
+++ b/demos/main/build.gradle
@@ -85,6 +85,7 @@ dependencies {
     implementation project(modulePrefix + 'lib-ui')
     implementation project(modulePrefix + 'lib-datasource-cronet')
     implementation project(modulePrefix + 'lib-exoplayer-ima')
+    implementation project(modulePrefix + 'lib-effect')
     withDecoderExtensionsImplementation project(modulePrefix + 'lib-decoder-av1')
     withDecoderExtensionsImplementation project(modulePrefix + 'lib-decoder-ffmpeg')
     withDecoderExtensionsImplementation project(modulePrefix + 'lib-decoder-flac')
diff --git a/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java b/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java
index 6be37323bd..2b3c11977a 100644
--- a/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java
+++ b/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java
@@ -258,7 +258,7 @@ public class PlayerActivity extends AppCompatActivity
   /**
    * @return Whether initialization was successful.
    */
-  protected boolean initializePlayer() {
+  @OptIn(markerClass = UnstableApi.class) protected boolean initializePlayer() {
     Intent intent = getIntent();
     if (player == null) {

@@ -279,6 +279,7 @@ public class PlayerActivity extends AppCompatActivity
       player.addAnalyticsListener(new EventLogger());
       player.setAudioAttributes(AudioAttributes.DEFAULT, /* handleAudioFocus= */ true);
       player.setPlayWhenReady(startAutoPlay);
+      player.setVideoEffects(new ArrayList<>());
       playerView.setPlayer(player);
       configurePlayerWithServerSideAdsLoader();
       debugViewHelper = new DebugTextViewHelper(player, debugTextView);

claincly commented 2 months ago

It looks like this debug preview is a Surface but not from/connected to a PlayerView if I'm reading the code correctly. Maybe that is a clue?

Interesting find. You are correct the debug preview is not a PlayerView (I think it's just a SurfaceView). The only difference I can think of is we call eglPresentationTimeANDROID() when rendering to a PlayerView. Now that method is enabled by an OpenGL extension which might now work reliably on all devices.

Although I don't recommend it, could you try commenting that line out and see if the video plays?

https://github.com/androidx/media/blob/c35a9d62baec57118ea898e271ac66819399649b/libraries/effect/src/main/java/androidx/media3/effect/FinalShaderProgramWrapper.java#L437-L442

damontecres commented 2 months ago

Although I don't recommend it, could you try commenting that line out and see if the video plays?

Commenting out the EGLExt.eglPresentationTimeANDROID method call does not resolve the issue, still just a black screen with audio.

I also tried updating to 1.4.1 just in case, but no joy.

claincly commented 2 months ago

One last thing to try - Nvidia might have very customized OpenGL stack),

Adding glFinish() after calling eglSwapBuffers(), and

change the color to something like red in clearFocusedBuffers(), here to something not black, maybe set red to 1

https://github.com/androidx/media/blob/c35a9d62baec57118ea898e271ac66819399649b/libraries/common/src/main/java/androidx/media3/common/util/GlUtil.java#L554-L555

damontecres commented 2 months ago

Adding glFinish() after calling eglSwapBuffers(), and

No change

change the color to something like red in clearFocusedBuffers(), here to something not black, maybe set red to 1

Screen is still black!

androidx / media