search

androidx / media

Jetpack Media3 support libraries for media use cases, including ExoPlayer, an extensible media player for Android
https://developer.android.com/media/media3
Apache License 2.0
1.65k stars 390 forks source link

Media codec initialisation fails when it switches from Exoplayer extension renderer to Media codec renderer run time. #1802

Open mayurk2 opened 1 week ago

mayurk2 commented 1 week ago

Version

ExoPlayer 2.16.1

More version details

I am developing an Extension renderer for dav1d decoder. The normal playback is working fine with this new renderer.

I have Main content in AV1 and mid-roll Ads in H264. So when the playback switches from main content to ads, internally Exoplayer switches from dav1d extension renderer to Media codec renderer.

I am noticing the below error when this switching happens.

10-12 03:53:36.539 I/MediaCodec(27074): MediaCodec will operate in async mode
10-12 03:53:36.540 D/MediaCodec(27074): flushMediametrics
10-12 03:53:36.541 D/SurfaceUtils(27074): connecting to surface 0xb400006fb3d3e010, reason connectToSurface
10-12 03:53:36.541 E/BufferQueueProducer(27074): [SurfaceView[in.abc.def/com.abc.MainActivity]#1(BLAST Consumer)1](id:69c200000002,api:2,p:27074,c:27074) connect: already connected (cur=2 req=3)
10-12 03:53:36.541 E/SurfaceUtils(27074): Failed to connect to surface 0xb400006fb3d3e010, err -22
10-12 03:53:36.541 E/MediaCodec(27074): nativeWindowConnect returned an error: Invalid argument (-22)
10-12 03:53:36.542 E/MediaCodec(27074): configure failed with err 0xffffffea, resetting...

And it eventually causes decoder initialisation failure.

10-12 03:53:36.623 W/MediaCodecRenderer(27074): Failed to initialize decoder: OMX.qcom.video.decoder.avc
10-12 03:53:36.623 W/MediaCodecRenderer(27074):   java.lang.IllegalArgumentException
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at android.media.MediaCodec.native_configure(Native Method)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at android.media.MediaCodec.configure(MediaCodec.java:2176)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at android.media.MediaCodec.configure(MediaCodec.java:2092)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at com.google.android.exoplayer2.mediacodec.AsynchronousMediaCodecAdapter.initialize(AsynchronousMediaCodecAdapter.java:165)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at com.google.android.exoplayer2.mediacodec.AsynchronousMediaCodecAdapter.access$100(AsynchronousMediaCodecAdapter.java:46)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at com.google.android.exoplayer2.mediacodec.AsynchronousMediaCodecAdapter$Factory.createAdapter(AsynchronousMediaCodecAdapter.java:107)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at com.google.android.exoplayer2.mediacodec.DefaultMediaCodecAdapterFactory.createAdapter(DefaultMediaCodecAdapterFactory.java:122)
10-12 03:53:36.623 W/MediaCodecRenderer(27074):       at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.initCodec(MediaCodecRenderer.java:1084)

I explored on this issue and I have some findings.

  1. It is similar to https://github.com/google/ExoPlayer/issues/6454 . But instead of direct surface.lockCanvas() in Java, I am doing ANativeWindow_lock() and ANativeWindow_unlockAndPost() just like it is done in existing extension-av1 for libgav1 renderer.
  2. I also found this post - https://stackoverflow.com/questions/21526989/native-window-api-connect-returned-an-error-invalid-argument-22

Both of these posts points that after calling lock() on surface, internally it connects surface to CPU producer. Surface::connect(NATIVE_WINDOW_API_CPU); Since the surface is already connected to a producer, it is failing when media codec is trying to connect some producer again during initialisation.

My questions are:

  1. Is my understanding correct?
  2. How do I disconnect surface from the CPU producer from my custom extension renderer? I did not find an API in native_window.h

  3. What are the alternate ways to achieve this? I need to re-use the same surface view for both AV1 and media codec renderers.

Additional findings I just checked how the media codec is handling surface.

  1. When surface is set in media codec, it tries connecting to surface (MediaCodec.connectToSurface())
  2. It calls SurfaceUtils.nativeWindoConnect() and finally window->perform(window, NATIVE_WINDOW_API_CONNECT, api);
  3. I find that window.h has native_window_api_disconnect() which does window->perform(window, NATIVE_WINDOW_API_DISCONNECT, api); .
  4. I think this is the way to disconnect. But these all APIs are in libstagefright. I am not sure how to access them from my native code.

Devices that reproduce the issue

Not device specific. I reproduced on Redmi Note 9 Pro.

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

I am using my own extension renderer. But the problem seems to be generic.

  1. Play AV1 content with H265 mid roll.
  2. Make sure av1 extension renderer is used.
  3. Wait for ads to start.

Expected result

It should play without any error.

Actual result

The player fails to switch from extension renderer to media codec renderer.

Media

The extension renderer is not public. Hence cannot share the media URL.

Bug Report

mayurk2 commented 1 week ago

Update:

I solved this problem by copying the decoder output buffer to surface using openGL. This stack overflow answer was a great help - https://stackoverflow.com/a/21564236

mayurk2 commented 1 week ago

This is still a problem with the most of the existing extension renderer I guess. I am re-opening this issue and let the moderator decide if any actions are required.