google / ExoPlayer

This project is deprecated and stale. The latest ExoPlayer code is available in https://github.com/androidx/media
https://developer.android.com/media/media3/exoplayer
Apache License 2.0
21.74k stars 6.03k forks source link

setOutputSurface workaround needed for some more devices #4468

Closed qvk closed 5 years ago

qvk commented 6 years ago

Issue description

We're getting multiple records of ExoPlaybackException caused by IllegalArgumentException (sometimes MediaCodec#CodecException) when MediaCodec#setOutputSurface is called as seen in collected stacktraces. These happen when exiting the video activity, so I'm assuming SimpleExoPlayer sends a MSG_SET_SURFACE when surfaceDestroyed is invoked by the system, which eventually leads to the playback exception.

We started logging decoder name, error stacktraces and device info inside onPlayerError callback only in recent releases, so the actual occurrences are likely much more than we could record.

It sure doesn't look good adding all these devices to the workaround list, but there are only 4 or 5 unique decoder names in the list, so maybe there is some pattern here?

Reproduction steps

We don't have the affected devices on hand but according to our logs, the issue occurs upon exiting any activity which is playing media using exoplayer.

Version of ExoPlayer being used

2.7.0

Device(s) and version(s) of Android being used

These are data collected from last couple of weeks for cases where onPlayerError callback is received with the above mentioned exception.

Many devices near the end of the list with very few occurrences might not warrant adding them to the exception list. Although they could be new devices just starting to grow in use.

count apiLevel  videoDecoderName        device  model   manufacturer

107 23  OMX.MTK.VIDEO.DECODER.AVC   CPH1609 CPH1609 OPPO
86  23  OMX.MTK.VIDEO.DECODER.AVC   A7020a48    Lenovo A7020a48 LENOVO
76  24  OMX.MTK.VIDEO.DECODER.AVC   1714    vivo 1714   vivo
71  23  OMX.MTK.VIDEO.DECODER.AVC   K50a40  Lenovo K50a40   LENOVO
57  23  OMX.MTK.VIDEO.DECODER.AVC   taido_row   XT1706  motorola
53  23  OMX.MTK.VIDEO.DECODER.AVC   BLACK-1X    BLACK-1XM   XOLO
52  23  OMX.MTK.VIDEO.DECODER.AVC   A10-70F Lenovo TAB 2 A10-70F    Lenovo
48  23  OMX.MTK.VIDEO.DECODER.AVC   CPY83_I00   Coolpad Y83-I00 Coolpad
48  23  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_GBL7360  F103 Pro    GIONEE
47  23  OMX.MTK.VIDEO.DECODER.AVC   Slate_Pro   Slate Pro   Swipe
37  23  OMX.MTK.VIDEO.DECODER.AVC   1601    vivo 1601   vivo
32  24  OMX.MTK.VIDEO.DECODER.AVC   Infinix-X572    Infinix X572    Infinix
24  24  OMX.MTK.VIDEO.DECODER.AVC   QX1 Sky Quantum
18  23  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_WBL5708  P7 Max  GIONEE
16  23  OMX.MTK.VIDEO.DECODER.AVC   Pixi4-7_3G  9003X   TCL
15  24  OMX.MTK.VIDEO.DECODER.AVC   Aura_Note_2 Aura Note 2 KARBONN
15  23  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_WBL7365  GIONEE P7   GIONEE
14  23  OMX.MTK.VIDEO.DECODER.AVC   V1  V1  CELLCOM
11  24  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_SWW1631  X1  GIONEE
11  23  OMX.MTK.VIDEO.DECODER.AVC   1601    vivo 1713   vivo
10  24  OMX.MTK.VIDEO.DECODER.AVC   C1  Micromax C1 Micromax
9   23  OMX.MTK.VIDEO.DECODER.AVC   XT1663  XT1663  motorola
9   24  OMX.MTK.VIDEO.DECODER.AVC   XT1663  XT1663  motorola
9   23  OMX.MTK.VIDEO.DECODER.AVC   CP8676_I02  CP8676_I02  Coolpad
8   23  OMX.MTK.VIDEO.DECODER.AVC   X3_HK   Le X507 Letv
8   24  OMX.MTK.VIDEO.DECODER.AVC   F3215   F3215   Sony
6   23  OMX.MTK.VIDEO.DECODER.AVC   ELUGA_Note  ELUGA Note  PANASONIC
6   23  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_WBL7519  S6s GIONEE
6   23  OMX.MTK.VIDEO.DECODER.AVC   TB3-730X    Lenovo TB3-730X LENOVO
6   23  OMX.MTK.VIDEO.DECODER.AVC   iball8735_9806  iBall Slide Wings 4GP   iBall
6   24  OMX.MTK.VIDEO.DECODER.AVC   iris60  iris60  LAVA
6   25  OMX.MTK.VIDEO.DECODER.AVC   marino_f    Lenovo K8 Plus  LENOVO
6   25  OMX.MTK.VIDEO.DECODER.AVC   nicklaus_f  Moto E (4) Plus motorola
5   23  OMX.MTK.VIDEO.DECODER.AVC   htc_e56ml_dtul  HTC One X9 dual sim HTC
5   25  OMX.MTK.VIDEO.DECODER.AVC   manning Lenovo K8 Note  lenovo
4   24  OMX.MTK.VIDEO.DECODER.AVC   F3116   F3116   Sony
4   23  OMX.MTK.VIDEO.DECODER.AVC   PGN528  PGN528  condor
4   23  OMX.MTK.VIDEO.DECODER.AVC   Q427    Micromax Q427   Micromax
3   23  OMX.MTK.VIDEO.DECODER.AVC   X3_HK   Le X509 Letv
3   23  OMX.MTK.VIDEO.DECODER.AVC   le_x6   LEX622  LeMobile
3   23  OMX.MTK.VIDEO.DECODER.AVC   A1601   A1601   OPPO
3   24  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_SWW1609  A1  GIONEE
3   23  OMX.MTK.VIDEO.DECODER.AVC   GiONEE_GBL7319  F103    GiONEE
3   23  OMX.MTK.VIDEO.DECODER.AVC   LS-5017 LS-5017 LYF
3   23  OMX.MTK.VIDEO.DECODER.AVC   NX573J  NX573J  nubia
3   23  OMX.MTK.VIDEO.DECODER.AVC   PGN610  PGN610  condor
3   23  OMX.MTK.VIDEO.DECODER.AVC   Q350    Micromax Q350   Micromax
3   23  OMX.MTK.VIDEO.DECODER.AVC   TB3-730F    Lenovo TB3-730F LENOVO
3   24  OMX.MTK.VIDEO.DECODER.AVC   Z80 Z80 LAVA
2   23  OMX.MTK.VIDEO.DECODER.AVC   A2016a40    Lenovo A2016a40 LENOVO
2   23  OMX.MTK.VIDEO.DECODER.AVC   A7000plus   Lenovo A7000-a  LENOVO
2   24  OMX.MTK.VIDEO.DECODER.AVC   ELUGA_A3_Pro    ELUGA_A3_Pro    PANASONIC
2   23  OMX.MTK.VIDEO.DECODER.AVC   ELUGA_Prim  ELUGA Prim  PANASONIC
2   24  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_SWW1627  X1S GIONEE
2   23  OMX.MTK.VIDEO.DECODER.AVC   JGZ TSP InFocus
2   27  OMX.qcom.video.decoder.avc  OnePlus5T   ONEPLUS A5010   OnePlus
2   23  OMX.MTK.VIDEO.DECODER.AVC   GIONEE_WBL7365  P7  GIONEE
2   23  OMX.MTK.VIDEO.DECODER.AVC   P85 Panasonic P85   PANASONIC
2   23  OMX.MTK.VIDEO.DECODER.AVC   Phantom6    Phantom6S   TECNO
2   23  OMX.MTK.VIDEO.DECODER.AVC   Q4260   Micromax Q4260  Micromax
2   24  OMX.MTK.VIDEO.DECODER.AVC   Q4310   Micromax Q4310  Micromax
2   23  OMX.MTK.VIDEO.DECODER.AVC   TB3-850M    Lenovo TB3-850M LENOVO
2   24  OMX.MTK.VIDEO.DECODER.AVC   vernee_M5   M5  vernee
2   25  OMX.MTK.VIDEO.DECODER.AVC   woods_f Moto E (4)  motorola
1   23  OMX.MTK.VIDEO.DECODER.AVC   A7000-a Lenovo A7000-a  LENOVO
1   24  OMX.MTK.VIDEO.DECODER.AVC   PGN611  PGN611  SPA Condor Electronics
1   23  OMX.MTK.VIDEO.DECODER.AVC   PGN611  PGN611  condor
1   23  OMX.MTK.VIDEO.DECODER.AVC   AquaPowerM  Aqua Power M    Intex
1   24  OMX.MTK.VIDEO.DECODER.AVC   ComioS1 COMIO S1    COMIO
1   23  OMX.MTK.VIDEO.DECODER.AVC   E5643   E5643   Sony
1   23  OMX.MTK.VIDEO.DECODER.AVC   ELUGA_Ray_X ELUGA Ray X PANASONIC
1   23  OMX.MTK.VIDEO.DECODER.AVC   EverStar_S  EverStar S  Cellcom
1   23  OMX.MTK.VIDEO.DECODER.AVC   le_x6   Le X620 LeMobile
1   23  OMX.MTK.VIDEO.DECODER.AVC   GiONEE_CBL7513  S6  GiONEE
1   24  OMX.IMG.MSVDX.Decoder.AVC   HWBLN-H BLN-L22 HUAWEI
1   23  OMX.MTK.VIDEO.DECODER.AVC   MEIZU_M5    MEIZU_M5    Meizu
1   25  OMX.MTK.VIDEO.DECODER.AVC   MX6 MX6 Meizu
1   23  OMX.MTK.VIDEO.DECODER.AVC   NX541J  NX541J  nubia
1   23  OMX.MTK.VIDEO.DECODER.AVC   P681    Micromax P681   Micromax
1   23  OMX.MTK.VIDEO.DECODER.AVC   PB2-670M    Lenovo PB2-670M LENOVO
1   26  OMX.qcom.video.decoder.avc  PLE TA-1021 HMD Global
1   24  OMX.MTK.VIDEO.DECODER.AVC   Pixi5-10_4G 9026T   TCL
1   23  OMX.MTK.VIDEO.DECODER.AVC   Q5  Quantum MUV PRO Quantum
1   23  OMX.MTK.VIDEO.DECODER.AVC   TB3-850F    Lenovo TB3-850F LENOVO
1   23  OMX.MTK.VIDEO.DECODER.AVC   V23GB   V23GB   LAVA
1   23  OMX.MTK.VIDEO.DECODER.AVC   V5  LAVA V5 M   Lava
1   23  OMX.MTK.VIDEO.DECODER.AVC   XE2X    Era 2X  LAVA International
1   23  OMX.MTK.VIDEO.DECODER.AVC   Z12_PRO Z12 PRO QMobile
1   25  OMX.qcom.video.decoder.avc  cv1 LM-X210(G)  LGE
1   25  OMX.qcom.video.decoder.avc  cv3 LM-X410(FG) LGE
1   24  OMX.qcom.video.decoder.avc  griffin XT1650  motorola
1   24  OMX.MTK.VIDEO.DECODER.AVC   itel_S41    itel S41    itel
1   23  OMX.sprd.h264.decoder   j2xlteins   SM-J210F    samsung
1   25  OMX.qcom.video.decoder.avc  mh  LG-M700 LGE
1   25  OMX.qcom.video.decoder.avc  whyred  Redmi Note 5 Pro    Xiaomi

A full bug report captured from the device

Unfortunately, we only noticed these errors from logs collected from remote user devices, so it's not possible to collect bug report using adb.

However, the stacktrace collected in all cases (collected inside onPlayerError callback) is either of the following two:

java.lang.IllegalArgumentException 
at android.media.MediaCodec.native_setSurface(Native Method) 
at android.media.MediaCodec.setOutputSurface(MediaCodec.java:1914) 
at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setOutputSurfaceV23(MediaCodecVideoRenderer.java:857) 
at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setSurface(MediaCodecVideoRenderer.java:383) 
at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.handleMessage(MediaCodecVideoRenderer.java:350) 
at com.google.android.exoplayer2.ExoPlayerImplInternal.deliverMessage(ExoPlayerImplInternal.java:854) 
at com.google.android.exoplayer2.ExoPlayerImplInternal.sendMessageToTarget(ExoPlayerImplInternal.java:829) 
...
android.media.MediaCodec$CodecException: Error 0xffffffff 
at android.media.MediaCodec.native_setSurface(Native Method) at 
android.media.MediaCodec.setOutputSurface(MediaCodec.java:1810) 
at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setOutputSurfaceV23(MediaCodecVideoRenderer.java:857) 
at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setSurface(MediaCodecVideoRenderer.java:383) 
at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.handleMessage(MediaCodecVideoRenderer.java:350) 
at com.google.android.exoplayer2.ExoPlayerImplInternal.deliverMessage(ExoPlayerImplInternal.java:854) 
at com.google.android.exoplayer2.ExoPlayerImplInternal.sendMessageToTarget(ExoPlayerImplInternal.java:829)
...
qvk commented 6 years ago

Also, I'm not sure if this is feasible or already being done, but is it possible (or even useful) to avoid dispatching/handling MSG_SET_SURFACE when ExoPlayer#release has been called?

In the lifecycle event onStop(), we call ExoPlayer#release on the application main thread. surfaceDestroyed should also be called sometime around this (although I believe it's timing is not tied to the application lifecycle methods). Since both release and setSurface are processed on the internal playback thread (correct me if I'm wrong about this), is it possible/safe/useful to skip setSurface if the release call was received earlier?

Thanks :)

ojw28 commented 6 years ago

Addressing your second comment first, since it's easy to tick off:

Also, I'm not sure if this is feasible or already being done, but is it possible (or even useful) to avoid dispatching/handling MSG_SET_SURFACE when ExoPlayer#release has been called?

I'm pretty sure this already happens. Which as an aside, implies that surfaceDestroyed is being called before onStop in cases where you're seeing this error.

ojw28 commented 6 years ago

Addressing your first comment:

We recently split the workaround you mention in two: codecNeedsSetOutputSurfaceWorkaround and codecNeedsDummySurfaceSurfaceWorkaround. It's better to use the latter if that's sufficient to fix the problem, since it still allows for seamless switching of playbacks from one valid surface to another.

Does your app ever switch the surface that it outputs too? If so, and if that case isn't failing, that would indicate that codecNeedsDummySurfaceSurfaceWorkaround would be sufficient. On the other hand, if your app doesn't ever do this, there's no easy way to tell which workaround is needed for each device.

If you could answer the question above, that would be great. In the meantime we'll try and get hold of some of the devices listed and test them manually.

It sure doesn't look good adding all these devices to the workaround list, but there are only 4 or 5 unique decoder names in the list, so maybe there is some pattern here?

I don't think we can blacklist solely based on decoder name. SOC vendors tend to use the same decoder name across all of their products. So for example, any device that uses a MediaTek chipset will have a decoder named OMX.MTK.VIDEO.DECODER.AVC. However most likely only a subset of MediaTek chipsets need the workaround, and there's no public API on Android for querying what the chipset is.

Still, the data provided is really useful for helping us to dig for a pattern. What we'll do on our side is:

After all that is done, we'll try and close the loop with you (i.e. check that some version you release in the future is not still logging these failures).

ojw28 commented 6 years ago

Note to self: It was suggested in #3835 that all devices that use MT6737 and MT6753 chipsets may be affected by this issue.

ojw28 commented 6 years ago

@qvk - Is it possible for you to add a new column to this data that shows more specific Android build information? I'm not sure what data you have exactly. Build fingerprints (Build.FINGERPRINT) would be ideal. They look something like:

motorola/taido_row/taido_row:6.0/MRA58K/XT1706_S149_180322_ROW:user/release-keys

If you don't have that just the build name (MRA58K) would be helpful, albeit not to the same extent!

Some of the devices in your list appear to have passed the tests that are supposed to prevent this issue from occurring, however it's unclear whether the reports you're seeing are from devices that are running earlier builds than those which passed the tests. It would be good to check that as part of our cross-referencing.

noamtamim commented 6 years ago

@ojw28 I suggest tracking all the devices that need any type of ExoPlayer workaround in a wiki page, here on GitHub. Doesn't have to be a public-edit wiki.

qvk commented 6 years ago

@ojw28 Thanks for the help!

We recently split the workaround you mention in two: codecNeedsSetOutputSurfaceWorkaround and codecNeedsDummySurfaceSurfaceWorkaround.

I believe this is present in dev-v2; yes indeed that seems like a better way to divide the generic workaround.

Does your app ever switch the surface that it outputs too?

No. I guess this means there isn't enough information to use the more targeted codecNeedsDummySurfaceSurfaceWorkaround workaround for these devices.

Is it possible for you to add a new column to this data that shows more specific Android build information? ... Build fingerprints (Build.FINGERPRINT) would be ideal.

Sorry, we currently don't have Build.FINGERPRINT or build name data. If you believe this information is critical to applying more targeted workarounds, we could additionally log this information on the affected devices in a future release, however this might take a couple of weeks to gather any significant data.

Is the build name available inside a Build class property, or does it need to be parsed out of the build fingerprint?

Also, I was wondering that if an application never needs to switch surfaces, is it ok to have codecNeedsDummySurfaceSurfaceWorkaround or codecNeedsSetOutputSurfaceWorkaround always true? Only a single Surface is involved and shouldInitCodec returns false when invoked anytime before surfaceCreated and invoked when surfaceDestroyed -> setSurface -> maybeInitCodec.

ojw28 commented 6 years ago

Sorry, we currently don't have Build.FINGERPRINT or build name data. If you believe this information is critical to applying more targeted workarounds, we could additionally log this information on the affected devices in a future release, however this might take a couple of weeks to gather any significant data.

Assuming it's relatively straightforward for you to add this to the data that's being logged, it would be great if you could! Unsure about the build name (by itself), but providing the full fingerprints would be ideal.

Also, I was wondering that if an application never needs to switch surfaces, is it ok to have codecNeedsDummySurfaceSurfaceWorkaround or codecNeedsSetOutputSurfaceWorkaround always true?

Yes, I think doing that would be fine for that use case. Alternatively, it's probably significantly easier for you just to call release() on the player from whatever happens first out of onStop and surfaceDestroyed (i.e. have both methods call a maybeReleasePlayer method, and release the player there if it's not been released already). This should also prevent the surface switch that you're currently seeing.

ojw28 commented 6 years ago

Some initial findings:

ValeryPonomarenko commented 6 years ago

@ojw28 we had the same issue on a Meizu Pro 7 phone (the device name is PRO7S). I added the device name inside the codecNeedsSetOutputSurfaceWorkaround method and now everything works well.

Could you add this device name too, please?

ojw28 commented 6 years ago

This should have been largely addressed in 2.8.3. Please let us know if you see further devices experiencing this issue using 2.8.3 or later (a list with occurrence counts, similar to the one above, would be great). Thanks!

qvk commented 6 years ago

Thanks! It might take a while to gather data with the 2.8.3 version though.

gapl commented 6 years ago

We're seeing this issue since updating to v2.8.4, so it seems this issue is still present. We don't have a list of occurrence counts per device yet as we didn't have this problem with an older version of ExoPlayer, but we have quite a significant count of these Exceptions. On Fabric alone we can see around 10,000 to be exact. Following information is what I was able to extract from Fabric's logged Exceptions.

We're using an instance of SimpleExoPlayer and we're not switching any surfaces. We are calling clearVideoSurface however when our player instance does not need to render video. This is when this exception occurs and prohibits audio-only playback.

Devices and their API levels from last several Exceptions being thrown:

Example log:

Non-fatal Exception: com.google.android.exoplayer2.ExoPlaybackException
       at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:354)
       at android.os.Handler.dispatchMessage(Handler.java:107)
       at android.os.Looper.loop(Looper.java:210)
       at android.os.HandlerThread.run(HandlerThread.java:61)

Caused by java.lang.IllegalArgumentException
       at android.media.MediaCodec.native_setSurface(MediaCodec.java)
       at android.media.MediaCodec.setOutputSurface(MediaCodec.java:1837)
       at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setOutputSurfaceV23(MediaCodecVideoRenderer.java:918)
       at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setSurface(MediaCodecVideoRenderer.java:408)
       at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.handleMessage(MediaCodecVideoRenderer.java:375)
       at com.google.android.exoplayer2.ExoPlayerImplInternal.deliverMessage(ExoPlayerImplInternal.java:861)
       at com.google.android.exoplayer2.ExoPlayerImplInternal.sendMessageToTarget(ExoPlayerImplInternal.java:829)
       at com.google.android.exoplayer2.ExoPlayerImplInternal.sendMessageInternal(ExoPlayerImplInternal.java:811)
       at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:328)
       at android.os.Handler.dispatchMessage(Handler.java:107)
       at android.os.Looper.loop(Looper.java:210)
       at android.os.HandlerThread.run(HandlerThread.java:61)
ojw28 commented 6 years ago

@gapl - I think we need better data (see e.g. the kind of data in the original post of this issue). In particular occurrence counts for each device (Util.DEVICE / Util.MODEL) are important.

gapl commented 6 years ago

We've since implemented the workaround by always returning true for codecNeedsSetOutputSurfaceWorkaround(:). So sadly I cannot provide additional data without pushing out a version that would not work for some of our users.

ojw28 commented 5 years ago

We're still adding the odd device to this list, but I think the bulk of the work here is done. We also made it easy for applications to override the method that decides whether the workaround is applied. I think it's fine to close at this point.

noamtamim commented 5 years ago

@ojw28 does Google plan to make ExoPlayer's OEM testing a mandatory part of Android certification? That should make sure this list doesn't get any longer...

ojw28 commented 5 years ago

A subset of the tests you refer to are already a mandatory part of Android certification, including the one covering this issue.

noamtamim commented 5 years ago

That's what I was hoping for, thanks.

jschamburger commented 5 years ago

Hi,

we got customer complaints which point to a similar problem when decoding DRM streams.

Caused by java.lang.IllegalArgumentException at android.media.MediaCodec.native_setSurface(MediaCodec.java) at android.media.MediaCodec.setOutputSurface(MediaCodec.java:1975) at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setOutputSurfaceV23(SourceFile:1014) at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.setSurface(SourceFile:422) at com.google.android.exoplayer2.video.MediaCodecVideoRenderer.handleMessage(SourceFile:387) at com.google.android.exoplayer2.ExoPlayerImplInternal.deliverMessage(SourceFile:871) at com.google.android.exoplayer2.ExoPlayerImplInternal.sendMessageToTarget(SourceFile:842) at com.google.android.exoplayer2.ExoPlayerImplInternal.sendMessageInternal(SourceFile:824) at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(SourceFile:333) at android.os.Handler.dispatchMessage(Handler.java:105) at android.os.Looper.loop(Looper.java:166) at android.os.HandlerThread.run(HandlerThread.java:65)

Device info: Build.MANUFACTURER: HUAWEI Build.MODEL: EML-L29 Android version: 8.1.0

Since this ticket is closed, should we enable the workaround for this device ourselves or do you guys still add devices to the list?

ojw28 commented 5 years ago

We can enable the workaround for that device (end EML-L09, which looks identical).

jschamburger commented 5 years ago

Perfect, thanks!