Switching Surface smoothly

[fabrantes]

This is actually more of a question rather than an issue.

Whenever you call MediaCodecVideoTrackRenderer.setSurface() with a new Surface, playback will stop until a new sync frame is received. I haven't looked too deep into the internals but I guess the buffers used by MediaCodec are obtained directly from a given surface so it needs to be reset. Is there any way of passing the current decoding state to a new MediaCodec/Surface?

The question is if it would be possible at all to avoid the playback gap at all and what modifications would it require to ExoPlayer.

Thanks!

[mick1418]

I'd be also interested in the answer to this question, as I'm trying to implement a fullscreen mode and need to change surface on the fly without an image gap.

[thomaspaulmann]

Hello @fabrantes,

what do you plan to do? What I do, is using two exo players with to surface/texture views and switching them is a lot faster then setting a new surface. Maybe this will help you.

[nguyentando]

Is there anyone figured out how to switch surface smoothly? I'd like to hear your solution.

[fabrantes]

At the moment I'm sharing the same SurfaceTexture on two TextureViews, that was the only decent approach I could find, but it requires some care with the View lifecycle. Still open to alternatives :smile:

[nguyentando]

So there is no solution with Android API < 14 (use SurfaceView instead of TextureView)? Because I can't find any methods like TextureView.setSurfaceView in SurfaceView class, so we can't share the same surface between 2 SurfaceView.

[ojw28]

What renderers are you using to target API levels under 14 anyway, given that the standard audio/video renderers require API level 16+? Note that only 4% of active Android devices are running API level 14 and below, and only 7% of active Android devices are running API level 15 and below.

[nguyentando]

I use MediaPlayer for API < 14. There are about 20% of my users use devices which run on Android < 14, so I simply can't ignore them.

[qqli007]

@fabrantes, @ojw28 ,I share the same SurfaceTexture,but when goto another TextureView ,it only has voice no video.Do you have any idea about it?

when goto another TextureView, I used like this to replay:

           if (surfaceTextureSaved != null) {
                Surface surface = new Surface(surfaceTextureSaved);
                player.setSurface(surface);
                player.setPlayWhenReady(true);
            }

[luckcoolla]

Hello, Guys, I have solved similar problem with workaround. I need to change surface for player during Activity switch (from ActivityA to ActivityB) on player paused state. Player had no any video output before I will call player.start() (or exoPlayer.setPlayWhenReady(true)).

My code for activity switch:

        //prepare to ActivityA destroy
        videoTrackToRestore = getSelectedTrackIndex(TYPE_VIDEO);
        selectTrack(TYPE_VIDEO, DISABLED_TRACK);
        blockingClearSurface();
        ...
        ...
        //surface on ActivityB prepared:
        selectTrack(TYPE_VIDEO, videoTrackToRestore);
        setSufraceView(surfaceView);

And this code workaround has helped me to make exo player render video picture on the new surface on paused state:

        savedPlayer.setPosition(savedPlayer.getPosition());

[razcakappa]

@fabrantes Hi, I'd like to discuss about the Cubed (3) player project. How shall we proceed?a

[ojw28]

We've made one minor improvement in dev-v2, which is to enable fast switching on API level 23 and above in the case of switching directly from one surface to another (i.e. not with a gap where there's no surface in between). Of course this is far away from being a complete solution, but it might be useful to some people. The relevant change is here: https://github.com/google/ExoPlayer/commit/a9617af29c2c760c8e439d390ef02f0b4e055a6b

[molexx]

Thanks @ojw28 - I have switching SurfaceView when zooming to fullscreen mostly working using that change.

I'm still having fun with rotation. In #1084 you noted:

It may be that setOutputSurface provides a more efficient solution from API level 23, although it's still slightly awkward if you don't have a surface at all for some period of time during the transition.

what were your thoughts towards 'slightly awkward' options please? Whilst I can now switch SurfaceViews it seems that both of them need to be in layout to avoid the player throwing an exception and releasing the codec, and whilst the Activity rotates there's no layout.

It would be ok to pause playback for the duration of the rotate - if the playback were actually paused (do I need to wait for playWhenReadyCommitted?) could the surface be swapped without releasing the codec in a similar way to a9617af and it not get upset when the previous surface is destroyed with the Activity?

[ojw28]

It's possible to implement a "dummy surface" that can be attached during the period where you otherwise wouldn't have one. It's possible to use this approach for non DRM'd content and DRM'd content that doesn't require a secure output path in its license policy. For DRM'd content that does require a secure output path, it's possible to use this approach only if the device supports EGL_EXT_protected_content, as otherwise the dummy surface isn't sufficiently secure.

We're aiming to push a change that adds a DummySurface class fairly soon. We may later automatically attach it when possible inside of MediaCodecVideoRenderer.

[ojw28]

Remaining work to do here is as follows:

• Investigate hooking DummySurface up internally where possible.
• Don't allow MediaCodecVideoTrackRenderer to decode/render a key-frame too far ahead of the current playback position. This causes bad behavior if the surface is repeatedly cleared and set, as described in https://github.com/google/ExoPlayer/issues/2703.

[molexx]

Thanks very much for this, I have fullscreen and rotation working well with a combination of 'fast surface switching on API level 23+' and DummySurface. Below 23 I think the best we can do is wait for a keyframe.

Please let the YouTube SDK guys know about this :-)

[PaulWoitaschek]

I implemented the workaround suggested in #2703 for Devices < 23:

class SurfaceManagerApi17(private val player: SimpleExoPlayer, private val trackSelector: DefaultTrackSelector) : SurfaceManager {

  override fun surfaceCreated(holder: SurfaceHolder) {
    player.setVideoSurface(holder.surface)
    videoRendererIndex()?.let {
      trackSelector.setRendererDisabled(it, false)
    }
  }

  override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {}

  override fun surfaceDestroyed(holder: SurfaceHolder) {
    player.setVideoSurface(null)
    videoRendererIndex()?.let {
      trackSelector.setRendererDisabled(it, true)
    }
  }

  private fun videoRendererIndex() = (0 until player.rendererCount)
      .firstOrNull {
        player.getRendererType(it) == C.TRACK_TYPE_VIDEO
      }

  override fun release() {
    player.setVideoSurface(null)
  }
}

That works fine when the surface gets destroyed but when it gets recreated, the audio pauses for a short time which is very disturbing.

Is it possible to prevent that?

[ojw28]

@molexx - In the dev-v2 branch we now wire DummySurface up for you automatically where possible inside of MediaCodecVideoRenderer, so you'll be able to remove your manual wiring once that hits a release-v2 (or if you're using dev-v2 directly).

[michalliu]

Thanks for all the efforts to resolve this problem. It's a device specific problem. My HuaWei phone works just fine while others not.

[molexx]

If the user has paused the video before surface replacement the new surface stays black rather than showing the paused frame. It resumes fine when pressing play. What's a clean way to set the paused frame image please?

[rantianhua]

I have a doubt about DummySurface. In your guys previous discussion, I get the information that we can use DummySurface to hold MediaCodec after surface is destroyed since api 17. But in the codes as follows:

https://github.com/google/ExoPlayer/blob/e7c60a2a234ab11bc75335453a7836fef9509610/library/core/src/main/java/com/google/android/exoplayer2/video/MediaCodecVideoRenderer.java#L753-L756

My question is that why the api should be larger than 23 if the DummySurface is designed to hold MediaCodec since api 17?

[ojw28]

DummySurface is designed to hold MediaCodec since api 17

It's not. Use of DummySurface for this purpose requires MediaCodec.setOutputSurface, which was only added in API level 23. DummySurface may have other uses on earlier API levels (although in practice, I can't really think of any).

[rantianhua]

@ojw28 Okay, I get it.

[AfzalivE]

Simpler workaround for anyone who had this problem when switching activities:

If you hide the player in onPause(), the black box will not show up. Sorry for Kotlin example

 override fun onPause() {
        super.onPause()
        // workaround for https://github.com/google/ExoPlayer/issues/677
        // this hides the playerView when the activity is paused
        // and video is not playing, avoids showing the black box
        if (player?.playWhenReady == false) {
            playerView?.visibility = View.GONE
        }
}

and then set the visible to View.VISIBLE right before you want to show it (and NOT in onResume()). I have a view in front of it to allow fading the player view in/out as well but I don't think that affects this logic.

        // workaround for https://github.com/google/ExoPlayer/issues/677
        // This shows the playerView if it's not already visible,
        // without the check, the surfaceView becomes a black box
        // briefly before showing the video
        if (playerView.visibility != View.VISIBLE) {
            playerView?.visibility = View.VISIBLE
        }

Side note: Maybe there should be a check in SimpleExoPlayerView#setVisibility if you're setting the same visibility as current.

[ghost]

@fabrantes, @ojw28 ,I share the same SurfaceTexture,but when goto another TextureView ,it only has voice no video.Do you have any idea about it?

when goto another TextureView, I used like this to replay:

           if (surfaceTextureSaved != null) {
                Surface surface = new Surface(surfaceTextureSaved);
                player.setSurface(surface);
                player.setPlayWhenReady(true);
            }

Kindly give full solution with proper class Name

[MostafaAnter]

Note, TextureView can only be used in a hardware accelerated window. When rendered in software, TextureView will draw nothing. so after set surface type

<com.google.android.exoplayer2.ui.PlayerView android:id="@+id/player_view"
 app:surface_type="texture_view"
 android:layout_width="match_parent"
 android:layout_height="match_parent"/>

you need to make sure that hardware acceleration is enabled, go to manifest file and ad this line

• At application level: <application android:hardwareAccelerated="true" ...>