Allow playback speed adjustments while tunneling

[rishun-fsi]

Hi, I am working on an Android TV application and I want to enable tunneling for the video playback to improve performance. But I encountered a problem by using set speed when I set tunnel mode on. Any way, It works fine, if tunnel mode off in this case. Please tell me how to avoid the following issue?

Issue description

Android TV (Adnroid 8.0.0, model:AQUOS_TVJ18, device:tcsu2uj)

Exo ver 2.8.4

test content: YouTube DASH / Google Glass (MP4,H264)

I have changed something on demos app for test below.

PlayerActivity (in the 183rd line)
//trackSelectorParameters = new DefaultTrackSelector.ParametersBuilder().build();
final DefaultTrackSelector.ParametersBuilder parametersBuilder = new DefaultTrackSelector.ParametersBuilder();
parametersBuilder.setTunnelingAudioSessionId(C.generateAudioSessionIdV21(this));
trackSelectorParameters = parametersBuilder.build();

//player.setPlayWhenReady(playWhenReady); (in the 28th line)
if(!playWhenReady && player.getPlaybackParameters().speed != 1.5f){
  player.setPlaybackParameters(new PlaybackParameters(1.5f, 1));
}
else {
  player.setPlaybackParameters(new PlaybackParameters(1.0f, 1));
}

When "player.setPlaybackParameters(new PlaybackParameters(1.0f, 1))" called, I get the following error.

E/EventLogger: playerFailed [6.99, 4.52, window=0]
               com.google.android.exoplayer2.ExoPlaybackException
                   at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:354)
                   at android.os.Handler.dispatchMessage(Handler.java:101)
                   at android.os.Looper.loop(Looper.java:164)
                   at android.os.HandlerThread.run(HandlerThread.java:65)
                Caused by: java.lang.IllegalStateException
                   at com.google.android.exoplayer2.util.Assertions.checkState(Assertions.java:81)
                   at com.google.android.exoplayer2.audio.DefaultAudioSink.writeBuffer(DefaultAudioSink.java:781)
                   at com.google.android.exoplayer2.audio.DefaultAudioSink.processBuffers(DefaultAudioSink.java:724)
                   at com.google.android.exoplayer2.audio.DefaultAudioSink.drainAudioProcessorsToEndOfStream(DefaultAudioSink.java:831)
                   at com.google.android.exoplayer2.audio.DefaultAudioSink.handleBuffer(DefaultAudioSink.java:647)
                   at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.processOutputBuffer(MediaCodecAudioRenderer.java:529)
                   at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:1097)
                   at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:582)
                   at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:518)
                   at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:301)
                   at android.os.Handler.dispatchMessage(Handler.java:101) 
                   at android.os.Looper.loop(Looper.java:164) 
                   at android.os.HandlerThread.run(HandlerThread.java:65)

Thanks!

[rishun-fsi]

DefaultControlDispatcher (in the 28th line) //player.setPlayWhenReady(playWhenReady); if(!playWhenReady && player.getPlaybackParameters().speed != 1.5f){ player.setPlaybackParameters(new PlaybackParameters(1.5f, 1)); } else { player.setPlaybackParameters(new PlaybackParameters(1.0f, 1)); }

[rishun-fsi]

same issue on BRAVIA 4K 2015 (Android 7.0.0)

[andrewlewis]

Speed adjustment can't be used with tunneling mode at the moment, I'm afraid. See the javadoc on DefaultAudioSink.

Marking as an enhancement to add support. In tunneling mode each buffer written to the AudioTrack needs to be prefixed by a timestamp, which the platform uses for hardware A/V synchronization.

[rishun-fsi]

Hi, I still want to use setSpeed function on our application. Here is one solution what I did. I turn off the tunnel mode when setSpeed 2.0f, whereas, turn on the tunnel mode when setSpeed 1.0f. I attached the demo app PlayerActivity which I changed some codes for testing(press pause key to change speed ). PlayerActivity.zip

It seems works fine, but always blinks with a black screen when I switch the SetSpeed between 1.0f and 2.0f. Is it possible to avoid the black screen in this case?

Test lib: ExoPlayer 2.8.4 Test device: BRAVIA_4K_GB Test content: SmoothingStreaming/Super speed

Thank you!

[andrewlewis]

@rishun-fsi I expect the black screen when switching between tunneled and untunneled playback is unavoidable (they are different output paths). You could try using a TextureView instead of SurfaceView for output but I doubt that will work with tunneled mode.

[ojw28]

@andrewlewis @krocard - This issue seems to imply that playback fails with an Assertion.checkState error if the player is configured for non-1x-playback-speed and tunnelled mode is enabled.

If we don't support the combination yet, then shouldn't we either fail in a clearer way, or disable audio processing as we do when audio offload is being used? The current failure mode (if it's what this issue implies it is) seems pretty confusing.

[krocard]

@andrewlewis @krocard - This issue seems to imply that playback fails with an Assertion.checkState error if the player is configured for non-1x-playback-speed and tunnelled mode is enabled.

Created b/170376195 to track this issue.

ExoPlayer 2.12.0 now supports (change 501f464) using the platform speed adjustment engine (for API 23+) which might or might not work with tunneling (I do not have a tunneling setup to test).

To enable using the platform speed adjustment engine instead of the bundled Sonic engine, call DefaultRenderersFactory#setEnableAudioTrackPlaybackParams(true) when constructing the player.

If that does not work, then tunneling can not be used with speed adjustment. Note that even if it works, you will probably be limited to low playback speed due to decoding hardware limitations.

[krocard]

If that does not work, then tunneling can not be used with speed adjustment. Note that even if it works, you will probably be limited to low playback speed due to decoding hardware limitations.

Actually, tunneling with ExoPlayer's time stretching engine* might be possible to implement by tweaking the video input buffer presentation time to force the video frames to render at the right time.

* without using the platform speed adjustment engine which can not be customized

This issue will track this change as low priority. Until this is implemented, I recommend to try to use the system adjustment engine as explained in my previous comment (https://github.com/google/ExoPlayer/issues/4803#issuecomment-705501297).