Frame drop when changed playback speed

[noranekotan]

Frame drop occurs when the playback speed is changed. For example, playing "Youtube DASH" > "Google Glass H264(MP4)" causes a frame drop.

Is there anything I can do to improve it?

Confirmation environment

package : com.google.android.exoplayer2.demo branch : release-v2、dev-v2

Changes

com.google.android.exoplayer2.demo.PlayerActivity

player =
          new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
              .setTrackSelector(trackSelector)
              .build();
      player.addListener(new PlayerEventListener());
      player.setAudioAttributes(AudioAttributes.DEFAULT, /* handleAudioFocus= */ true);
      player.setPlayWhenReady(startAutoPlay);
      player.addAnalyticsListener(new EventLogger(trackSelector));
      player.setPlaybackParameters(new PlaybackParameters(1.5F)); ← Added

[ueno-yuhei]

I have also confirmed that this phenomenon occurs. May be caused by video.

[tonihei]

I can't reproduce any dropped frames when setting a different playback speed. Not even if I increase the speed to 3.0f for example. What device are you using? If the device is too slow with decoding, frame drops might be expected for higher speeds if the video decoder can't decode fast enough to keep up with the fast audio.

[noranekotan]

@tonihei Thank you for your answer.

This issue has been reproduced on some Android TV devices. I also confirmed that Fire TV will drop frames. As far as I can see, it doesn't seem to drop frames on mobile devices.

And I wrote that the playback speed was set to 1.5 times as an implementation example, but frame drops occur even at 1.1 times etc.

Confirmed device

• AQUOS-4KTVJ17 LC50US5 Android 8.0.0
• BRAVIA 4K 2015 Android 7.0.0

[tonihei]

Thanks for the update. So it seems this issue is indeed device-specific.

In the original problem description you mention "a frame drop". Is that only a single dropped frame right at the beginning of playback? Or is that a continuous problem during playback? I'm trying to understand if this is clear user-visible problem or more of a technical glitch that is only visible in logs.

In any case, I'm afraid I can't really debug or reproduce this problem at the moment because the BRAVIA 4K device we have is in the office and thus unreachable as long as we work from home.

[noranekotan]

Thanks for the update.

Frame drop is continuous problem during playback and this is clear user-visible problem.

[andrewlewis]

When playing back at 1.5 speed on Sony/BRAVIA_ATV2_EU/BRAVIA_ATV2:8.0.0/OPR2.170623.027.S32/665551:user/release-keys in the demo app, video playback looks very 'jumpy' (like it's repeatedly speeding up then slowing down again). Audio sounds fine. The dropped frame counters in the demo app's debug view don't show frames getting dropped continuously.

I see the same on 2.10.0 so this doesn't seem to be a recent regression.

I tried various things to diagnose what was causing the jumpy playback, but didn't really get anywhere:

• Logging the position of the audio sink to see if it's jumping up. It was increasing smoothly.
• Disabling frame release time adjustment made no difference.
• Using the legacy frame release path (without nanosecond timing) made no difference.
• Disabling setting the codec operating rate.

Given that the player position is increasing smoothly, the problem probably lies with the timing of releasing frames. Maybe the decoder on this device is trying to do some automatic frame rate conversion and that is getting confused by the frame presentation times increasing too quickly. As this doesn't seem to be a regression, I'm afraid we probably won't get round to investigating further for a while.

[noranekotan]

@andrewlewis Thank you for your answer.

I understand that it's not a problem that will be solved soon. Please let us know if you have any useful settings or useful information for improvement.

[benoitdion]

hi @andrewlewis, do you have recommendations for how the encoding could be tweaked to minimize jumping to the next I-frame?

if audio isn't important, how would you recommend tweaking the decoder to drop frames at a consistent rate (e.g.: drop every 3 frames instead of jump to the next I-frame)?

[andrewlewis]

I'm not sure the jumping is to I-frames (the jumps seemed to be more frequent). The next thing I'd try is probably to tweak/scale the presentation times of frames sent to the video decoder to make them match real-time when using speed adjustment (and correcting them on the output side), going on the theory that the decoder is trying to do frame rate conversion and the non-realtime rate of video frames is confusing that logic.