Dropped frames on Fire TV 4K

[MarcusWichelmann]

Hi,

when playing a HLS H.265/HEVC stream on a Fire TV 4K, the playback is very stuttering and the decoder seems to be dropping a lot of frames. This can also be seen in logcat where a lot of com.android.mediaplayer.HW.video.hevc Dropping frame messages appear just a few seconds after the stream has started playing.

But when playing the same stream using the VLC app for Fire TV, the playback is smooth and the log output clean, so I suppose that VLC for Android passes some parameters to libvlc that I don't. But I have not yet discovered which these are.

Maybe you have an idea what options I'm missing?

My simple test code (the commented out options were some unsuccessful tests from me):

import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:flutter_vlc_player/flutter_vlc_player.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  final Map<LogicalKeySet, Intent> _shortcuts = {
    LogicalKeySet(LogicalKeyboardKey.select): const ActivateIntent()
  };

  @override
  Widget build(BuildContext context) {
    return Shortcuts(
      shortcuts: _shortcuts,
      child: MaterialApp(
        title: 'Test',
        theme: ThemeData(
          primarySwatch: Colors.blue,
        ),
        home: MyHomePage(),
      ),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key}) : super(key: key);

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  late VlcPlayerController _videoPlayerController;

  @override
  void initState() {
    super.initState();

    _videoPlayerController = VlcPlayerController.network(
      'http://<private server>/stream.m3u8',
      hwAcc: HwAcc.AUTO, // Also tried FULL, didn't help.
      autoPlay: true,
      options: VlcPlayerOptions(
//      advanced: VlcAdvancedOptions([VlcAdvancedOptions.clockSynchronization(0)]),
//      extras: ["--android-display-chroma=RV32", "--aout=opensles"]
      ),
    );
  }

  @override
  void dispose() async {
    super.dispose();
    await _videoPlayerController.stopRendererScanning();
    await _videoPlayerController.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(),
      body: Center(
        child: VlcPlayer(
          controller: _videoPlayerController,
          aspectRatio: 16 / 9,
          placeholder: Center(child: CircularProgressIndicator()),
        ),
      ),
    );
  }
}

The stuttering is reproducable with above code as well as with the sample application from this repository (when changing the code to use my stream). Interestingly, this is not reproducable with this stream: http://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel.ism/.m3u8.

This issue is also only present on the Fire TV and the stream runs fine on e.g. an Android Smartphone.

Information about my stream:

Input #0, hls, from 'http://<private server>/stream.m3u8':
  Duration: N/A, start: 83559.964622, bitrate: N/A
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 256 kb/s
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 256 kb/s
    Metadata:
      variant_bitrate : 0
    Stream #0:2: Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, stereo, fltp, 448 kb/s
    Metadata:
      variant_bitrate : 0
    Stream #0:3: Video: hevc (Main) (HEVC / 0x43564548), yuv420p(tv), 1280x720 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
    Metadata:
      variant_bitrate : 0

Log output:

E/IMGSRV  (13893): :0: IsTextureConsistent: IMGEGLImage is not consistent
W/libOpenSLES(13893): class OutputMix interface 0 requested but unavailable MPH=43
E/libc    (13893): Access denied finding property "dolby.dma.minbufcnt"
D/AudioTrack(13893): min buf count is limited to 2 times HAL buf
D/AmazonAudioTrackCallback(13893): No AudioTrackFlags provided.  Using internal defaults.
D/AmazonAudioTrackCallback(13893): No Audio content type provided.
I/android.media.AudioTrack(13893): AUDIOINFO: audio_input: format: 2 sample_rate: 48000 channels: 2 AudioAttributes: AudioAttributes: usage=1 content=2 flags=0x0 tags= bundle=null
E/libc    (13893): Access denied finding property "dolby.dma.minbufcnt"
D/AudioTrack(13893): min buf count is limited to 2 times HAL buf
D/AudioTrack(13893): Client defaulted notificationFrames to 1026 for frameCount 3080
W/VideoCapabilities(13893): Unrecognized profile/level 1/32 for video/mp4v-es
W/VideoCapabilities(13893): Unrecognized profile/level 32768/2 for video/mp4v-es
W/VideoCapabilities(13893): Unrecognized profile/level 32768/64 for video/mp4v-es
W/android_audiotr(13893): type=1400 audit(0.0:514): avc: denied { read } for name="u:object_r:dolby_prop:s0" dev="tmpfs" ino=12300 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:dolby_prop:s0 tclass=file permissive=0
W/android_audiotr(13893): type=1400 audit(0.0:515): avc: denied { read } for name="u:object_r:dolby_prop:s0" dev="tmpfs" ino=12300 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:dolby_prop:s0 tclass=file permissive=0
W/VideoCapabilities(13893): Unsupported mime video/dolby-vision
W/VideoCapabilities(13893): Unsupported mime video/dolby-vision
W/VideoCapabilities(13893): Unsupported mime video/dolby-vision
W/VideoCapabilities(13893): Unsupported mime video/dolby-vision
W/VideoCapabilities(13893): Unsupported mime video/dolby-vision
W/VideoCapabilities(13893): Unsupported mime video/dolby-vision
W/VideoCapabilities(13893): Unrecognized profile/level 0/0 for video/mpeg2
W/VideoCapabilities(13893): Unrecognized profile/level 0/2 for video/mpeg2
W/VideoCapabilities(13893): Unrecognized profile/level 0/3 for video/mpeg2
I/VideoCapabilities(13893): Unsupported profile 4 for video/mp4v-es
E/VLC     (13893): [849ca3b0/374d] libvlc window: request 1 not implemented
E/VLC     (13893): [849ca3b0/374d] libvlc window: request 1 not implemented
E/VLC     (13893): [8492c3b0/374d] libvlc gl: shader 0: Success.
E/VLC     (13893): [8492c3b0/374d] libvlc gl: shader 1: Success.
E/VLC     (13893): [8492c3b0/374d] libvlc gl: shader 0: Success.
E/VLC     (13893): [8492c3b0/374d] libvlc gl: shader 1: Success.
E/VLC     (13893): [849ca3b0/374d] libvlc window: request 0 not implemented
E/VLC     (13893): [849ca3b0/374d] libvlc window: request 0 not implemented
E/VLC     (13893): [849ca3b0/374d] libvlc window: request 1 not implemented
I/OMXClient(13893): MuxOMX ctor
I/MediaCodec(13893): [OMX.MTK.VIDEO.DECODER.HEVC] setting surface generation to 14226433
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc P2 + 88756705
I/ACodec_vendorExt(13893): Setting OMX_IndexParamVideoHWComposed
W/ACodec_vendorExt(13893): Error OMX_IndexParamVideoHWComposed: Unknown error 1010 (1010)
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc P2 - 88756741
D/MediaCodecLogger(13893): Got a Service. Calling PID = 13893
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc P3 + 88756745
D/SurfaceUtils(13893): set up nativeWindow 0x8bebc008 for 1280x736, color 0x7f000001, rotation 0, usage 0x2933
W/ACodec  (13893): [OMX.MTK.VIDEO.DECODER.HEVC] setting nBufferCountActual to 7 failed: -2147483648
W/ACodec  (13893): [OMX.MTK.VIDEO.DECODER.HEVC] setting nBufferCountActual to 6 failed: -2147483648
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc P3 - 88756764
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc Got First Input Frame 88756764
W/AHierarchicalStateMachine(13893): Warning message AMessage(what = 'omxI') = {
W/AHierarchicalStateMachine(13893):   int32_t type = 0
W/AHierarchicalStateMachine(13893):   int32_t event = 3
W/AHierarchicalStateMachine(13893):   int32_t data1 = 1
W/AHierarchicalStateMachine(13893):   int32_t data2 = 117440527
W/AHierarchicalStateMachine(13893): } unhandled in root state.
D/SurfaceUtils(13893): set up nativeWindow 0x8bebc008 for 1920x1088, color 0x7f000001, rotation 0, usage 0x2933
W/ACodec  (13893): [OMX.MTK.VIDEO.DECODER.HEVC] setting nBufferCountActual to 9 failed: -2147483648
W/ACodec  (13893): [OMX.MTK.VIDEO.DECODER.HEVC] setting nBufferCountActual to 8 failed: -2147483648
I/MediaCodecLogger(13893): updateFormatChanged width = 1280 height = 720
E/VLC     (13893): [83c703b0/3758] libvlc decoder: output: 2130706433 unknown, 1920x1088 stride 1920 1088, crop 0 0 0 0
I/MediaCodecLogger(13893): updatePTSTime [HW.video.hevc] First PTS after Flush or reset = 81493964623
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc Got First Frame Ready 88756882
W/AudioTrack(13893): releaseBuffer() track 0x88a3fc00 disabled due to previous underrun, restarting
I/MediaCodecLogger(13893): [ASAP] HW.video.hevc Got First Frame Render 88756886
I/MediaCodecLogger(13893): App uses Timestamps for AVSync
I/MTK_IMGX(13893): MTK EGL_IMGX Max (2147483647 x 2147483647)
E/VLC     (13893): [849ca3b0/374d] libvlc window: request 3 not implemented
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc.bitrateInKbps = 3733
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Resolution change from 0x0 to 1280x720 took = 4
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame
I/MediaCodecLogger(13893): com.android.mediaplayer.HW.video.hevc Dropping frame

[MarcusWichelmann]

I've now hooked a debugger into VLC for Android and checked which options it passes to libvlc to pass the same from my app. But had no luck, the playback is still laggy.

Parameters:

"--audio-time-stretch",
"--avcodec-skiploopfilter",
"1",
"--avcodec-skip-frame",
"0",
"--avcodec-skip-idct",
"0",
"--subsdec-encoding",
"",
"--stats",
"--android-display-chroma",
"RV16",
"--audio-resampler",
"soxr",
"--audiotrack-session-id=1985",
"--freetype-rel-fontsize=16",
"--freetype-color=16777215",
"--freetype-background-opacity=0",
"-vv",
"--no-sout-chromecast-audio-passthrough",
"--sout-chromecast-conversion-quality=2",
"--sout-keep",
"--preferred-resolution=-1",
"--aout=opensles",

The extended logging now shows some additional hints what might be the issue:

D/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture might be displayed late (missing 9 ms)
W/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture is too late to be displayed (missing 33 ms)
I/MediaCodecLogger( 2954): com.android.mediaplayer.HW.video.hevc Dropping frame
W/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture is too late to be displayed (missing 13 ms)
I/MediaCodecLogger( 2954): com.android.mediaplayer.HW.video.hevc Dropping frame
D/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture might be displayed late (missing 0 ms)
W/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture is too late to be displayed (missing 28 ms)
I/MediaCodecLogger( 2954): com.android.mediaplayer.HW.video.hevc Dropping frame
W/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture is too late to be displayed (missing 10 ms)
I/MediaCodecLogger( 2954): com.android.mediaplayer.HW.video.hevc Dropping frame
D/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture might be displayed late (missing 1 ms)
W/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture is too late to be displayed (missing 23 ms)
I/MediaCodecLogger( 2954): com.android.mediaplayer.HW.video.hevc Dropping frame
D/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture might be displayed late (missing 4 ms)
W/VLC     ( 2954): [84dfb930/d0e] libvlc video output: picture is too late to be displayed (missing 10 ms)
...

But I still don't know how to fix this. :/

[alr2413]

Did you test with HwAcc.Disabled? Also i googled about your problem, it seems that this issue is related to Fire TV device not the plugin.

Check this link https://emby.media/community/index.php?/topic/27322-stutteringchoppy-playback-lately-on-firetv-stick/ --> "The Stick has very limited processing power so anything that requires processing power to decode could be an issue. For this reason, we don't allow the use of direct stream with VLC on the stick for any HD content."

[MarcusWichelmann]

Thanks for your fast response.

When disabling HwAcc, the stream still stutters the same (looks like ~15 FPS) but the HW decoder errors in logcat disappeared (of course).

Thanks for your link. But I'm not sure if this statement from 2015 is still relevant for the current generations of Fire TVs. Also, the stream runs absolutely fine in the official VLC for Android app, just not when embedding libvlc into flutter using this library. So maybe the embedding into Flutter introduces too much overhead?

Maybe it's worth to mention, that logcat is spammed with Invalid display messages as long as the app is run in release-mode on the FireTV. These messages aren't there when running in debug-mode. But I'm not sure, if this is any related.

04-01 17:29:16.902   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:16.917   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:16.935   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:16.954   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:16.967   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:16.984   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.000   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.019   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.036   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.053   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.069   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.094   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.119   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.137   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.186   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.202   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.222   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.234   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.253   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.267   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.289   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.304   241   241 E HWComposer: getLayerReleaseFence: Invalid display
04-01 17:29:17.324   241   241 E HWComposer: getLayerReleaseFence: Invalid display
...

[alr2413]

You said "This issue is also only present on the Fire TV and the stream runs fine on e.g. an Android Smartphone.". Did you test your app on an android smartphone too?

"So maybe the embedding into Flutter introduces too much overhead?" No, in flutter we only pass the parameters to the android platform.

[MarcusWichelmann]

Yeah, I ran the app on two Android Smartphones (bq Aquaris X, Pixel 4a 5G, smooth playback), on an Android TV emulator (smooth, too) and on the Fire TV 4K (stuttering). A friend also tried it on a Sony TV with Android TV (KD55XG8505) where he saw stuttering too, but less extreme. And I've just received a Chromecast with Google TV which I can test the app on on tuesday.

If you don't think that this issue could be caused by rendering overhead, then I'm really out of ideas now regarding what could be the difference between the Flutter VLC Player and VLC for Android.

[alr2413]

please share the streaming url so we can test that.

[dragonfly-d]

We have same problem on MiBox4 issue 209

[pharshdev]

@MarcusWichelmann i had the same issue https://github.com/solid-software/flutter_vlc_player/issues/32, spent countless hours fiddling with vlc params but nothing worked. Finally i gave up on this plugin and used the amazon's exoplayer port which is well optimised to work with fire devices.

[MarcusWichelmann]

@alr2413 Unfortunately, I cannot share the streams I used for testing, but this should be reproducable with any HD/4K H.265 Stream.

@pharshdev Do you use ExoPlayer with Flutter or directly in Kotlin/Java? I noticed that, on some Android TVs, ExoPlayer in Flutter (video_player) has similar laggs/low framerate like described above. But when using ExoPlayer without Flutter, the same devices were well able to play even 4K 60 FPS HDR streams smoothly without issues. It looks like that there is some optimization missing on the Flutter side because the issue seems to be not limited to only flutter_vlc_player.

I decided to skip Flutter for now and write the app with native Android and the Java ExoPlayer APIs instead, because that way the videos will even play smoothly on low-end devices.

[pharshdev]

@MarcusWichelmann yes i used the exoplayer natively. For me flutter + native exoplayer works for now and i dont have to maintain separate code for mobile and tv devices.

[MarcusWichelmann]

@pharshdev So you embedded the ExoPlayer into a Flutter AndroidView? That's the same that video_player does afaik and that didn't perform very well on some devices I tested it on.

[pharshdev]

@MarcusWichelmann nope, no embedding. I simply launch the PlayerActivity from the FlutterActivity with all the params necessary to play the video from the flutter side. Once the playback is done, I'm back to Flutter goodness.

[MarcusWichelmann]

@pharshdev Aah good idea, thank you. I'll see if that works for me.

[alr2413]

@MarcusWichelmann @pharshdev, if you don't need to access any player parameters on flutter side that solution might work well.

[jeffmikels]

@MarcusWichelmann nope, no embedding. I simply launch the PlayerActivity from the FlutterActivity with all the params necessary to play the video from the flutter side. Once the playback is done, I'm back to Flutter goodness.

Can you show a code example?

[MarcusWichelmann]

@jeffmikels I resolved this by showing a transparent Flutter app using a FlutterFragment as an overlay for the ExoPlayer view. See: https://flutter.dev/docs/development/add-to-app/android/add-flutter-fragment Communication with the player instance happens through method channels.

This strategy also works on iOS, even thought it's way less documented.

Unfortunately, I cannot share any code right now, but it should be fairly straight forwarded to implement with above documentation.

[Fernker]

@pharshdev Do you happen to have an example of this? I've been hitting my head against a wall to get 4k60fps video working in Flutter (StackOverflow issue)

This might be the final breakthrough I need to getting this working for my scenario.

[jaydip-neon]

Hello, I'm working with the tv and ExoPlayer with native Kotlin and I'm not streaming media just playing from local I'm facing the same issue with FireStick 4K it crashes within 3 to 4 minutes and in FireStick Lite it's working up to 15 to 30 munites.