[Bug Report][Android] Live FLV rendering causing issue

[liuchuancong]

windows 11 works well,but on android device bad work

[alexmercerind]

Provide sample.

[liuchuancong]

simple code

 Widget _buildVideoFrame() {
    return media_kit_video.Video(
          key: widget.controller.key,
          controller: widget.controller.controller,
           filterQuality:FilterQuality.high,
          fit: widget.controller.videoFit.value,
          controls: (state) => _buildVideoPanel(),
        );
  }

import 'dart:async';

import 'dart:io';

import 'package:battery_plus/battery_plus.dart';
import 'package:flutter_barrage/flutter_barrage.dart';
import 'package:get/get.dart';
import 'package:media_kit/media_kit.dart';
import 'package:media_kit_video/media_kit_video.dart' as media_kit_video;
import 'package:pure_live/common/index.dart';
import 'package:screen_brightness/screen_brightness.dart';
import 'package:flutter_volume_controller/flutter_volume_controller.dart';
import 'package:wakelock_plus/wakelock_plus.dart';

import 'danmaku_text.dart';
import 'video_controller_panel.dart';

class VideoController with ChangeNotifier {
  final GlobalKey playerKey;
  final LiveRoom room;
  final String datasourceType;
  String datasource;
  final bool allowBackgroundPlay;
  final bool allowScreenKeepOn;
  final bool allowFullScreen;
  final bool fullScreenByDefault;
  final bool autoPlay;
  final videoFit = BoxFit.contain.obs;

  // Video player status
  // A [GlobalKey<VideoState>] is required to access the programmatic fullscreen interface.
  late final GlobalKey<media_kit_video.VideoState> key =
      GlobalKey<media_kit_video.VideoState>();
  // Create a [Player] to control playback.
  late final player = Player();
  // Create a [VideoController] to handle video output from [Player].
  late final controller = media_kit_video.VideoController(player, configuration: const media_kit_video.VideoControllerConfiguration(
    enableHardwareAcceleration: true
  ));
  ScreenBrightness brightnessController = ScreenBrightness();

  final hasError = false.obs;
  final isPlaying = false.obs;
  final isBuffering = false.obs;
  final isPipMode = false.obs;
  final isFullscreen = false.obs;
  final isWindowFullscreen = false.obs;
  bool get supportPip => Platform.isAndroid;
  bool get supportWindowFull => Platform.isWindows || Platform.isLinux;

  bool get fullscreenUI => isFullscreen.value || isWindowFullscreen.value;

  // Controller ui status
  Timer? showControllerTimer;
  final showController = true.obs;
  final showSettting = false.obs;
  final showLocked = false.obs;
  bool playBackisPlaying = false;
  void enableController() {
    showControllerTimer?.cancel();
    showControllerTimer = Timer(const Duration(seconds: 2), () {
      showController.value = false;
    });
    showController.value = true;
  }

  // Timed shutdown control
  final shutdownMinute = 0.obs;
  Timer? _shutdownTimer;
  void setShutdownTimer(int minutes) {
    showControllerTimer?.cancel();
    _shutdownTimer?.cancel();
    shutdownMinute.value = minutes;
    if (minutes == 0) return;
    _shutdownTimer = Timer.periodic(const Duration(minutes: 1), (timer) {
      shutdownMinute.value--;
      if (shutdownMinute.value == 0) exit(0);
    });
  }

  VideoController({
    required this.playerKey,
    required this.room,
    required this.datasourceType,
    required this.datasource,
    this.allowBackgroundPlay = false,
    this.allowScreenKeepOn = false,
    this.allowFullScreen = true,
    this.fullScreenByDefault = false,
    this.autoPlay = true,
    BoxFit fitMode = BoxFit.contain,
  }) {
    videoFit.value = fitMode;
    if (allowScreenKeepOn) WakelockPlus.enable();
    initVideoController();
    initDanmaku();
    initBattery();
  }

  // Battery level control
  final Battery _battery = Battery();
  final batteryLevel = 100.obs;
  void initBattery() {
    if (!Platform.isWindows) {
      _battery.batteryLevel.then((value) => batteryLevel.value = value);
      _battery.onBatteryStateChanged.listen((state) async {
        batteryLevel.value = await _battery.batteryLevel;
      });
    }
  }

  void initVideoController() {
    FlutterVolumeController.showSystemUI = false;
    setDataSource(datasource);
    player.stream.playing.listen((bool playing) {
      if (playing) {
        isPlaying.value = true;
      } else {
        isPlaying.value = false;
      }
    });
     // fix auto fullscreen
      if (fullScreenByDefault && datasource.isNotEmpty) {
        Timer(const Duration(milliseconds: 500), () => toggleFullScreen());
      }
  }

  // Danmaku player control
  final danmakuController = BarrageWallController();
  final hideDanmaku = false.obs;
  final danmakuArea = 1.0.obs;
  final danmakuSpeed = 8.0.obs;
  final danmakuFontSize = 16.0.obs;
  final danmakuFontBorder = 0.5.obs;
  final danmakuOpacity = 1.0.obs;

  void initDanmaku() {
    hideDanmaku.value = PrefUtil.getBool('hideDanmaku') ?? false;
    hideDanmaku.listen((data) {
      PrefUtil.setBool('hideDanmaku', data);
    });
    danmakuArea.value = PrefUtil.getDouble('danmakuArea') ?? 1.0;
    danmakuArea.listen((data) {
      PrefUtil.setDouble('danmakuArea', data);
    });
    danmakuSpeed.value = PrefUtil.getDouble('danmakuSpeed') ?? 8;
    danmakuSpeed.listen((data) {
      PrefUtil.setDouble('danmakuSpeed', data);
    });
    danmakuFontSize.value = PrefUtil.getDouble('danmakuFontSize') ?? 16;
    danmakuFontSize.listen((data) {
      PrefUtil.setDouble('danmakuFontSize', data);
    });
    danmakuFontBorder.value = PrefUtil.getDouble('danmakuFontBorder') ?? 0.5;
    danmakuFontBorder.listen((data) {
      PrefUtil.setDouble('danmakuFontBorder', data);
    });
    danmakuOpacity.value = PrefUtil.getDouble('danmakuOpacity') ?? 1.0;
    danmakuOpacity.listen((data) {
      PrefUtil.setDouble('danmakuOpacity', data);
    });
  }

  void sendDanmaku(LiveMessage msg) {
    if (hideDanmaku.value) return;
    danmakuController.send([
      Bullet(
        child: DanmakuText(
          msg.message,
          fontSize: danmakuFontSize.value,
          strokeWidth: danmakuFontBorder.value,
          color: Color.fromARGB(255, msg.color.r, msg.color.g, msg.color.b),
        ),
      ),
    ]);
  }

  @override
  void dispose() {
    if (allowScreenKeepOn) WakelockPlus.disable();
    _shutdownTimer?.cancel();
    brightnessController.resetScreenBrightness();
    danmakuController.dispose();
    player.dispose();
    super.dispose();
  }

  void refresh() {
    setDataSource(datasource);
  }

  void setDataSource(String url) {
    datasource = url;
    // fix datasource empty error
    if (datasource.isEmpty) {
      hasError.value = true;
      return;
    }
    player.open(Media(datasource));
  }

  void setVideoFit(BoxFit fit) {
    videoFit.value = fit;
    notifyListeners();
  }

  void togglePlayPause() {
    player.playOrPause();
  }

  void toggleFullScreen() {
    // disable locked
    showLocked.value = false;
     // fix danmaku overlap bug
    if (!hideDanmaku.value) {
      hideDanmaku.value = true;
      Timer(const Duration(milliseconds: 500), () {
        hideDanmaku.value = false;
      });
    }
    // fix obx setstate when build
    showControllerTimer?.cancel();
    Timer(const Duration(milliseconds: 500), () {
      enableController();
    });
    if (key.currentState?.isFullscreen() ?? false) {
      key.currentState?.exitFullscreen();
    } else {
      key.currentState?.enterFullscreen();
    }
    isFullscreen.toggle();
  }

  void toggleWindowFullScreen() {
    // disable locked
    showLocked.value = false;
    // fix danmaku overlap bug
    if (!hideDanmaku.value) {
      hideDanmaku.value = true;
      Timer(const Duration(milliseconds: 500), () {
        hideDanmaku.value = false;
      });
    }
    // fix obx setstate when build
    showControllerTimer?.cancel();
    Timer(const Duration(milliseconds: 500), () {
      enableController();
    });

    if (Platform.isWindows || Platform.isLinux) {
      if (!isWindowFullscreen.value) {
        Get.to(() => DesktopFullscreen(controller: this));
      } else {
        Get.back();
      }
      isWindowFullscreen.toggle();
    } else {
      throw UnimplementedError('Unsupported Platform');
    }
    enableController();
  }

  void enterPipMode(BuildContext context) async {}

  // volumn & brightness
  Future<double?> volumn() async {
     return await FlutterVolumeController.getVolume();
  }

  Future<double> brightness() async {
    if (Platform.isWindows || Platform.isLinux) {
      return await brightnessController.current;
    } else if (Platform.isAndroid || Platform.isIOS) {
      return await brightnessController.current;
    } else {
      throw UnimplementedError('Unsupported Platform');
    }
  }

  void setVolumn(double value) async {
    await FlutterVolumeController.setVolume(value);
  }

  void setBrightness(double value) async {
    await brightnessController.setScreenBrightness(value);
  }
}

// use fullscreen with controller provider
class MobileFullscreen extends StatefulWidget {
  const MobileFullscreen({
    Key? key,
    required this.controller,
  }) : super(key: key);

  final VideoController controller;

  @override
  State<MobileFullscreen> createState() => _MobileFullscreenState();
}

class _MobileFullscreenState extends State<MobileFullscreen>
    with WidgetsBindingObserver {
  @override
  void initState() {
    WidgetsBinding.instance.addObserver(this);
    super.initState();
  }

  @override
  void didChangeAppLifecycleState(AppLifecycleState state) {
    if (state == AppLifecycleState.resumed) {
      widget.controller.refresh();
    }
  }

  @override
  dispose() {
    WidgetsBinding.instance.removeObserver(this);
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      resizeToAvoidBottomInset: false,
      body: WillPopScope(
        onWillPop: () {
          widget.controller.toggleFullScreen();
          return Future(() => true);
        },
        child: Container(
          alignment: Alignment.center,
          color: Colors.black,
          child: Stack(
            alignment: Alignment.center,
            children: [
              VideoControllerPanel(controller: widget.controller),
            ],
          ),
        ),
      ),
    );
  }
}

class DesktopFullscreen extends StatelessWidget {
  const DesktopFullscreen({Key? key, required this.controller})
      : super(key: key);

  final VideoController controller;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      resizeToAvoidBottomInset: true,
      body: Stack(
        children: [
          media_kit_video.Video(
                filterQuality:FilterQuality.high,
                controller: controller.controller,
                fit: controller.videoFit.value,
                controls: (state) => VideoControllerPanel(controller: controller),
              )
        ],
      ),
    );
  }
}

// datasource is live stream
https://al.flv.huya.com/src/1634546845-1634546845-7020325243054981120-3269217146-10057-A-0-1-imgplus.flv?wsSecret=6d758a1e873ed5f847d61262849077db&wsTime=64f4ca0a&ctype=tars_mobile&fs=bgct&sphdcdn=al_7-tx_3-js_3-ws_7-bd_2-hw_2&sphdDC=huya&sphd=264_*-265_*&exsphd=264_500,264_2000,264_4000,&t=103&ver=1&sv=2110211124&seqid=1695143848709686&uid=1466142673855&uuid=864557549

Just sound,no video,and no error has been print

[liuchuancong]

Performs well on Windows

[liuchuancong]

I/flutter (29778): media_kit: wakelock: _count = 1
I/media_kit(29778): com.alexmercerind.media_kit_video.VideoOutputManager.create: 140735004246608
I/media_kit(29778): flutterJNIAPIAvailable = true
I/media_kit(29778): com.alexmercerind.media_kit_video.VideoOutput: id = 3
I/flutter (29778): {id: 3}
I/OMXClient(29778): IOmx service obtained
D/        (29778): PlayerBase::PlayerBase()
D/        (29778): TrackPlayerBase::TrackPlayerBase()
I/libOpenSLES(29778): Emulating old channel mask behavior (ignoring positional mask 0x3, using default mask 0x3 based on channel count of 2)
W/AudioTrack(29778): AUDIO_OUTPUT_FLAG_FAST denied by server; frameCount 0 -> 3776
E/libOpenSLES(29778): Configuration error: unknown key
W/libOpenSLES(29778): Leaving AndroidConfiguration::GetConfiguration (SL_RESULT_PARAMETER_INVALID)
D/        (29778): PlayerBase::stop() from IPlayer
D/AudioTrack(29778): stop() called with 0 frames delivered
I/media_kit(29778): com.alexmercerind.mediakitandroidhelper.MediaKitAndroidHelper.deleteGlobalObjectRef: 9962
E/BufferQueueProducer(29778): [SurfaceTexture-0-29778-2] cancelBuffer: BufferQueue has been abandoned
I/HostConnection(29778): HostConnection::~HostConnection, pid=29778, tid=30851, this=0x7fff45ee8c80, m_stream=0x7fff5abf19c0
I/        (29778): fastpipe: close connect
2
W/System  (29778): A resource failed to call release.
D/        (29778): PlayerBase::stop() from IPlayer
D/AudioTrack(29778): stop() called with 5337864 frames delivered

[liuchuancong]

@alexmercerind sometimes always green screen

[alexmercerind]

Hi!

Unfortunately it's really hard for me to say anything. I couldn't really access the media source either.

I should at-least be able to reproduce the issue before being able to solve it.

You may try disabling H/W acceleration, by passing enableHardwareAcceleration as false in VideoControllerConfiguration. You may refer to our docs for further details. There was another issue with live FLV few days ago, you may possible get some knowledge from there:

• https://github.com/media-kit/media-kit/issues/399

[liuchuancong]

@alexmercerind I've tried all kinds of things,but the problem persists. enableHardwareAcceleration,filterQuality,vo,etc

[liuchuancong]

Previously performed well with better_player

[liuchuancong]

@alexmercerind l fixed this problem!

  String? vo = Platform.isAndroid ? 'mediacodec_embed' : 'libmpv';
  String? hwdec = Platform.isAndroid ? 'mediacodec' : 'auto';
  late final controller = media_kit_video.VideoController(player,
      configuration: media_kit_video.VideoControllerConfiguration(
          enableHardwareAcceleration: false, vo: vo, hwdec: hwdec));

And sometimes will be black screen,retry player.open(Media(datasource));

[alexmercerind]

Careful!

--vo=mediacodec_embed may fail on some devices.

[ykhedar]

Hi, first of all, really great work with the library!!!! Working great on the desktop platform (tested on macos and windows regularly). Just started to test it on android. We mostly have network streams consisting of RTMP and RTSP.

I can confirm the same issue as reported by the OP of this issue: black screen - no video playback - when streaming a rtmp video source over network with FLV video codec. Changing the vo and hwdec as suggested by @liuchuancong seems to fix the issue but as suggested by @alexmercerind, may work on limited devices.

To reproduce the issue:

1. I used the test example provided in the media_kit library with library version 1.1.7.
2. Started a media server locally providing rtmp ingest and subscribe support:

docker run --rm -it -e MTX_PROTOCOLS=tcp -p 1935:1935 bluenviron/mediamtx https://github.com/bluenviron/mediamtx

3. Started a local file playback to RTMP using ffmpeg:

ffmpeg -re -stream_loop -1 -i test_file.mp4 -c copy -f flv rtmp://local-ip-address:1935/live

4. Used the Url in video playback in the video widget:

player.open(Media('rtmp://local-ip-address:1935/live'));

Not sure if the issue is in the MPV itself or Mediakit. Adding this comment for someone willing to reproduce and debug the issue. Until then I will continue using the suggestion provided by @liuchuancong.

[liuchuancong]

@ykhedar Now,on android, I'm using better_player.

[alexmercerind]

I can reproduce with procedure above.

[alexmercerind]

@ykhedar @liuchuancong

I added a new parameter to specifically handle your case. You must initialize your VideoController as follows:

late final Player player = Player();
late final VideoController controller = VideoController(
  player,
  configuration: const VideoControllerConfiguration(
    // NOTE:
    androidAttachSurfaceAfterVideoParameters: false,
  ),
);

https://github.com/alexmercerind/media-kit/commit/d06aa69d6ad5914fad546238a69903ab8fbcab3a

You can use dependency_overrides to use from git until next stable release (I aim next week).

---

It's not possible to "fix" the problem by default because current implementation is actually preferred by other users e.g. #339.

These FLV live-streams do seem like a edge-case.

[ykhedar]

Thanks @alexmercerind for the fast solution. I can confirm its working in the tests i did just now. Frankly I dont like RTMP but cannot avoid it due to a camera which can only publish rtmp and we have to support that :( I will use the git repo as library until the next version is released. Thanks again!!

[zq7695zq]

@alexmercerind l fixed this problem!

  String? vo = Platform.isAndroid ? 'mediacodec_embed' : 'libmpv';
  String? hwdec = Platform.isAndroid ? 'mediacodec' : 'auto';
  late final controller = media_kit_video.VideoController(player,
      configuration: media_kit_video.VideoControllerConfiguration(
          enableHardwareAcceleration: false, vo: vo, hwdec: hwdec));

And sometimes will be black screen,retry player.open(Media(datasource));

谢谢，我按照你的代码可以正常使用了，节约了我很多时间！

[alexmercerind]

@zq7695zq

Please refer to my last comment, we have added a new option to customize the behavior.

I will publish the new update soon. My computer actually died a day ago, so I'm in a bit of a problem.

Thanks!

[liuchuancong]

@alexmercerind Thanks,you are my hero!

[alexmercerind]

A new update has been released. Please refer to the "Installation" section of the README.

Thanks!