[mdk.dart aka backend player api] Choppy playback after setting external audio track, application freezes in between switching media source.

[aimsson]

Describe the bug The example is based on a modified mdk-examples/flutter/simple/lib/multi_textures.dart. The player starts with a single video file (no audio, no subtitles), after .prepare it I added the external audio track (online radio stream) via .setMedia. Everything was fine, video playback went smoothly, audio worked well. If I do switch to another regular (video track + audio track) mp4, video and audio playback become choppy. Stop/restart dont help. The app can also fall into deadlock in between switching media source or at a try to set PlaybackState.stopped with no log message. I have recorded the explanation video: youtube

Expected behavior Expected smooth playback like when the .setMedia line with external audio is commented out. Also expected some error log entry at deadlock moment. Deadlock can happen even with a commented out .setMedia

Example Code

import 'package:flutter/material.dart';
import 'package:fvp/mdk.dart';
import 'package:logging/logging.dart';

void main(List<String> args) async {
  Logger.root.level = Level.ALL;
  Logger.root.onRecord.listen((record) {
    debugPrint('${record.loggerName}.${record.level.name}: ${record.time}: ${record.message}');
  });

  setGlobalOption('logLevel', 'All');
  setGlobalOption('ffmpeg.loglevel', 'warning');

  Logger log = Logger('mdktest');
  setLogHandler((p0, p1) => log.log(Level.ALL, '$p0 --- $p1'));

  runApp(const SinglePlayerMultipleVideoWidget());
}
class SinglePlayerMultipleVideoWidget extends StatefulWidget {
  const SinglePlayerMultipleVideoWidget({super.key});

  @override
  State<SinglePlayerMultipleVideoWidget> createState() =>
      _SinglePlayerMultipleVideoWidgetState();
}

class _SinglePlayerMultipleVideoWidgetState
    extends State<SinglePlayerMultipleVideoWidget> {
  late final player = Player();

  void videoNoAudioExternalAudioStream() {
    player.media = 'https://getsamplefiles.com/download/mkv/sample-3.mkv';
    player.prepare().then((int position) {
      debugPrint('Pos "Muted video + audio stream" after prepare: $position');
      debugPrint('PlaybackState after .prepare: ${player.state}');
      if (!position.isNegative) {
        player.setMedia('https://stream.tunerplay.com/radio/8020/jazzradiospain.mp3', MediaType.audio);
      }
    });
  }

  void videoWithAudio() {
    player.media = 'https://sample-videos.com/video321/mp4/480/big_buck_bunny_480p_30mb.mp4';
    player.prepare().then((int position) {
      debugPrint('Pos "Regular video with audio" after prepare: $position');
      debugPrint('PlaybackState after .prepare: ${player.state}');
      if (!position.isNegative) {

      }
    });
  }

  void stop() {
    player.state = PlaybackState.stopped;
    debugPrint('PlaybackState.stopped initiated');
    player.waitFor(PlaybackState.stopped);
    debugPrint('PlaybackState.stopped set');
  }

  void play() {
    player.state = PlaybackState.playing;
  }

  void pause() {
    player.state = PlaybackState.paused;
  }

  bool mediaStatusChanged(MediaStatus oldValue, MediaStatus newValue) {
    if (newValue.test(MediaStatus.prepared) && !oldValue.test(MediaStatus.prepared)) {
      play();
    }
    return true;
  }

  @override
  void initState() {
    super.initState();

    player.loop = -1;
    player.volume = .4;
    player.onMediaStatus((oldValue, newValue) => mediaStatusChanged(oldValue, newValue));
    videoNoAudioExternalAudioStream();

    player.updateTexture();
  }

  @override
  void dispose() {
    player.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'package:fvp',
      home: Scaffold(
        body: Stack(
          children: [
            ValueListenableBuilder<int?>(
              valueListenable: player.textureId,
              builder: (context, id, _) => id == null
                  ? const SizedBox.shrink()
                  : Texture(textureId: id),
            ),
            Align(
              alignment: Alignment.bottomLeft,
              child: Column(
                mainAxisAlignment: MainAxisAlignment.end,
                crossAxisAlignment: CrossAxisAlignment.start,
                children: [
                  TextButton(
                    onPressed: videoNoAudioExternalAudioStream,
                    child: const Text('Change to video with no audio + external music stream'),
                  ),
                  TextButton(
                    onPressed: videoWithAudio,
                    child: const Text('Change to video with audio'),
                  ),
                  TextButton(
                    onPressed: stop,
                    child: const Text('Stop'),
                  ),
                  TextButton(
                    onPressed: play,
                    child: const Text('Play'),
                  ),
                  TextButton(
                    onPressed: pause,
                    child: const Text('Pause'),
                  ),
                ],
              ),
            ),
          ],
        )

      ),
    );
  }
}

Complete Log of a video explanation pastebin

[wang-bin]

a workaround for now is adding player.setMedia('', MediaType.audio); in videoWithAudio. I will fix it later

  void videoWithAudio() {
    player.setMedia('', MediaType.audio);
    player.media = 'https://sample-videos.com/video321/mp4/480/big_buck_bunny_480p_30mb.mp4';
    player.prepare().then((int position) {
      debugPrint('Pos "Regular video with audio" after prepare: $position');
      debugPrint('PlaybackState after .prepare: ${player.state}');
      if (!position.isNegative) {

      }
    });
  }

[wang-bin]

for dead lock, you have to set stop state before prepare

  void videoNoAudioExternalAudioStream() {
    player.state = PlaybackState.stopped;
    player.media = 'https://getsamplefiles.com/download/mkv/sample-3.mkv';
    player.prepare().then((int position) {
      debugPrint('Pos "Muted video + audio stream" after prepare: $position');
      debugPrint('PlaybackState after .prepare: ${player.state}');
      if (!position.isNegative) {
        player.setMedia('https://stream.tunerplay.com/radio/8020/jazzradiospain.mp3', MediaType.audio);
      }
    });
  }

  void videoWithAudio() {
    player.state = PlaybackState.stopped;
    player.setMedia('', MediaType.audio);
    player.media = 'https://sample-videos.com/video321/mp4/480/big_buck_bunny_480p_30mb.mp4';
    player.prepare().then((int position) {
      debugPrint('Pos "Regular video with audio" after prepare: $position');
      debugPrint('PlaybackState after .prepare: ${player.state}');
      if (!position.isNegative) {

      }
    });
  }

[wang-bin]

run

pod cache clean mdk
pod deintegrate macos/Runner.xcodeproj
rm macos/Podfile.lock

and build again, then no need to call player.setMedia('', MediaType.audio);

[aimsson]

run

pod cache clean mdk
pod deintegrate macos/Runner.xcodeproj
rm macos/Podfile.lock

and build again, then no need to call player.setMedia('', MediaType.audio);

yep, did it

but I'm still getting troubles with playback and deadlocks. Now the nosound video lags really bad, sometimes it doesnt. The big buck bunny is unpredictable too, it can lag or not. I've made a new video and log youtube pastebin

[wang-bin]

run

pod cache clean mdk
pod deintegrate macos/Runner.xcodeproj
rm macos/Podfile.lock

and build again, then no need to call player.setMedia('', MediaType.audio);

yep, did it

mdk version is not correct from the log. maybe you have to run rm -rf macos/Pods after above commands

[aimsson]

Yes, after removing the Pods folder and building again everything turned out to be fine. @wang-bin thank you very much for your help and blazingly fast replies. I really love the backend player api, it's awesome! Would you be so kind to answer one more question - should I always switch video sources through setting PlaybackState.stopped or is there any way of "hot" swapping the video source?

[wang-bin]

must stop current then play a new one. there is another api setNext, it will preload the video, and play gaplessly when current is stopped, but does not support individual audio track source.