How to get available subtitles text from subtitle track

[basemosama]

I have a video with some subtitle tracks and i want to allow the user to choose the subtitle track from the video and display it.

How can I have the current available subtitle text or all text of the current selected subtitle track?

And Can we render it automatically using fvp?

Here is some videos with a subtitle track https://storage.googleapis.com/exoplayer-test-media-1/mp4/dizzy-with-tx3g.mp4 http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8

For example in Exoplayer there's oncues function that return current text to display based on current playback time

public void onCues(@NonNull CueGroup cueGroup) {
   final Cue cue = cueGroup.cues.get(0);
   final text = cue.text.toString() ;
    }

Thanks

[wang-bin]

subtitle rendering requires libass, it's already supported on windows, linux, iOS and macOS. I haven't tested on android.

[wang-bin]

run dart pub cache clean and build again, then libass will be added for android. for iOS, you have to download https://sourceforge.net/projects/mdk-sdk/files/deps/dep.7z/download and add ass.framework manually in xcode project. for linux, installing system libass is enough.

[basemosama]

Currently I am focusing in Android.

Can I have access to the selected subtitle track text? And Is there any configuration to render subtitle?

[wang-bin]

Currently I am focusing in Android.

Can I have access to the selected subtitle track text?

No. which api are you using? video_player via fvp.dart or Player via mdk.dart?

And Is there any configuration to render subtitle?

No configuration is required.

[basemosama]

Player via mdk.dart

[basemosama]

When trying the latest version of fvp, The app doesn't open and I get this error when using the app.

[ +775 ms] E/flutter (12374): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'libfvp_plugin.so': dlopen failed: library "libass.so" not found
[        ] E/flutter (12374): #0      _open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:43)
[        ] E/flutter (12374): #1      new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:22:12)
[        ] E/flutter (12374): #2      Libfvp._load (package:fvp/src/lib.dart:52:29)
[        ] E/flutter (12374): #3      Libfvp.instance (package:fvp/src/lib.dart:58:27)
[        ] E/flutter (12374): #4      Libfvp.instance (package:fvp/src/lib.dart)
[        ] E/flutter (12374): #5      Libfvp.isEmulator (package:fvp/src/lib.dart:80:29)
[        ] E/flutter (12374): #6      Libfvp.isEmulator (package:fvp/src/lib.dart)
[        ] E/flutter (12374): #7      PlatformEx.isAndroidEmulator (package:fvp/src/extensions.dart:16:19)
[        ] E/flutter (12374): #8      MdkVideoPlayerPlatform.registerVideoPlayerPlatformsWith (package:fvp/src/video_player_mdk.dart:138:42)
[        ] E/flutter (12374): #9      registerWith (package:fvp/fvp.dart:39:26)
[        ] E/flutter (12374): #10     main (package:untitled6/main.dart:15:7)
[        ] E/flutter (12374): #11     _runMain.<anonymous closure> (dart:ui/hooks.dart:301:23)
[        ] E/flutter (12374): #12     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:297:19)
[        ] E/flutter (12374): #13     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:184:12)
[        ] E/flutter (12374): 

[wang-bin]

can you upload your apk? no error in my tests

[basemosama]

I'm testing on the default example app.

Steps to reproduce:

• clean cache using dart pub cache clean
• delete pubspec.lock
• run flutter clean
• run flutter pub get and run the app.

There is an issue when in the latest mdk sdk version that is downloaded to the cache from here https://sourceforge.net/projects/mdk-sdk/files/nightly/mdk-sdk-android.7z

Here is the code

import 'package:flutter/material.dart';
import 'package:fvp/fvp.dart' as fvp;
import 'package:video_player/video_player.dart';

void main() {
  fvp.registerWith();

  runApp(
    MaterialApp(
      home: _App(),
    ),
  );
}

class _App extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      key: const ValueKey<String>('home_page'),
      appBar: AppBar(
        title: const Text('Video player example'),
      ),
      body: _BumbleBeeRemoteVideo(),
    );
  }
}

class _BumbleBeeRemoteVideo extends StatefulWidget {
  @override
  _BumbleBeeRemoteVideoState createState() => _BumbleBeeRemoteVideoState();
}

class _BumbleBeeRemoteVideoState extends State<_BumbleBeeRemoteVideo> {
  late VideoPlayerController _controller;

  Future<ClosedCaptionFile> _loadCaptions() async {
    final String fileContents = await DefaultAssetBundle.of(context)
        .loadString('assets/bumble_bee_captions.vtt');
    return WebVTTCaptionFile(
        fileContents); // For vtt files, use WebVTTCaptionFile
  }

  String subtitle = "Subtitle";

  @override
  void initState() {
    super.initState();

    _controller = VideoPlayerController.networkUrl(
      Uri.parse(
          'http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8'),
    );
    _controller.setLooping(true);
    _controller.initialize();
  }

  @override
  void dispose() {
    _controller.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return SingleChildScrollView(
      child: Column(
        children: <Widget>[
          Container(padding: const EdgeInsets.only(top: 20.0)),
          const Text('With remote mp4'),
          Container(
            padding: const EdgeInsets.all(20),
            child: AspectRatio(
              aspectRatio: _controller.value.aspectRatio,
              child: Stack(
                alignment: Alignment.bottomCenter,
                children: <Widget>[
                  VideoPlayer(_controller),
                  ClosedCaption(text: subtitle),
                  _ControlsOverlay(controller: _controller),
                  VideoProgressIndicator(_controller, allowScrubbing: true),
                ],
              ),
            ),
          ),
        ],
      ),
    );
  }
}

class _ControlsOverlay extends StatelessWidget {
  const _ControlsOverlay({required this.controller});

  static const List<Duration> _exampleCaptionOffsets = <Duration>[
    Duration(seconds: -10),
    Duration(seconds: -3),
    Duration(seconds: -1, milliseconds: -500),
    Duration(milliseconds: -250),
    Duration.zero,
    Duration(milliseconds: 250),
    Duration(seconds: 1, milliseconds: 500),
    Duration(seconds: 3),
    Duration(seconds: 10),
  ];
  static const List<double> _examplePlaybackRates = <double>[
    0.25,
    0.5,
    1.0,
    1.5,
    2.0,
    3.0,
    5.0,
    10.0,
  ];

  final VideoPlayerController controller;

  @override
  Widget build(BuildContext context) {
    return Stack(
      children: <Widget>[
        AnimatedSwitcher(
          duration: const Duration(milliseconds: 50),
          reverseDuration: const Duration(milliseconds: 200),
          child: controller.value.isPlaying
              ? const SizedBox.shrink()
              : Container(
                  color: Colors.black26,
                  child: const Center(
                    child: Icon(
                      Icons.play_arrow,
                      color: Colors.white,
                      size: 100.0,
                      semanticLabel: 'Play',
                    ),
                  ),
                ),
        ),
        GestureDetector(
          onTap: () {
            controller.value.isPlaying ? controller.pause() : controller.play();
          },
        ),
        Align(
          alignment: Alignment.topLeft,
          child: PopupMenuButton<Duration>(
            initialValue: controller.value.captionOffset,
            tooltip: 'Caption Offset',
            onSelected: (Duration delay) {
              controller.setCaptionOffset(delay);
            },
            itemBuilder: (BuildContext context) {
              return <PopupMenuItem<Duration>>[
                for (final Duration offsetDuration in _exampleCaptionOffsets)
                  PopupMenuItem<Duration>(
                    value: offsetDuration,
                    child: Text('${offsetDuration.inMilliseconds}ms'),
                  )
              ];
            },
            child: Padding(
              padding: const EdgeInsets.symmetric(
                // Using less vertical padding as the text is also longer
                // horizontally, so it feels like it would need more spacing
                // horizontally (matching the aspect ratio of the video).
                vertical: 12,
                horizontal: 16,
              ),
              child: Text('${controller.value.captionOffset.inMilliseconds}ms'),
            ),
          ),
        ),
        Align(
          alignment: Alignment.topRight,
          child: PopupMenuButton<double>(
            initialValue: controller.value.playbackSpeed,
            tooltip: 'Playback speed',
            onSelected: (double speed) {
              controller.setPlaybackSpeed(speed);
            },
            itemBuilder: (BuildContext context) {
              return <PopupMenuItem<double>>[
                for (final double speed in _examplePlaybackRates)
                  PopupMenuItem<double>(
                    value: speed,
                    child: Text('${speed}x'),
                  )
              ];
            },
            child: Padding(
              padding: const EdgeInsets.symmetric(
                // Using less vertical padding as the text is also longer
                // horizontally, so it feels like it would need more spacing
                // horizontally (matching the aspect ratio of the video).
                vertical: 12,
                horizontal: 16,
              ),
              child: Text('${controller.value.playbackSpeed}x'),
            ),
          ),
        ),
      ],
    );
  }
}

[wang-bin]

i need your apk because i can't reproduce your error

[basemosama]

Here is the apk https://drive.google.com/file/d/1aoEyq15vNhf9xdKjcbefBmqrHi-gx06A/view?usp=sharing

I tested it on Huawei Mate 20

[wang-bin]

show me the result of flutter doctor --verbose

github action build is ok too, libass is automatically added in to apk, so i guess the error is from your build environment.

[basemosama]

[✓] Flutter (Channel stable, 3.16.8, on Microsoft Windows [Version 10.0.22621.3007], locale ar-EG)
    • Flutter version 3.16.8 on channel stable at E:\flutter\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 67457e669f (8 days ago), 2024-01-16 16:22:29 -0800
    • Engine revision 6e2ea58a5c
    • Dart version 3.2.5
    • DevTools version 2.28.5

[✓] Windows Version (Installed version of Windows is version 10 or higher)

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at E:\Sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = E:\Sdk
    • ANDROID_SDK_ROOT = E:\Sdk
    • Java binary at: C:\Program Files\Android\Android Studio\jbr\bin\java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-10027231)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[✗] Visual Studio - develop Windows apps
    ✗ Visual Studio not installed; this is necessary to develop Windows apps.
      Download at https://visualstudio.microsoft.com/downloads/.
      Please install the "Desktop development with C++" workload, including all of its default components

[✓] Android Studio (version 2022.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-10027231)

[✓] VS Code, 64-bit edition (version 1.79.2)
    • VS Code at C:\Program Files\Microsoft VS Code
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (4 available)
    • LYA L29 (mobile)  • LHS7N19216018187 • android-arm64  • Android 10 (API 29)
    • Windows (desktop) • windows          • windows-x64    • Microsoft Windows [Version 10.0.22621.3007]
    • Chrome (web)      • chrome           • web-javascript • Google Chrome 120.0.6099.225
    • Edge (web)        • edge             • web-javascript • Microsoft Edge 120.0.2210.144

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.
The Flutter CLI developer tool uses Google Analytics to report usage and diagnostic
data along with package dependencies, and crash reporting to send basic crash
reports. This data is used to help improve the Dart platform, Flutter framework,
and related tools.

Telemetry is not sent on the very first run. To disable reporting of telemetry,
run this terminal command:

    flutter --disable-analytics

If you opt out of telemetry, an opt-out event will be sent, and then no further
information will be sent. This data is collected in accordance with the Google
Privacy Policy (https://policies.google.com/privacy).

Please note that analytics reporting was already disabled, and will continue to be disabled.

[wang-bin]

The difference is I use macos and ubuntu to build android apps. Please open fvp project in vscode, and run task(ctrl + p, then type task and a space), select Build APK to build the example, copy all the terminal output and paste here.

[basemosama]

Can you try rerun the apk github action build again is it was built 3 days ago as the issue has something to do with the nightly mdk- sdk here https://sourceforge.net/projects/mdk-sdk/files/nightly/mdk-sdk-android.7z that was update recently.

I will try to build the apk and send the ouptut

[wang-bin]

i ran the jobs again a few hours ago.

[basemosama]

Yes, The action apk is working, but when building on my side it's not working. Here is the log from vscode https://drive.google.com/file/d/1fQc8wcZXBkStRePHF24QmRtS4Bfy0gAV/view?usp=sharing

[wang-bin]

I add android on windows in github actions, it works as expected. From your build log a can't see libass.so, it's weired. libass.so is processed exactly the same as libffmpeg.so, I don't know why no libass.so in your build. Other differences are:

• ndk version, mime is ndk 25 or later
• you use fvp 0.13.1, what about building fvp/example directly(open fvp project in vscode, run Build APK task)?

[wang-bin]

I used a cmake feature not supported in old versions. run flutter clean and build again should fix the issue

[basemosama]

Yes, It's working again. Thanks. But I still can't manage to render the subtitle track.

[wang-bin]

https://storage.googleapis.com/exoplayer-test-media-1/mp4/dizzy-with-tx3g.mp4 this subtitle requires a font does not exist on android. I see some other players(e.g. mpv-android) provides a font file, it's about 7M. I have to find out a way to optionally adding the font

[wang-bin]

http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8 hls subtitle is disabled by ffmpeg by default. There is an option to enable it.

[wang-bin]

Try the master branch code, add assets/subfont.ttf in pubspec.yaml, then subtitle can be renderer required font is not found in the system

[basemosama]

Thanks, It's working now.

I have some questions. How can we enable subs for hls? And Can we configure the subtitle style like font, font size and color? And for the future It will be nice to have an event for subtitles that shows the current subtitle text need to be displayed.

[wang-bin]

I have some questions. How can we enable subs for hls?

You can enable it via player.setProperty('avformat.strict', 'experimental');. But not sure if it works for http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8 because i fail to download segments

And Can we configure the subtitle style like font, font size and color? And for the future It will be nice to have an event for subtitles that shows the current subtitle text need to be displayed.

Will be supported in a future version.

[basemosama]

Is teletext subtitles supported? I am trying to test some samples with teletext subtitles but can't manage to show subs. Here is some samples : https://mega.nz/folder/dBQSwaJL#MUgvR9DNiqkM59MRCSLhzQ

And some logs related to subtitle

19:21:02.270  I  mdk.INFO: CopyToAppFilesDir: assets://flutter_assets/assets/subfont.ttf => /data/user/0/com.mediadevkit.fvp_example/files/mdk/subfont.ttf
19:21:02.270  I  mdk.INFO: CopyToAppFilesDir: /data/user/0/com.mediadevkit.fvp_example/files/mdk/subfont.ttf already exists
19:21:02.347  I  fvp.FINE: 783605235 player507254545920 onEvent: thread.subtitle 1
19:21:02.347  I  mdk.INFO: subtitle stream#3 starting decoding loop from decoder index 0...
19:21:02.347  I  mdk.INFO: creating subtitle decoder: auto...
19:21:02.348  I  mdk.INFO: opening subtitle decoder: FFmpeg...
19:21:02.349  I  mdk.INFO: subtitle decoder not found for codec: dvb_teletext/force: 
19:21:02.350  I  mdk.WARNING: ERROR! failed to setup decoder: subtitle
19:21:02.350  I  fvp.FINE: 783605235 player507254545920 onEvent: decoder.subtitle -1
19:21:02.350  I  mdk.INFO: 0x76226d5800 1st subtitle frame @0.000000
19:21:02.350  I  mdk.INFO: 0x76226d5800 seek end subtitle frame @0.000000 seek_pos_: -1
19:21:02.350  I  mdk.INFO: EOS subtitle frame of track 0
19:21:02.351  I  mdk.INFO: subtitle stream#3 decoding loop is finished. packets: 8, pts: [34930.641944, 34930.921944]
19:21:02.351  I  fvp.FINE: 783605235 player507254545920 onEvent: thread.subtitle 0

[wang-bin]

teletext requires another external library

[basemosama]

Teletext is the main subtitle in the app I am building. Can you try to add it to the package?

[wang-bin]

teletext requires libzvbi, it's license is GPL, it's not acceptable because the whole app must be open source, is it ok for you? You can build ffmpeg yourself with libzvbi enabled, then replace libffmpeg.so. I can also try to build a new libffmpeg.so for you, but I won't redistribute it with my library.

[basemosama]

Ok, No problem. Can you build the libffmpeg.so with libzvbi enabled?