[video_player_web] Integrate Shaka Player for DASH and HLS support.

[ditman]

Use case

There has been multiple requests to support DASH/HLS videos in the web:

• https://github.com/flutter/flutter/issues/53011
• https://github.com/flutter/flutter/issues/46778

Proposal

We should explore Shaka Player, which is an open-source JavaScript library for adaptive media. It plays adaptive media formats (such as DASH and HLS) in a browser, without using plugins or Flash. Instead, Shaka Player uses the open web standards MediaSource Extensions and Encrypted Media Extensions.

TODO

In order to integrate this with video_player_web, we need a few things:

• Can Shaka's JS be dynamically downloaded when needed? This is to minimize the amount of JS a Flutter web app uses to bootstrap.
• Does Shaka support non-DASH/HLS videos? Can we use it as the core videoplayer? This is to decide how to integrate with the current plugin, or maybe write a new plugin for these types of videos that users optionally use.
• Create a js-interop layer of shaka's API. AFAIK, nobody has integrated Shaka in Dart before.

[joeyparrish]

• Can Shaka's JS be dynamically downloaded when needed? This is to minimize the amount of JS a Flutter web app uses to bootstrap.

Yes, absolutely. The JS and CSS (if using the UI-enabled build) are available from a various CDNs with versioned URLs, including one that is Google-owned: https://developers.google.com/speed/libraries/#shaka-player

If not using the UI, the JS simply needs to be fully loaded before calling the shaka.Player constructor. See tutorial: https://shaka-player-demo.appspot.com/docs/api/tutorial-basic-usage.html

If using the UI build and UI features, the JS and CSS need to be fully loaded first.

If using the UI with a declarative DOM-based setup, the app needs to wait for the shaka-ui-loaded event on document before referencing video.ui to get access to the controls and player instances. See tutorial: https://shaka-player-demo.appspot.com/docs/api/tutorial-ui.html

If using an API-driven setup, the app can call the shaka.ui.Overlay constructor as soon as the JS is loaded. There is no tutorial for this yet. See API docs (https://shaka-player-demo.appspot.com/docs/api/shaka.ui.Overlay.html) and pending issue to add an API-driven UI tutorial (https://github.com/google/shaka-player/issues/2655).

• Does Shaka support non-DASH/HLS videos? Can we use it as the core videoplayer? This is to decide how to integrate with the current plugin, or maybe write a new plugin for these types of videos that users optionally use.

Yes. For convenience, Shaka Player will pass single-file progressive videos through to the platform. A single API can be used for both adaptive content like DASH & HLS and simple progressive content like MP4s or WebM. (Note that not every browser supports WebM, but almost all support MP4.)

[ditman]

the app needs to wait for the shaka-ui-loaded event on document before referencing video.ui to get access to the controls and player instances.

@joeyparrish is this event thrown when the basic shaka js loads, or only when the UI library loads? Is there a shaka-loaded event? Is it enough to wait for the load event of the script tag? (we've had issues with this in the past 😄)

Also: what's the Overlay for?

[joeyparrish]

No, the Shaka library, without a UI, doesn't do anything on load, so it's ready to use right away.

The UI, however, will scan the document for data-shaka-player and data-shaka-player-container attributes to create the UI elements and attach JS objects to them automatically. So if you use our UI, and you use the DOM-based auto-setup feature I just described, the shaka-ui-loaded event tells you when that's done. Then you can access the auto-created player and controls objects from the video element like so:

const ui = videoElement.ui;
const controls = ui.getControls();

const player = controls.getPlayer();
const video = controls.getVideo();

If you use the UI, but not the DOM-based setup, you would instead create the UI like this:

// "local" because it is only for local playback.
// When casting, a player proxy object must be used to control the remote playback session.
const localPlayer = new shaka.Player(videoElement);
// "Overlay" because the UI will add DOM elements inside the container,
// to visually overlay the video element
const ui = new shaka.ui.Overlay(localPlayer, videoContainerElement, videoElement);

// As with DOM-based setup, get access to the UI controls and player from the UI.
const controls = ui.getControls();

// These are cast-enabled proxy objects, so that when you are casting,
// your API calls will be routed to the remote playback session.
const player = controls.getPlayer();
const video = controls.getVideo();

This snippet is essentially what needs to be added to our UI tutorials in https://github.com/google/shaka-player/issues/2655.

[ditman]

@joeyparrish this is great, that second snippet is exactly what flutter would need. The problem that I can foresee with the auto-setup is that flutter platform views in the Web are rendered inside a ShadowDOM, and wouldn't be visible to the auto-setup. (We're working on fixing that too, but it is what it is for now 😁)

[joeyparrish]

Yes, we always anticipated that the convenience of DOM-based setup wouldn't work for some frameworks. Glad to hear that these snippets are helpful! Please let us know what else we can do to help.

[hbengali]

Since TypeScript defs had come up in a prior discussion (outside of GitHub) w.r.t Shaka Player integration, I wanted to share that Shaka Player recently shipped a new release with official TypeScript definitions:

https://groups.google.com/g/shaka-player-users/c/w9_fwrFvHGA/m/CreMUAOrAQAJ

[AriasBros]

Hi all,

I am writing a package to integrate Shaka Player with Flutter. The initial version can be found here: https://github.com/AriasBros/flutter_video_player_web_shaka

For the people interested in use it as the default video_player_web package:

dependency_overrides:
  video_player_web:
    git: https://github.com/AriasBros/flutter_video_player_web_shaka.git

For now it is necessary modify the index.html of your application to add the Shaka script by yourself:

<script src="https://cdnjs.cloudflare.com/ajax/libs/shaka-player/4.1.0/shaka-player.compiled.js"></script>

or, from Google API CDN, an older version for now:

<script src="https://ajax.googleapis.com/ajax/libs/shaka-player/4.0.1/shaka-player.compiled.js"></script>

@ditman @joeyparrish I am trying to implement lazy load of the Shaka script in this branch:

https://github.com/AriasBros/flutter_video_player_web_shaka/tree/feat/shaka-lazy-load

But after the load event of the script tag, I am getting that shaka namespace is not in the scope. Even trying to access to the shaka namespace using the browser console gives me an undefined as result. In the network tab of the browser I can see how the request to download the script is done successfully.

Could you give me some feedback about this to try a fix?

@ditman Once the lazy load is fixed, I would like to send a PR to the video_player plugin to use Shaka Player as the default web player. It would be my first contribution so I need to pass all the process to send the PR.

---

Outside of the scope of this issue, in this branch I have implemented audio track selection:

https://github.com/AriasBros/flutter_video_player_web_shaka/tree/feat/track-selection

Following the interface defined in this other issue:

https://github.com/flutter/flutter/issues/79079

But the idea is to keep this work outside of the first PR sent with the Shaka integration.

---

Other work I will do in the next days is to play DRM content. For a future hypothetical PR too.

[ditman]

Thank you very much @AriasBros!! This is great!

A few comments:

For now it's necessary to modify the index.html...

I think that is fair, people already have to tweak their index.html for Google Maps, docs, because their JS SDK doesn't have any "onload" callback that tell us when the javascript has been parsed (and not just "loaded"). (Google Sign-In, does have an "onLoad" mechanism in the script URL. Docs.)

But after the load event of the script tag, I am getting that shaka namespace is not in the scope. Even trying to access to the shaka namespace using the browser console gives me an undefined as result.

I don't know the Shaka loading mechanism very well, but as I said above, the JS needs to be parsed by the browser (not just downloaded) for you to be able to access the Shaka player. Looking at your code I'd try the following:

1. Inject the script tag it in the head, not the body.
2. Do not defer; I don't think you want to defer this script :) Code.
3. Inject the Shaka JS in the plugin initialization code, not when it is RIGHT to play a video (near the registerWith call). This gives convenience to the user, doesn't block flutter from loading, but also gives it some time to load, before the app starts attempting to create instances.

If the lazy loading doesn't work, it's not the end of the world IMO.

I would like to send a PR to the video_player plugin to use Shaka Player as the default web player.

Don't worry too much about that; we can promote your video package to those people who need more advanced features than what a raw <video> tag can provide (DRM, HLS and tracks, etc...). You can own it and stabilize it for a while, without having to worry about the "firehose" of issues that might come from being the endorsed implementation of the plugin.

Once the plugin is mature, we can take a look at how to continue to maintain it (move it to flutter/plugins, or to somewhere the Shaka team owns, or... you decide to own it forever and we don't need to move it anywhere!)

Again, thanks for the contribution! I hope my message helped!

[AriasBros]

Thanks @ditman, very appreciated your answer.

About the lazy load, it is working now. It was a problem with the loading system from Shaka:

if(typeof exports!="undefined")
  for(var k in exportTo.shaka)
    exports[k]=exportTo.shaka[k];
else if(typeof define=="function"&&define.amd)
  define(function(){return exportTo.shaka});
else 
  innerGlobal.shaka=exportTo.shaka}

As we can see above, Shaka checks if RequireJS is being used, if it is, then it loads the library with RequireJS.

So I did it in a way that Flutter loads the library using RequireJS. I also included a fallback in case that RequireJS is not available anymore, just in case if RequireJS is removed as a dependency in the future. I am not sure about the necessity of this fallback or not, to be honest. I don't want abuse of your help, but any advice is very welcome.

https://github.com/AriasBros/flutter_video_player_web_shaka/blob/90000343bf4a47307deac01d5efb504c004685fe/lib/src/shaka_video_player.dart#L50

[AriasBros]

This comment will contain some off-topic outside of the scope of this issue, my apologies for that.

Once the plugin is mature, we can take a look at how to continue to maintain it (move it to flutter/plugins, or to somewhere the Shaka team owns, or... you decide to own it forever and we don't need to move it anywhere!)

The company where I am working is betting a lot with Flutter. We have ported this year our first (web) application (http://fantasy.formula1.com/). And now we are working in an OTT video application, with several requirements that the current video_player plugin (in any flavor, not only web) doesn't meet sadly.

Some of those requirements are:

• Audio track selection.
• Multi angle camera selection.
• Fairplay DRM support.
• Widevine DRM support.
• HLS TS support.
• HLS CMAF support.
• DASH MP4 support.

We would like to implement these requirements for Web, Android and iOS; and we would like to do it in a way that they can be promoted (in the future) to be the official Flutter implementation, not only as a payback to all the work that Flutter framework saves us, but also because in that way it is a way to save an hypothetical migration to the Flutter implementation of these features when they land :sweat_smile:

For these reasons, I would like to know where I should go now, to implement each of these requirements, because a lot of them will require add new methods to the interface of the video_player.

For example, I saw this issue open to track "Support video, audio and text track selections": https://github.com/flutter/flutter/issues/79079

Should I continue in that issue to get feedback from Flutter team and share my progress about that requirement? or should I open a new issue for each of the requirement I want to implement?

[vishwakarmakrishna]

@AriasBros how we can add licenseUrl which is required

---

Error: PlatformException(NO_LICENSE_SERVER_GIVEN, DRM, shaka.util.Error { "severity": 2, "category": 6, "code": 6012, "data": [ "com.widevine.alpha" ], "handled": false }, null) at Object.createErrorWithStack (http://localhost:55820/dart_sdk.js:5093:12) at Error._throw (http://localhost:55820/dart_sdk.js:20399:18) at Error.throwWithStackTrace (http://localhost:55820/dart_sdk.js:20396:18) at async._AsyncCallbackEntry.new.callback (http://localhost:55820/dart_sdk.js:40921:18) at Object._microtaskLoop (http://localhost:55820/dart_sdk.js:40778:13) at _startMicrotaskLoop (http://localhost:55820/dart_sdk.js:40784:13) at http://localhost:55820/dart_sdk.js:36261:9

---

CODE

---

class HomeLocalVideo extends StatefulWidget { const HomeLocalVideo({Key? key}) : super(key: key);

@override _HomeState createState() => _HomeState(); }

class _HomeState extends State { late VideoPlayerController _videoPlayerController;

@override void initState() { super.initState(); videoPlayerController = VideoPlayerController.network( "https://contents.pallycon.com/bunny/stream.mpd", videoPlayerOptions: VideoPlayerOptions(), httpHeaders: { 'pallycon-customdata-v2': 'eyJrZXlfcm90YXRpb24iOmZhbHNlLCJyZXNwb25zZV9mb3JtYXQiOiJvcmlnaW5hbCIsInVzZXJfaWQiOiJ0ZXN0LXVzZXIiLCJkcm1fdHlwZSI6IldpZGV2aW5lIiwic2l0ZV9pZCI6IkRFTU8iLCJoYXNoIjoiRFNEQ0JwWmhJYVR5VG1MMzlCXC9Yb2IyNzRobWpWXC9oWEp4T1V0K29hZ1pjPSIsImNpZCI6ImJpZ2J1Y2tidW5ueSIsInBvbGljeSI6Im41eDI4dVltRGRQQ0ZpbW9NM25HTnc9PSIsInRpbWVzdGFtcCI6IjIwMjEtMDEtMDZUMDk6MjI6MzZaIn0=' }) ..initialize().then(() { _videoPlayerController.play(); _videoPlayerController.setLooping(true); setState(() {}); }); }

@override void dispose() { _videoPlayerController.dispose(); super.dispose(); }

@override Widget build(BuildContext context) { return SafeArea( child: Scaffold( body: _videoPlayerController.value.isInitialized ? Stack( children: [ SizedBox.expand( child: FittedBox( fit: BoxFit.fill, child: SizedBox( width: _videoPlayerController.value.size.width, height: _videoPlayerController.value.size.height, child: VideoPlayer(_videoPlayerController), ), ), ), Container( width: 100, height: 100, child: Text("Hi"), ) ], ) : Container(), ), ); } }

[AriasBros]

Hi @kv0871916, sorry, but for now there is not DRM support. It is something that I will start to work in the next days.

The only work done right now is the Shaka player integration itself and the ability to change the audio track following the interface done in this other issue:

https://github.com/flutter/flutter/issues/79079

To use multi language you will need to use other dependency overrides:

dependency_overrides:
  video_player:
    git:
      url: https://github.com/AriasBros/flutter_plugins.git
      path: packages/video_player/video_player
      ref: feature_track_selection
  video_player_platform_interface:
    git:
      url: https://github.com/AriasBros/flutter_plugins.git
      path: packages/video_player/video_player_platform_interface
      ref: feature_track_selection_platform_interface
  video_player_web:
    git:
      url: https://github.com/AriasBros/flutter_video_player_web_shaka.git
      ref: feat/track-selection

[AriasBros]

@kv0871916

Here you have the branches with Widevine DRM support (Other DRMs based in Server URL license should work too)

dependency_overrides:
  video_player:
    git:
      url: https://github.com/AriasBros/flutter_plugins.git
      path: packages/video_player/video_player
      ref: feat/drm-support
  video_player_platform_interface:
    git:
      url: https://github.com/AriasBros/flutter_plugins.git
      path: packages/video_player/video_player_platform_interface
      ref: feat/drm-support
  video_player_web:
    git:
      url: https://github.com/AriasBros/flutter_video_player_web_shaka.git

Then, in your application, you can do something like:

      _videoPlayerController = VideoPlayerController.network(
        video.url,
        drmType: video.drmType,
        drmUriLicense: video.drmUrlLicense,
        drmHttpHeaders: {'authToken': adapter.authToken},
        withCredentials: true,
      )..initialize().then((_) {
          setState(() {});
        });

[ditman]

So I did it in a way that Flutter loads the library using RequireJS. I also included a fallback in case that RequireJS is not available anymore, just in case if RequireJS is removed as a dependency in the future. I am not sure about the necessity of this fallback or not, to be honest. I don't want abuse of your help, but any advice is very welcome.

@AriasBros have you tried compiling your app in "release" mode? I don't think "requireJS" is available there at all (and you shouldn't trust it being there, or use it directly). See this:

• https://github.com/dart-lang/sdk/issues/33979

Should I continue in that issue to get feedback from Flutter team and share my progress about that requirement? or should I open a new issue for each of the requirement I want to implement?

No, issues are free to create, so please open (at least) an issue about "The future of the video_player plugin", where you list all the features that you know that you're going to need, and your plan to execute on those. I see two possible outcomes here (there's probably more): either the existing plugin gets expanded in all the directions needed, or you guys end up creating/owning an drm_video_player plugin that ends up being the preferred one by the community, and we end up pointing people to using your video player instead of video_player :)

I don't know if you've seen the Plus plugins (https://plus.fluttercommunity.dev), but those were plugins originally created by the flutter team, that are now owned and maintained by the community!

Definitely create a new issue so this can be discussed there!

...fantasy.formula1.com...

 (PS: Love it!)