Preload API

[joeyparrish]

We should have an API to allow an application to pre-load a manifest and some of the media segments without involving MediaSource or a <video> element. An application could speculatively pre-load the beginning of several pieces of content while the user is making a decision. That content could then begin playing almost instantly.

This requires some redesign. Currently, new Player() requires an HTMLMediaElement, and the load() pipeline connects from manifest parsing all the way to streaming and MediaSource. These things would need to be decoupled, and a new preload() method would have to be introduced.

Note, though, that if we had it today, this new preload() method would not be usable from a service worker just yet, because our DASH manifest parser depends on DOMParser for XML parsing, and DOMParser is not available in a service worker.

Until manifest parsing can be done from a service worker context, operations involving a manifest must be done from the page context.

[joeyparrish]

API ideas, copied from discussion in https://github.com/google/shaka-player/issues/961#issuecomment-326415628

We will have to change the API so that a video element is not required in the Player constructor. MediaSource would be set up later. Examples:

// current flow
player = new shaka.Player(video);  // attach right away, set up MediaSource
player.load('foo.mpd');

// preload flow
player = new shaka.Player();
player.preload('foo.mpd');  // start loading content
player.attach(video);  // now set up MediaSource and complete the pipeline

// wait to set up MediaSource, without preloading
player = new shaka.Player();
// time passes...
player.attach(video);  // MediaSource setup happens now, when you want it to

The HTMLMediaElement argument to the constructor becomes optional, which gives us API backward compatibility. If a video element is supplied, we will automatically attach in the constructor.

[chrisfillmore]

This is a feature we anticipate wanting to provide to our clients. And more than that, I expect we'd want to preload more than just one piece of content. This is basically to support rapid channel switching behaviour, and the like. There's another wrinkle: I also anticipate we'd want to preload some piece of content and then provide it to a specific Player instance. The reason for this is we may have multiple Players going at once.

If I'm told we need this feature, I think I could work on a PR. But given the above I'd like your input.

[chrisfillmore]

Idea, perhaps total brain fart:

/**
 * Create a Preload, which can be passed to any Player so that it
 * instantly has preloaded content.
 * @static
 * @param {object} params Anything it would need to preload content
 * @returns {shaka.Player.Preload}
 */
Player.createPreload (params) {
  // Do preloading work
}

[bennypowers]

I'd really like to be able to pre load segments or ranges from a single manifest

player = new shaka.Player(video);
// preload with optional ranges param 
player.load('foo.mpd', [
  [0, 1234], // time range in ms
  [2345, 3456],
  [4567, 5678],
  6789, // time index of single segment to pre cache
]);
// party 🎉
player.attach(video)

[chrisfillmore]

Another thought about this: it would be nice if Player#load did not require a video element to be set. It would also be nice if we could configure the player to fetch segments or not. A player with no video element attached would then just fetch manifest updates, and (optionally) build a video buffer.

[joeyparrish]

Hi @chrisfillmore, @bennypowers,

Since #1087, a video element is no longer required in the constructor, but it must be attached by the time load() is called. I do not expect to change this, so that the meaning of load() will continue to be "load all into MediaSource".

For preload, I am thinking along the lines of what Chris suggested in https://github.com/google/shaka-player/issues/880#issuecomment-360896727, where we return a token that can be used to complete the load process. The player would be capable of starting multiple preloads in parallel, to support speculation about what the end-user will click on. When the user finally clicks on a thing, you can then choose which preload to continue with. The others would then be invalidated.

Maybe something like this:

let player = new shaka.Player();
let token1 = player.preload('foo1.mpd');
let token2 = player.preload('foo2.mpd');
let token3 = player.preload('foo3.mpd');
// time passes...
player.attach(video);
await token3.load();  // like player.load('foo3.mpd'), but we already fetched segments
// tokens 1, 2, and 3 are all now invalidated.
// buffered data in tokens 1 and 2 has been dropped.
// buffered data in token3 has been transferred to Player and MediaSource.

As for Benny's suggestion in https://github.com/google/shaka-player/issues/880#issuecomment-365308775, Shaka Player does not buffer multiple independent ranges in general, so I don't plan to incorporate this into the preload design.

Thanks!

[chrisfillmore]

@joeyparrish that looks pretty good, the only suggestion I have is that it would be preferable if the client could explicitly call something like token.unload(), instead of having Player do the invalidation implicitly. This would enable e.g. the user switching back and forth between two or more channels.

[joeyparrish]

I can see how that might be useful for a TV-like scenario: you could keep the next and previous channel preloaded so the user could "channel surf" with low latency.

But, I wasn't thinking of how this would interact with live streams. I was imagining that we could buffer up to rebufferingGoal and then stop. For live streams, we would need to keep the manifest up-to-date and potentially continue buffering as the live edge moves.

It's not undoable, but I had envisioned something much smaller and simpler. We'll need to give this some careful thought.

What would your expectation be for having several tokens representing live streams? What should they be doing while they wait for a call to load()?

[chrisfillmore]

I would expect that a token for a live stream would, at minimum, continually fetch and parse manifests. At any time, the client could tell it to start downloading segments as well, via an explicit call like token.startBuffering().

This way the client could define its own user behaviours that signal an upcoming channel change (browsing a guide, rapid channel up-down behaviour, "previous channel" button on the remote, etc). Furthermore, the end-user could even tell the application how many channels to preload, potentially useful if they have a very high quality connection.

[joeyparrish]

Okay. I was thinking of keeping the API as simple as possible. Calling preload() would start buffering, and token.load() would fully connect the pipeline to MediaSource.

It seems like you're suggesting something more like this: preload() starts fetching and updating the manifest, token.startBuffering() would start buffering, and token.load() would then fully connect the pipeline.

Is that right?

In this three-stage process, when would each stage be triggered by the app? (Manifest, buffering, playback.) I guess I'm having trouble understanding why buffering would need to be a separate stage.

[chrisfillmore]

Speculatively buffering multiple streams at the same time might be undesirable, particularly on slower connections. We may also already have up to 4 streams already playing on-screen, so we don't necessarily want to use bandwidth doing more buffering.

I have seen a proof-of-concept on Android of a custom player doing rapid channel switching without buffering the background streams, and it looks good. I was actually impressed, it looked like real TV channel switching. So I think this is worth investigating.

We want to be able prefetch, where possible:

1. Manifests
2. Initialization segment
3. License

I think doing so should substantially reduce stream startup time. Regarding (3), perhaps, when preloading, the player could fetch a single segment from the stream in order to generate the challenge and fetch the license.

Thoughts on this?

[bennypowers]

If that rapid channel switching PoC is public I'd love to see a link, since my use case is similar :D

[chrisfillmore]

@bennypowers Sorry unfortunately it's not, it was work done by someone else on my team.

[chrisfillmore]

Random thought: perhaps Shaka could define an interface for e.g PreloadManager, and have a SimplePreloadManager implementation. The client app could override this implementation to customize functionality.

[chrisfillmore]

While writing my response to #1500, I realized a problem with preloading may be, where does NetworkingEngine come from? Should a "preload instance" (whatever it looks like) have its own NetworkingEngine?

[joeyparrish]

In my plan, Player would have a preload() method alongside load(). Since there's no separate PreloadManager, there's no separate NetworkingEngine. The preload() method would return an object that represents that particular preload operation, and allows you to upgrade any preload to a full-blown load(). If Player were destroyed, any associated preload object would be invalidated.

[chrisfillmore]

Will licenses be preloaded? If so, how will the client handle custom authorization? We would need to be able to handle formatting license requests for individual streams (our license proxy resides at a static URL, and identifying information is provided in the request body).

[joeyparrish]

Quick update: based on our timeline for v2.6 in Q2 2019, we expect to be working on preload in Q3. Refactors planned for v2.6 stand in the way of preload.

[OrenMe]

Hi @joeyparrish I saw this has been moved to backlog? is there any due date estimation for this feature?

[joeyparrish]

There is no due date yet. We haven't begun work yet, but it's on the top of our list, along with several other features like HLS discontinuities and ad SDK integration. Though preload was originally planned for v2.6, refactoring took much longer than anticipated, so we cut the v2.6 milestone down.

[OrenMe]

Thanks @joeyparrish, if I must use one video element I assume the current "workaround" is to: when it is time to preload content do

1. create new Instance
2. attach dummy video element
3. load requested content to be preloaded

when it is time to promote preloaded content do

1. call detach
2. call attach with the main video element
3. call load again with same video content - it will load content form cache

Is there another way to accomplish this?

[osmestad]

How we do it currently is to always have two video elements (and Shaka instances) and then switch which instance is visible to the the user (and that we listen to events from) at preloaded content switchover time :-)

[OrenMe]

Yes, this will generally work, but the are some limitations like managing autoplay restrictions on video elements, or devices that support only one video element at a time(smart TVs, cast device etc) Having such an api to load the data and possibly storing it in an array of array buffers and feeding it to a source buffer that will be attached to only one vide element will enable support across platforms with different limitations.

[joeyparrish]

Thanks @joeyparrish, if I must use one video element I assume the current "workaround" is to: when it is time to preload content do

1. create new Instance
2. attach dummy video element
3. load requested content to be preloaded

when it is time to promote preloaded content do

1. call detach
2. call attach with the main video element
3. call load again with same video content - it will load content form cache

Is there another way to accomplish this?

I don't think this will work at all. When we're attached to a media element, the buffered content will be in MSE SourceBuffers attached to that video element. When you detach from that element, you will lose everything you've buffered.

This is why preload is more than just a missing API. We have to rearchitect some of the components to separate StreamingEngine (which fetches content) from MediaSourceEngine (which feeds it to the video element). To preload content, we need to be able to store fetched content in memory until we're ready to send it through MediaSource to a committed video element.

[diogoazevedos]

Is there a way to achieve the preloading by caching the manifest and segments somewhere else, like in the fetch layer?

[enjikaka]

@diogoazevedos Like @osmestad mentioned earlier, having two shaka instances and switching between them is the best you can do for now. By doing that it will "prefetch" the upcoming chunks to be able to start pretty quickly.

If you instead of that method cache the manifest in for example the service worker you are still going to have boot time (shaka fetching chunks) when switching songs, so that's not going to be faster than two instances.

For gapless, proper integration with MediaSource is needed like so -> https://developers.google.com/web/fundamentals/media/mse/seamless-playback

[NoChance777]

@enjikaka The approach with two instances sometimes doesn't work, because a browser's policy requires a user activations/interactions for producing actions such as autoplay. See details here https://developers.google.com/web/updates/2017/09/autoplay-policy-changes

[enjikaka]

@NoChance777 You won't ever get around the autoplay block. But you only need to activate the media element played first, the second one in shaka instance two will be allowed to play after.

[NoChance777]

@enjikaka @OrenMe gave the more extended answer https://github.com/google/shaka-player/issues/880#issuecomment-590261093 why two instances won't work and from my experience, I agree with him

[enjikaka]

@joeyparrish Would this planned Preload API also have support for gapless playback or should that be a separate issue? In our case the dash chunks of the next song needs to be appended to the SourceBuffer of the MediaSource for gapless to happen, ref this link from before.

[joeyparrish]

@enjikaka, that sounds more like #764, "A method to stitch together VOD clips". That would allow you to take multiple VOD clips and play them as if they were a single multi-period DASH manifest or concatenated HLS with discontinuities. It would apply equally to HLS and DASH, and you could even mix them. It would not be possible to append anything to the end of a live stream, however, since live streams effectively have no fixed end.

See also this comment: https://github.com/google/shaka-player/issues/764#issuecomment-455549866

[michellezhuogg]

Hi everyone,

Our design for Preload API is posted for review. Please take a look and let us know if you have any questions, comments or suggestions. Thank you!

[OrenMe]

@michellezhuogg this looks really great. Thank you! My initial thoughts -

1. preload constraints - how does the preload behave in terms of ABR? How can I control what bitrates, codecs and languages are downloaded, is there a new config section that is passed to preload API? I would assume that you can just define the fixes bitrate and languages of text and audio tracks and it will behave more like a download manager without any ABR logic.
2. Can I pause/resume a loader during preload? Let’s say I see that player is buffering or lowering the playback level of playing content due to network - I would like to see if stopping the other preloders affect it. Maybe even have some baked in buffer control with configuration to pause preloaders if player hits buffering.
3. Are you adding more events to this new flow so app can understand what is the preloading status?
4. Have you thought about using additional storage beside memory, like local cache or actually reusing some parts of the download manager to store it locally? This can reduce memory usage especially if you need to preload a large amount of videos. It might come on expense of read speed for start time if accessing storage APIs are slower but this needs to be verified.

[michellezhuogg]

Hello @OrenMe ,

Thank you for your feedback!

1. That's a great question! Are you thinking about having different configs for different loaders? If so, we can provide configurations to choose codecs and languages for Preload API at the Loader level, same as the configurations at the Player level. If both the configurations in Player and Loader are set, the configs on the Loader level would take privilege.
2. Currently, we don't have the pause/resume mechanism. For VOD content, Preload downloads the init and the first segment, so we don't need to pause the downloading process. For live stream content, Preload downloads of the latest segment, and we can add a config in the future to control that if it causes buffering problems.
3. We will definitely add more events to indicate the preloading status.
4. With one or two segments downloaded for each content, our estimate is that memory would be sufficient. We'll take additional storage options into consideration if we see that as a problem.

Let us know if you have further questions or thoughts! Thank you!

[michellezhuogg]

Also, we would be happy to discuss implementation details if anyone is interested in implementing this and create a PR for it. Thanks!

[girayk]

any idea more less when that feature will availible?

[joeyparrish]

Nobody is working on the implementation at the moment, but contributions are welcome. Let us know if you'd like to take on the implementation, or if you have any questions about the design we published. Thanks!

[girayk]

I have no idea to how to implement. :( Even I couldn't find any info on how I can prefetch video and read that from the cache without SW.

[joeyparrish]

You probably couldn't find any info on that because it wouldn't quite work like that. Preload would be a Shaka Player feature loading media into memory and keeping it buffered there without connecting to MediaSource SourceBuffers. It wouldn't use Cache or ServiceWorker to fetch or store the media.

This is our design doc: https://github.com/shaka-project/shaka-player/blob/main/docs/design/preload.md

No worries if you don't want to tackle this yourself. We will get there eventually, but the team is prioritizing HLS improvements right now.

[zangue]

Hi everyone,

As it appears that this feature request is not currently being worked on, and, considering it's a pretty valuable and anticipated one, I've conducted some investigations and I'd like to share my ideas (an alternate design proposal) with you on how I think we could go about adding this feature to the shaka player. I would appreciate your feedback.

Proposal

The current design document suggest as a first step to extract a part of the load graph in a loader class but I believe an easier first step would be to enable single manifest preloading with a player instance. I believe this is better approach because it's more "accessible" (less code refactor needed) and it will enable preloading single manifest already which, imo, is a valuable feature the shaka player community can start benefiting from. As next steps, we should make critical components reusable and then introduce the loader class to make multiple manifest preloading in parallel possible. In the following sections I will present the steps in more details.

1. Enable single manifest preloading with a player instance

This step can be broken down into two substeps: (a) enable "detached" initialization of the player and (b) add preload capability.

1.a Enable "detached" player initialization

This means being able to load the player without video element: With the current load graph we always need to attach the player to a video element first. This is not strictly necessary as all the critical components (manifest parsers, media source engine, DRM engine, text displayers) can be refactor to be initialised without media element (actually it's only the media source engine and the text displayers that need said refactor). The load path will change from: Detach -> Attach -> Media Source -> Parser -> Manifest -> DRM -> Load To: Detach -> Media Source -> Parser -> Manifest -> DRM -> Attach -> Load

So that the following will be possible:

const player = new shaka.Player();
// Init/load the player in detached mode i.e.
// Detach -> Media Source -> Parser -> Manifest -> DRM
// Note that the first two step can be done in the constructor already.
player.load(manifestUri);
// Starts streaming when a media element gets attached
player.attach(mediaElement);

The player will remain backwards compatible with the current API

const player = new shaka.Player(mediaElement);
// Will init/load the player and start streaming right away
// Detach -> Media Source -> Parser -> Manifest -> DRM -> Attach -> Load
player.load(manifestUri);

To make this possible, we will need to remove the media element dependency for the media source engine construction (MediaSource initialization doesn't require media element) and add an attach(mediaElement: HTMLMediaElement); method to provide a media element later on (at which point the object URL will be created and the media element tied to the previously created media source). Similarly the text displayer will be refactored to be provided with a media element once available; Furthermore, the load graph for media source playback will be changed as described in the following diagram (note that some nodes are duplicated to simplify the diagram).

I've been experimenting with the above and currently have a working PoC for this substep. I'd be happy to submit a draft PR for feedback.

1.b Add preload capability

Now we can do the following:

const player = new shaka.Player();
player.load(manifestUri)

which will walk the load graph like this: Detach -> Media Source -> Parser -> Manifest -> DRM We don't have a media element but at this point we already have a parsed manifest and an instance of the streaming engine which is all we need to start preloading. I can see two ways of approaching this: a. we add a preload() method to the streaming engine to instruct it to start loading media segment into memory. In this case the streaming engine is aware it's preloading and will need to manage preload buffers accordingly or b. we make it transparent to the streaming by introducing a layer of abstraction between the streaming engine and the media source engine. Said layer will provide an interface similar to the media source engine to append to and manage buffers and will be responsible of processing and eventually appending the media data to media source engine. A minimal interface would look like this

// Manages fetched media data:
// - Keep them in memory while preloading (MS not attached)
// - Push data to source buffer once a MS engine is provided.
// - Provide interface to manage buffered/staged data
const BufferSink = class {
    // Each buffer sink manage one type of content.
    constructor(contentType: shaka.util.ManifestParserUtils.ContentType) {
        this.buffer_ = ... // Some adequate data structure to store data + append context
        this.mediaSourceEngine_ = null;
    }

    // Before media source engine is provided, keep data in memory
    setMediaSourceEngine(mediaSource: shaka.media.MediaSourceEngine) {
        this.mediaSourceEngine_ = mediaSource;
        // We have the MSE now, process buffer data i.e. pipeline data to MSE
        this.bufferWorkLoop_(); // async, similar to operation queue work in MSE
    }
    // Keep data in memory if not attached to the MS engine else pass down data
    // to the MS engine
    async appendBuffer(/** same args list as MS engine*/) {
        this.buffer_.push(...);
        // ...
    }
}

Then, in the streaming engine we can have a buffer sink per media state so that after fetch the data will be appended by calling mediaState.bufferSink.appendBuffer(...). Obviously there is more to it but I hope I could successfully convey the general idea. After the above step is completed, preloading manifest will be possible. The feature could put behind a config flag e.g. config.streaming.preload; Note: The part where we initiate the preload can be added as an separate "Preload" step in the load graph.

3. Make resources as reusable as possible

Now that we can preload single manifest, the next step towards preloading multiple manifest in parallel is to make resources as reusable as possible. Among reusable resources I can identify:

• Media Source Engine
• Manifest parsers: Once a parser is created we can keep it throughout the whole player lifetime. E.g. if the previous load creates e.g. a DASH parser and the next load is a DASH manifest we re-use the already existing parser for it (Though, for this to work, the parsers will need to be made stateless).
• Graph walker implementation (not too sure about this one but should be possible)

The above resources will be owned by the player and can be provided to the loaders (see next section) on demand e.g. via an interface.

4. Introduce loader class

Now we will be ready to add loader class as described in the current design document but with the following suggestions:

• Non reusable resources are created by the loader. Non reusable resources include: the DRM engine, the Streaming engine, the manifest, the graph walker instance, the player configuration. When a loader is passed to player.load(), its resources are transferred to the player so that the player have all required resources and the correct context to start playback.
• Instead of extracting parts of the load graph into the loader class, each loader class creates the whole graph (new walker instance) and walks it until the preload step. Once the loader gets loaded (player.load(loader)), the walker (together with the streaming engine and DRM engine) is transferred and taken over by player which then walks the remaining steps in the graph until load. See diagram below for illustration.

Considering the above a loader class could look like this:

// Pseudo code loader
shaka.Loader = class extends shaka.util.FakeEventTarget {
    constructor(playerInterface, playerConfig, manifestUri, startTime, mimeType) {
        ...
        this.playerInteface_ = playerInterface;
        this.playerConfig_ = playerConfig;
        this.streamingEngine_ = ...;
        this.drmEngine_ = ...;
        this.manifest_ = null;
        // walker for the whole load graph. Will be transferred to the player
        // when it's time to load
        this.walker_ = new shaka.routing.Walker({
            playerInterface.getNodes().detach,
            playerInterface.createEmptyPayload(),
            playerInterface.getWalkerImplementation()
        });
    }

    start() {
        // Walks the load graph until preload
        this.walker_.startNewRoute((p) => {
            return {
                node: this.playerInterface.getNodes().preload,
                ...
            };
        });

        // When it's time to parse the manifest
        this.manifest_ = await this.playerInterface_.parseManifest(this.manifestUri_, this.mimeType_);
        ...
    }

    getWalker() {
        return this.walker_;
    }

    getDrmEngine() {
        return this.drmEngine_;
    }

    getManifest() {
        ...
    }

    ...
};

/**
 * @typedef {{
 *   parseManifest: function(string, string): Promise<shaka.extern.Manifest>,
 *   getWalkerImplementation: function():!shaka.routing.Walker.Implementation,
 *   ...
 * }}
 */
shaka.Loader.PlayerInterface;

And in the player it could be used as follow:

// Pseudo code player
shaka.Player = class extends shaka.util.FakeEventTarget {
    ...
    preload(playerConfig, manifestUri, startTime, mimeType) {
        const playerInterface = {
            ...
        };
        const preloader = new shaka.Loader(playerConfig, manifestUri, startTime, mimeType, playerInterface);

        preloader.start();

        return preloader;
    }

    async load(preloader) {
        // Must certainly do things before...
        ...
        // Transfer resources
        this.config_ = preloader.getConfig();
        // this.applyConfig_(preloader.getConfig())?
        this.drmEngine_ = preloader.getDrmEngine();
        this.streaminEngine_ = preloader.getStreamingEngine();
        this.walker_= preloader.getWalker();
        ...

        // Invalide loader
        async preloader.destroy();

        if (preload) {
            assert(this.walker_.getCurrentNode() == this.preloadNode_, 'Manifest should have been preloaded already!');
        }

        // Proceed with load i.e. new from "Preload" to "Load"
        // Note: if we don't have a video element yet, we will wait for `attach()` to start this route.
        this.walker_.startNewRoute((p) => {
            return {
                node: this.loadNode_,
                ...
            }
        });

        // Do more things...
    }
};

5. Preload API

Once all the above steps are done the preload API for loading multiple manifests in parallel should be fully functional and the player can the provide a preload() method with the following signature:

shaka.Player = class extends shaka.util.FakeEventTarget {
    /**
     * @param {string} manifestUri The location of the manifest to preload
     * @param {number=} startTime The time in the presentation timeline we should preloading from
     * @param {string=} mimeType Manifest MIME type
     * @param {shaka.extern.PlayerConfiguration=} playerConfig Specific player config
     * @return {!shaka.Loader}
     * @throws
     */
    preload(manifestUri, startTime, mimeType, playerConfig) {};
}

Example usage:

const player = new shaka.Player();

const loader1 = player.preload('foo1.mpd');
const loader2 = player.preload('foo2.mpd', 10, undefined, {streaming: { bufferingGoal: 15 } });
// Can also be done after load() as well
player.attach(video);
player.load(loader1);

Open questions

How should we deal with player events that occur inside the loader instance during preload?

It ended up being a lengthy document, I appreciate if you took the time to go through all of it. Please don't hesitate provide feedback. Thank you!