DanielBaulig commented 7 years ago

The Videos team at Facebook has a growing need to move calls to Media Source Extensions off of the main thread and into a Worker. While most parts of your off-the-shelve streaming video player can be implemented in a Worker (fetching and parsing of the manifest as well as selecting and fetching of segments) the calls to and interactions with the Media Source Extensions API (e.g. SourceBuffer.prototype.appendBuffer, SourceBuffer.prototype.remove, etc) remain a bottleneck and have to happen on the main browser rendering thread.

On a sizable and complex web site large blocks of JS execution can regularly block the main thread making videos slow to start playing and causing lots of rebuffering, especially in live videos where you can only buffer ahead very little data. To ensure great web video performance in the long run, even as web apps continue to grow in size and complexity, we believe we need to allow video players to move into Workers and off of the main thread.

We would like to propose this change to the working group and involve other API users as well as implementors in the discussion.

bmaurer commented 7 years ago

14 also seems relevant here -- based on my reading of that bug it would help with part of this problem (once you fetch a segment you'd pass a stream to that audio to MediaSource and would bypass the main thread, or even a worker thread in terms of getting that data to where it needs to go).

But that still leaves the issue of fetching a manifest and initiating a request for a segment

jyavenard commented 7 years ago

the calls to and interactions with the Media Source Extensions API (e.g. SourceBuffer.prototype.appendBuffer, SourceBuffer.prototype.remove, etc) remain a bottleneck and have to happen on the main browser rendering thread.

That it's a bottleneck would be mostly a user agent implementation issue. Those APIs are designed to be asynchronous, they should take no time at all and not block the main thread. With current Firefox, running those two operations in a worker would make little (or no) performance difference, they are in effect running internally in a worker.

I wish issue #100 progressed further, it would certainly facilitate handling those operations.

bmaurer commented 7 years ago

The issue here isn't the cost of video on the main thread. The issue is that if the main thread is busy with other work (say the javascript needed to render a page) that it can not be responsive to input from the network. This means that the video will be slower to buffer even though the client has been able to get the video over the network.

The goal is for the video loading code to be isolated on a worker thread where it can operate in real time without the risk of latency induced by work on the main thread.

zhaoz commented 7 years ago

We are seeing the same issue on YouTube and I'd like to add our voice to support a solution here as well. Moving append buffer off the main thread would greatly simplify the juggling we need to do to prioritize video playback.

bmaurer commented 7 years ago

It's great to see that youtube has the same take on this issue. Would it be worth taking the next step here and drafting out what a spec change would look like? I think we can model this on the offscreen canvas api (https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/)

var canvas = document.getElementById('myCanvas'); var offscreen = canvas.transferControlToOffscreen(); var worker = new Worker('worker.js'); worker.postMessage({ canvas: offscreen }, [offscreen]);

Essentially MediaSource would offer an API which would pass control of the media source APIs to a worker thread. Any APIs that MediaSource offers (eg to add source buffers) would only be callable from the worker.

MediaSource is a fairly complex API with a number of events / methods. It'd be great to get some feedback from @zhaoz, @DanielBaulig as well as folks who work on browsers and this spec to comment on what kind of threading issues might come up with this API.

DanielBaulig commented 7 years ago

The main limitation with transferring MediaSource object that I can currently see is that in general most video player implementations heavily depend on certain video node attribute for determining which video segments to fetch and play. These are as far as I can tell primarily video.currentTime, video.paused and video.buffered. The main thread will need to regularly update the worker with the current values of these for most video player implementations to be able to operate correctly.

However, the API itself could have support for this: the worker could get copies of these values on every event loop exposed. This exposure could happen as part of the transferred MediaSource object, e.g. on a bag of exposed values or directly on the object:

onmessage = function(message) {
  if (message.mediaSource.mediaElementState.paused) {
    // don't do anything while the video is paused
    return;
  }
};

Another problem may be that the main thread indirectly retains a reference to the MediaSource as an object URL on the videos src attribute. I am not sure how this will complicate the safe usage of object URLs.

bmaurer commented 7 years ago

That makes total sense. It does seem like the MediaSource API should be responsible for relaying that information -- accessing it directly from the video element seems like it violates layering and would make the threading far more difficult here.

One other thing we should think about -- maybe this is a totally separate issue -- I worry about the latency of starting the worker, downloading the necessary JS, etc. On Facebook we've found that the amount of JS necessary to have the logic for streaming the video can be pretty high. I wonder if there's a way to allow the server to list the URLs necessary to start streaming so that they can be fetched and inserted into the buffer while the more advanced control starts up.

DanielBaulig commented 7 years ago

One other thing we should think about -- maybe this is a totally separate issue -- I worry about the latency of starting the worker, downloading the necessary JS, etc. On Facebook we've found that the amount of JS necessary to have the logic for streaming the video can be pretty high. I wonder if there's a way to allow the server to list the URLs necessary to start streaming so that they can be fetched and inserted into the buffer while the more advanced control starts up.

I don't think there's anything preventing video player implementations from doing something like this currently. That said, in the context of a transferable MediaSource, this becomes a little harder. This could potentially mean creating a MediaSource object and several sourceBuffers on the main thread, appending data to the sourceBuffers and only later transferring ownership to the Worker. This transfer won't be simple anymore though since there's a complex mix of objects and possibly tasks that has to be dealt with.

jyavenard commented 7 years ago

One of the issue in dealing with a sourcebuffer in a worker thread, is how, currently, the source buffer notifies that the current operation has completed: through events fired on the attached media element and which is only ever accessed on the main thread.

So if your main thread is busy, no update/updateend event and you're back to the issue of not being able to keep up with your source buffer.

This problem could be easily worked around if rather than relying on events such as update/updateend to know when you can feed new data to the source buffer we used Promises. I opened an issue a while back to implement this: https://github.com/w3c/media-source/issues/100

The beauty with SourceBuffer::appendBuffer and SourceBuffer::remove with returning Promises is that it would be fully backward compatible. Once the appendBuffer operation has completed, resolve the Promise and queue update/updateend Up to the player to use one mechanism or the other, never both.

Making a SourceBuffer usable on a worker thread wouldn't be a too complicated task (famous last words) in gecko, the architecture is already asynchronous and thread-safe. The interaction with the media element would be the issue.

But if we limit the interaction to the source buffer to appendBuffer and remove operations, and checking of the buffered range to be performed on the source buffers alone, and we use Promises then it should be straight forward.

wolenetz commented 7 years ago

@https://github.com/w3c/media-source/issues/175#issuecomment-278253566, 'updateend' is fired on the SourceBuffer , not HTMLMediaElement. However, readyState transitions (and consequent events like 'loadedmetadata') would occur on the media element on the main thread.

I'm less clear that Promises are required for MSE-on-SW, though I'm not opposed (I just don't want to tangle the two features).

wolenetz commented 7 years ago

Here's a summary of this issue so far, from my understanding. I have a set of questions, below, for which web author input would be greatly appreciated.

Proposals in [2] and [3] help to limit the scope of this to be specific to just MSE, and not require exposing all of HTMLMediaElement API also to Service Worker (SW):

[2] Proposes that the MediaSource API be made “transferrable” to a SW [3] Adds that major HTMLMediaElement attributes necessary for MSE API usage be periodically copied from the main thread to SW (perhaps as part of the transferred MediaSource object).

[1] https://github.com/w3c/media-source/issues/175 [2] https://github.com/w3c/media-source/issues/175#issuecomment-271909355 [3] https://github.com/w3c/media-source/issues/175#issuecomment-272210121

Questions:

Should the “transfer” be allowed on a MediaSource object which has already had SourceBuffers added to it? (This might allow for initial buffering to occur on main thread on page load, while the SW context is still spinning up.) (I’m not clear this will help so much in practice, as some web players already experience high time to first frame due to other activities contending on the main thread immediately on page load.)
Is there potential “safe usage of Object URLs” problem with transferring the MediaSource context to a SW? (I would defer to web platform security experts. I suspect that a set of reasonable restrictions/checks on the transfer may be needed. Any suggestions?)
What portion of the MediaSource and SourceBuffer API need to be transferred?
What portion of the HTMLMediaElement state need to be periodically copied into the transferred MediaSource’s SW context?
Precisely when should such state be copied?
Are any HTMLMediaElement events needed in the SW context? (This would increase complexity; I’m not certain any are needed in SW context, since the web app could forward these -- though with consequent potentially high latency -- into the SW context itself.)
Once transferred to SW, would the original MediaSource be restricted from access from any other context?
I assume the HTMLMediaElement itself must provide continued access; though would queries of it like “buffered”, which MSE extends, need to round-trip into the SW context?

-- edit: substitute "Worker" for "Service Worker" in this post, please. I had conflated them.

DanielBaulig commented 7 years ago

Am I assuming correctly that you are using the term Service Worker and the general Worker API interchangeably?

I'll try to answer most of the questions for which I believe I have enough context:

Should the “transfer” be allowed on a MediaSource object which has already had SourceBuffers added to it? (This might allow for initial buffering to occur on main thread on page load, while the SW context is still spinning up.) (I’m not clear this will help so much in practice, as some web players already experience high time to first frame due to other activities contending on the main thread immediately on page load.)

This is probably not necessary. The sourceopen event takes some time to fire and the application probably needs time to load data to append to the source buffers too. If those steps take significantly less time than spinning up the Worker then there may be some room for optimization, but I believe that this is likely better done by improving Worker startup time, than introducing bloat here. This obviously assumes that we do not have to wait for the "sourceopen" event to fire before we can transfer the MediaSource object.

Is there potential “safe usage of Object URLs” problem with transferring the MediaSource context to a SW? (I would defer to web platform security experts. I suspect that a set of reasonable restrictions/checks on the transfer may be needed. Any suggestions?)

I am not sure I can answer this question.

What portion of the MediaSource and SourceBuffer API need to be transferred?

We've spent some time internally thinking about this and we came to the conclusion that the entirety of the SourceBuffer API is probably the minimally viable subset of the API for the application.

Having the MediaSource object itself available is preferred but not necessarily required. We believe though due to the amount of dependencies between the MediaSource object and it's SourceBuffer objects, that transferring the MediaSource object is likely the easier approach from a spec and implementation perspective.

We'd love to hear some thoughts from implementors on this though.

What portion of the HTMLMediaElement state need to be periodically copied into the transferred MediaSource’s SW context?

Are any HTMLMediaElement events needed in the SW context? (This would increase complexity; I’m not certain any are needed in SW context, since the web app could forward these -- though with consequent potentially high latency -- into the SW context itself.)

The Worker would generally require access to information that is helpful in making decisions on how much, how fast and what kind of data to buffer. Generally this could be the current playhead (currentTime), the current state of the video buffer (buffered attribute) and current payback rate (playbackRate), if audio is audible (muted and volume) and probably more. The exact information here however is highly dependent on the details and sophistication of the application and probably highly vary from application to application. Note also, that a small delay in updating the Worker with this information should not lead to any significant problems for the application. In fact, there are plenty of applications and scenarios conceivable in which no state synchronization at all would be required. I think generally the same is true for any HTMLMediaElement events.

For those reasons and to make the specification and implementation process easier we would probably suggest to leave this synchronization completely up to the application. The application can use postMessage to update any necessary state in the Worker as required.

Once transferred to SW, would the original MediaSource be restricted from access from any other context?

I think that is the current understanding.

I assume the HTMLMediaElement itself must provide continued access; though would queries of it like “buffered”, which MSE extends, need to round-trip into the SW context?

This seems to be more of an implementation / specification related question that I am not entirely sure I am able to give feedback on. My understanding here however is that the buffered attribute both for sourceBuffer as well as the video node already need information from the (already) asynchronous and in a different thread running process of media decoding.

wolenetz commented 7 years ago

Thanks for your answers - those do help describe the possible feature scope more precisely. w.r.t.:

Am I assuming correctly that you are using the term Service Worker and the general Worker API interchangeably?

You're indeed correct - I had mistakenly constrained the scope too far :). Unless the general Worker API is too broad, let's target it, not just Service Workers.

wolenetz commented 7 years ago

@https://github.com/w3c/media-source/issues/175#issuecomment-299345346:

My understanding here however is that the buffered attribute both for sourceBuffer as well as the video node already need information from the (already) asynchronous and in a different thread running process of media decoding.

Decoding itself doesn't impact buffered ranges (unless there's a decode error and transition of MediaSource readyState to closed.) Rather, operations on the MSE API itself can change the buffered ranges. What I was trying to get at is: what are the potentially required synchronization points between the HTMLMediaElement attached by a transferred MSE worker context? For example:

HTMLMediaElement.buffered and duration: If an app synchronously queries the 'buffered' attribute of such an HTMLMediaElement, yet the MSE in worker context is in the process of, or perhaps has recently completed, a SourceBuffer appendBuffer(), SourceBuffer remove(), SourceBuffer mode or timestampOffset change when previously 'ended', MediaSource endOfStream(), or MediaSource set duration operation, is there some guarantee of synchronization of the result of such operation that needs to be handled? What should such a guarantee look like? Note that actual playback/decode doesn't factor into this (buffered time range updates due to appendBuffer()/remove()/duration setting don't require actual decode). I assume there needs to be some specified synchronization of such results; in non-worker/current case, if the SourceBuffer.updating attribute is false across all SourceBuffers in the attached MediaSource, then the buffered range query on the parent media element should be stable and correct. Is some similar guarantee required in the worker case?
HTMLMediaElement.seekable and set/clearLiveSeekableRanges (also involves HTMLMediaElement.buffered state): ditto: should there be some guarantee of synchronization with the worker before a query of seekable can be satisfied?
Worker MediaSource.buffered (potential new API method): Note that the worker context can itself inspect and intersect manually the SourceBuffer buffered ranges to mimic what the parent element's logic would do. Should the transferred MSE API provide such copy of the parent media element's logic that can be processed without synchronizing state with the parent media element?

bmaurer commented 7 years ago

These are great questions. I think my intuition is that generally the worker thread which holds the MSE will be communicating with the main thread, eg via postMessage. I think it would make sense to enforce logical causality. For example, if you call appendBuffer, then send a message to the main thread the result of that appendBuffer should be observable when processing the send message.

I agree that MediaSource.buffered probably makes sense -- there's no need to query the main thread for this information.

wolenetz commented 7 years ago

For example, if you call appendBuffer, then send a message to the main thread the result of that appendBuffer should be observable when processing the send message.

To be clear, I assume you mean more like "if you call appendBuffer, and upon later receiving updateend you send a message to the main thread, the result of the appendBuffer should be visible on the main thread when processing the receipt of that message." Please correct me if I misunderstand :)

wolenetz commented 7 years ago

Also, does it make sense to use such a transferred MSE API from a Service Worker context (I don't think so, because such a context is independent of a web page, and therefore from a DOM containing a media element). ISTM, this makes sense from a web worker context, not from SW context.

bmaurer commented 7 years ago

To be clear, I assume you mean more like "if you call appendBuffer, and upon later receiving updateend you send a message to the main thread, the result of the appendBuffer should be visible on the main thread when processing the receipt of that message." Please correct me if I misunderstand :)

yep, exactly.

Also, does it make sense to use such a transferred MSE API from a Service Worker context (I don't think so, because such a context is independent of a web page, and therefore from a DOM containing a media element). ISTM, this makes sense from a web worker context, not from SW context.

By worker I meant web worker here

DanielBaulig commented 7 years ago

I agree with what @bmaurer said: maintaining logical causality with postMessage probably makes sense.

If we are considering replicating the buffered HTMLMediaElement extension on MediaSource - which I totally think makes sense -, we should probably do the same thing for seekable.

wolenetz commented 7 years ago

Thanks for your responses to my questions. Here's my newbie-to-workers question: can a Worker do a fetch that is intercepted by a Service Worker, thereby enabling a combination of offline MSE-in-Worker being fed by data provided by the Service Worker? Where are known gaps in spec or implementation to accomplish such a scenario (other than, of course, this issue's MSE-in-Worker discussion :) )?

kentuckyfriedtakahe commented 7 years ago

I'm with @jyavenard thinking that adding #100 while workerising SourceBuffer would be worthwhile. It may be timely because the worker codepath may end up being a fork of the non-worker path. I'd like to hear from @bmaurer and @zhaoz about whether this will make the player implementation simpler or better.

bakulf commented 7 years ago

I would like to share my point of view about the transferring of MediaSource and SourceBuffer objects to workers. If I have understood the spec correctly, a normal use of a MediaSource is via HTMLMediaElement.src = URL.createObjectURL(MS). The output of URL.createObjectURL() is a DOMString, fully cloneable via postMessage().

Would be probably cleaner to expose MediaSource and SourceBuffer constructors to workers. A worker can create a MediaSource and SourceBuffer(s) then, simply send a blob URL to the window: worker.onmessage = e => { mediaElement.src = e.data; }

The change I propose is mark MediaSource interface in this way:

[Constructor, Exposed=(Window,Worker)]
interface MediaSource {...

same for SourceBuffer.

Note that SourceBuffer supports the appending of ArrayBuffer and ArrayBufferView. Both are already transferable to workers. Another important point, maybe the most important, so far, EventTargets are not transferable to workers because EventHandler cannot be handled correctly when the object is transferred.

alastor0325 commented 6 years ago

Agree with comment, it would be easy if we restrict the media source and source buffer could only be used in one thread all the time and disallow them being transferrable.

If they’re transferable, how should the behavior of event handlers be defined? It seems a difficult problem, and the spec doesn't mention anything about it.

eg. If I transfer an media source before its “onsourceopen” be called, which thread the event should be dispatched? If the source buffer can also be transferred, whether we can transfer multiple source buffers to multiple workers? …e.t.c

—

If what I understand it correctly, the usage scenario would like the following.

First, we create both media source and source buffer on the WT, and send the URL back to MT after wrapping media source as URL. After the media source is attached successfully to media element, “sourceopen” would be triggered on WT. Next, MT should send the video URL to WT, then all things could be done independently inside the WT from fetching data to appending data into buffer (and with some new API for getting the status of media element).

Please correct me if I was wrong and welcome to propose other possible scenarios.

DanielBaulig commented 6 years ago

I see the intentions here and agree that this would make several aspects easier. The only concern I have is that this would add a significant amount of additional startup time.

On top of the time it takes to complete sourceopen, now we may also have to wait for a Worker to spin up and a message to be sent form the Worker to the main thread (which may be busy doing other stuff). All these things will have to happen in sequence and it will require two dedicated time slices on the main thread to complete.

In a scenario with a transferable MediaSource, waiting for sourceopen and the bootstrapping of the Worker can happen in parallel and the Worker will not have to round trip back to the main thread to complete the transaction reducing the main thread time slices require to one.

The biggest risks associated with moving MSE into Workers from a product perspective is probably the startup latency. The solution as suggested in https://github.com/w3c/media-source/issues/175#issuecomment-327715837 increases this risk significantly.

alastor0325 commented 6 years ago

This might be a stupid question, why couldn't we bootstrap worker in advance, before starting playing video?

Can we bootstrap worker in the beginning? and then use postMessage() to notify worker to return the URL of media source when we would start MSE soon?

In a scenario with a transferable MediaSource, waiting for sourceopen and the bootstrapping of the Worker can happen in parallel and the Worker will not have to round trip back to the main thread to complete the transaction reducing the main thread time slices require to one.

If we can start worker in advance, the following might be the possible use case. In this case, MT only need one time slice to handle the message sent from WT.

worker.postMessage(url /* used for fetching on WT */);
worker.onMessage = e => { mediaElement.src = e.data; };

alastor0325 commented 6 years ago

The main problem to make media source transferable is "how to handle the event handler of transferable object", because it's undefined behavior.

I think this topic is out of the media scope, and I would like to know how strong intention we want to go with this direction? If do so, we need to discuss and modify with worker's working group and make sure the behavior can be well defined.

However, it might be take a long time, and @bakulf, who is our worker expert, also suggest don't try to transfer an object with event handler.

DanielBaulig commented 6 years ago

@alastor0325 We can only bootstrap the Worker ahead of time, if we have an opportunity to do so. If the first thing we are doing on a blank page load is to create and assign a MediaSource, there's no way for us to bootstrap a Worker ahead of time.

In addition to that, creating the object URL on the Worker thread will require us to touch the main thread twice. Consider the following example:

    /* main thread code */
    // first main thread execution
    const video = document.querySelector('video');
    const worker = new Worker('mse_worker.js');
    worker.addEventListener('message', (event) => {
      // second main thread execution
      video.src = event.data;
    });

But if we can create the MediaSource on the main thread, we only need one time slice on the main thread:

    /* main thread code */
    // first main thread execution
    const video = document.querySelector('video');
    const worker = new Worker('mse_worker.js');
    const ms = new MediaSource();
    const url = window.URL.createObjectURL(ms);
    video.src = url;
    worker.postMessage(ms, [ ms ]);

We don't need a second time slice on the main thread before we can start loading video.

Talking to some folks at Mozilla, they wanted to look into why the sourceopen event even is required/asynchronous in the first place.

Another option could be to require transferring of the MediaSource to the Worker before the URL assignment happens. In that case any events firing on the MediaSource could be deterministically scheduled to execute on the Worker instead of the main thread.

Lastly, I could imagine something TransferableMediaSource, that internally does all the steps MediaSource would do upon creation, but doesn't expose any external API interface except for a getMediaSource() method. After the method was called, the TransferableMediaSource looses it's ability to be transferred. This could be created on the main thread, assigned to the video node's src through an Object URL and then transferred to the Worker, where the Worker would call getMediaSource to obtain a reference on the "true", underlying MediaSource.

jyavenard commented 6 years ago

Why not only having the source buffers in a worker? They really are the only ones we care about when it comes to feeding data fast enough.

alastor0325 commented 6 years ago

@DanielBaulig I think it's a good idea to have "TransferableMediaSource", it could fix the problem about the event handler and also it would only need one time slice on the main thread.

In addition, you've mention we would need something like "mediaSource.mediaElementState" which can be reflected the status of media element. Maybe we could start to define the specific attributes which we need.

DanielBaulig commented 6 years ago

In addition, you've mention we would need something like "mediaSource.mediaElementState" which can be reflected the status of media element. Maybe we could start to define the specific attributes which we need.

For the most part I think it's sufficient to provide these media element states via postMessage to the worker at a regular rate.

Unless the platform can significantly improve update performance or maybe even provide thread-safe, but otherwise real-time variations of these, I don't think they are of highest priority.

The most important states/properties are probably currentTime and error. Virtually any media related property can be useful though to optimize certain conditions, prevent over- or underfetching and allow for a more consistent player and data fetching state (e.g. buffered, playbackRate, paused, seeking, ended, readyState, duration).

Why not only having the source buffers in a worker? They really are the only ones we care about when it comes to feeding data fast enough.

Generally this is correct. I remember that this was brought up before as an option, but I'm not entirely sure what the arguments against it were. There's definitely the limitation of having to touch the main thread twice though (you have to wait for 'sourceopen' to fire before you can create SourceBuffer instances), but I believe the major concern was about how to model the APIs around this, since SourceBuffers are exposed on several MediaSource APIs and all of them would become invalid when the SourceBuffer is transferred into a Worker.

wolenetz commented 6 years ago

I'm considering putting together a proposal to incubate MSE-in-Workers in WICG (similar to how we've been incubating codec switching). I've read through the history on this issue so far and am curious if there are any new concerns, suggestions or use cases folks might have since ~1 year ago.

estobbart commented 6 years ago

To me this is sort of at odds with Google's policy on autoplay & the MEI. If video is being done in a worker, it would hint to me that video playback isn't the primary objective of the user on the site. If it is the primary objective, can't the other rendering operations be limited instead of tying up the main loop? I do see some advantages in preparing MediaSource/SourceBuffers in a worker, but if that MediaSource blob were to be assigned to HTMLMediaElement, then it would be moved to the main loop once the user determines that the video playback is the most important to them.

bradisbell commented 6 years ago

@estobbart Google's MEI and autoplay policies are based on narrow assumptions about the scope of video on the web, and have actively damaged our ability to build more complex applications around client-side audio and video creation. While Google is certainly in their right to determine what's best for their users, I don't think those policies should have much bearing in a W3C discussion.

The proposal at hand seems to have the potential to resolve some performance issues, particularly around the loading of new content. Even if that's all that was solved (before the srcObject were set), it would be a significant improvement for some types of web applications.

bmaurer commented 6 years ago

If video is being done in a worker, it would hint to me that video playback isn't the primary objective of the user on the site. If it is the primary objective, can't the other rendering operations be limited instead of tying up the main loop?

Wanting to run the video decoder in a worker is not incongruous with the video being the user's primary objective on the site. While the idea of trying to limit the main thread is admirable, it's often something that's easier said than done. If the user's primary objective is to view the video, running it in a dedicated thread is the best way to guarantee that we provide smooth playback and isolate the user from other things that might be going on in the main thread.

wolenetz commented 5 years ago

I have a proposal for incubating a solution for this issue, hopefully with eventual inclusion in MSE vNext. Please take a look: https://github.com/wicg/media-source/blob/mse-in-workers-using-handle/mse-in-workers-using-handle-explainer.md

jernoble commented 5 years ago

This proposal seems unnecessarily complex; couldn't we achieve the low-startup-latency problem by adding a synchronous open() method, and solve the "in a worker" problem by exposing MediaSource to Worker contexts, and use the existing not-optimal-but-not-being-deprecated blob:// URL syntax as the transferrable handle to the MediaSource?

DanielBaulig commented 5 years ago

@wolenetz I have looked at the proposal and from an application developer perspective, this will give us the tools we need.

As discussed before and outlined in your open questions, having the ability to create the MediaSourceHandle in the Window context and extracting the MediaSource in the Worker context would be something that we would fundamentally be interested in, but we understand that the specification and implementation complexity might not be worth it.

Other than this, we do not see any objectionable or controversial parts in the proposal, and I think we can be fully supportive.

wolenetz commented 5 years ago

@jernoble @DanielBaulig @bmaurer @jyavenard I have a rough Chrome prototype of MSE-in-workers now beginning to approach "working" at https://chromium-review.googlesource.com/c/chromium/src/+/1405697. It is not landable in Chromium as-is, as several items (see CL description and patch list title history) need resolution. Especially, please be aware that cross-thread teardown/MSE detachment currently hits a crash (I hope to resolve before FOMS next week, but may not have time) -- in patch set 41. -- Update: I've fixed this teardown crash last night.

I look forward to any and all feedback, and will hopefully see many of you at FOMS/next week. Thanks!

[Edited to add Update:...]

beaufortfrancois commented 5 years ago

Here's some Javascript code to get a grasp of latest proposal:

// main.js

const worker = new Worker('worker.js');
worker.postMessage('pleaseCreateMediaSource');
worker.onmessage = event => {
  document.querySelector('video').src = event.data.blobUrl;
};

// worker.js

self.onmessage = event => {
  if (event.data == 'pleaseCreateMediaSource') {
    const mediaSource = new MediaSource();
    self.postMessage({ blobUrl: URL.createObjectURL(mediaSource) });

    mediaSource.open();
    mediaSource.onsourceopen = _ => {
      // TODO: Add source buffer, fetch video chunks, ... 
    };
  }
};

Context: https://groups.google.com/a/chromium.org/d/msg/blink-dev/CNRywDqgKjY/F0nnA4tTAwAJ

MattiasBuelens commented 5 years ago

Small nit-pick (EDIT: fixed)

> ```js > mediaSource.onsourceopen(_ => { > // TODO: Add source buffer, fetch video chunks, ... > }); > ``` I think that's supposed to be: ```js mediaSource.onsourceopen = _ => { // TODO: Add source buffer, fetch video chunks, ... }); ```

beaufortfrancois commented 5 years ago

Thank you @MattiasBuelens!

jonas-db commented 4 years ago

@beaufortfrancois @wolenetz MediaSource is still unavailable in a webworker context (chrome 83). Any updates?

wolenetz commented 4 years ago

@https://github.com/w3c/media-source/issues/175#issuecomment-630383367, thank you for your interest. After initial internal design reviews for the (new) Chromium prototype, I'm continuing work and hope to be able to announce an experimental Chrome Origin Trial for MSE-in-Workers in the short term.

Today, I've updated the original WICG Explainer to reflect the simplified approach being taken in the prototype.

tjenkinson commented 4 years ago

Looks good. How do you access the current time in the worker? Post messaging time updates from the media element?

wolenetz commented 4 years ago

@https://github.com/w3c/media-source/issues/175#issuecomment-640099375 - yes, to maintain the simplicity and interoperability of browser implementations of this feature, app will need to use normal communication such as postMessage to internally communicate the state of MSE to their main thread components and the state of the media element to their worker thread components. Internally, some of the MSE algorithms require some of this on some implementations, but they will handle that as necessary in those implementations without requiring all implementations expose some new field on the object model. For instance, in Chromium, MSE's coded frame eviction algorithm relies upon internal state, including currentTime, to more conservatively keep existing buffered media around currentTime and the last appended media. However, the worker MSE will not expose that notion of currentTime to the worker app; the worker app will need to get that info from the media element on the main thread and postMessage it.

wolenetz commented 3 years ago

Feature discovery of whether or not an implementation supports constructing a MediaSource in a dedicated worker might need exposure of an attribute on the MediaSource interface on at least the main Window context, so API users can feature detect in advance of spinning up a dedicated worker. Credit to @joeyparrish for pointing this out to me.

wolenetz commented 3 years ago

@https://github.com/w3c/media-source/issues/175#issuecomment-695038710, longer term, if we end up also adding srcObject or other approaches for attaching cross-context, we could do feature detection on the object type itself perhaps. For example, can main context construct a MediaSourceHandle? If so, MSE-in-workers should be supported. Or alternatively, does main thread's MediaSource allow transferability? If so, then MSE-in-workers should be supported. The routes we can take in future around srcObject/etc are flexible and not settled at all yet.

In short term, experimental Chromium prototype that is nearing readiness right now for experimentation, we use the objectURL approach described in the updated WICG explainer, which by itself, doesn't let the main thread know whether or not a dedicated worker supports creation of a MediaSource, hence the feature discoverability might need an attribute or some other way so API users don't have to first start a dedicated worker to determine if MediaSource can be constructed there.

wolenetz commented 3 years ago

@https://github.com/w3c/media-source/issues/175#issuecomment-695040660 - as of Chromium 88.0.4296.0, experimental MSE-in-Workers is available and functioning behind experimental flag (--enable-experimental-web-platform-features). Today, I'm landing a fix for a rare Chromium crash made less rare by this feature (https://chromium-review.googlesource.com/c/chromium/src/+/2486074).

I'm also working on a way of exposing to the main thread ability for apps to proactively detect if MSE-in-Workers is supported. How does a (static, exposed wherever MediaSource is exposed) boolean-valued, always true if it exists, attribute named: "canConstructInDedicatedWorker" on the MediaSource interface sound to folks? Alternatively, are there examples in the web-platform where collections of values are exposed to enable feature-detection of a variety of features?

joeyparrish commented 3 years ago

canConstructInDedicatedWorker sounds good to me, but I'm also not terribly picky about the exact name of it.

I'm not sure what the best practice is for feature detection broadly in web API design.

In other APIs, we have done feature detection by checking for the existence of a class or the existence of a method on a prototype. In some rare cases, we use the number of arguments to a constructor to determine which version of an API was implemented (so many TextTrackCue variations on old devices).

wolenetz commented 3 years ago

Ok, I'll proceed with canConstructInDedicatedWorker in the experimental implementation in Chromium. I'll update here once that ability has landed (on top of the already-landed experimental feature in 88.0.4296.0 already).

w3c / media-source

MediaSource in Worker #175

14 also seems relevant here -- based on my reading of that bug it would help with part of this problem (once you fetch a segment you'd pass a stream to that audio to MediaSource and would bypass the main thread, or even a worker thread in terms of getting that data to where it needs to go).