MediaSource in Worker

[DanielBaulig]

The Videos team at Facebook has a growing need to move calls to Media Source Extensions off of the main thread and into a Worker. While most parts of your off-the-shelve streaming video player can be implemented in a Worker (fetching and parsing of the manifest as well as selecting and fetching of segments) the calls to and interactions with the Media Source Extensions API (e.g. SourceBuffer.prototype.appendBuffer, SourceBuffer.prototype.remove, etc) remain a bottleneck and have to happen on the main browser rendering thread.

On a sizable and complex web site large blocks of JS execution can regularly block the main thread making videos slow to start playing and causing lots of rebuffering, especially in live videos where you can only buffer ahead very little data. To ensure great web video performance in the long run, even as web apps continue to grow in size and complexity, we believe we need to allow video players to move into Workers and off of the main thread.

We would like to propose this change to the working group and involve other API users as well as implementors in the discussion.

[wolenetz]

As of Chromium 88.0.4300.0, experimental MSE-in-Workers is available and functioning behind experimental flag (--enable-experimental-web-platform-features), and includes support for proactive feature detection from the main thread via MediaSource.canConstructInDedicatedWorker as described in https://github.com/w3c/media-source/issues/175#issuecomment-712516853.

I'm putting together more details to share soon regarding how to use the feature, areas of the feature which could use further definition, and a very simple toy demo app.

[wolenetz]

Demo:

• Avoiding main thread contention and associated "buffering jank"
• Github repo for the demo

Please try it out (in Chromium 88.0.4300.0 or greater, with chrome://flags/#experimental-web-platform-features enabled.)

Some specific questions we wanted to get answers to (for clarity when specifying this feature and for improved interop):

• If the DedicatedWorker MediaSource's context is terminated (e.g. worker.terminate()), then the main thread element still remains connected, but seekable and buffered range information reset. There is no other immediate error exposed to the main-thread media element. Would you prefer other behavior in this scenario?
• If the main thread media element has begun detachment of the DedicatedWorker MediaSource, that detachment is communicated to the worker MediaSource where the transition to closed will be executed. During that transition period (when the detachment message is pending handling by the worker context), usage of the SourceBuffer and MediaSource objects which either require the MediaSource to be in the "open" or "ended" readyState or for the SourceBuffer to not be removed from its parent MediaSource will synchronously fail with DOMException (Invalid State Error) even if they otherwise satisfy those requirements. The reasoning for this is that the MediaSource object is known to be closing soon by the implementation, and the implementation is already destructing the underlying demuxer and player, so proceeding without giving error would be worse in this case. Would you prefer a distinct error in this scenario, to assist identifying this edge case?
• Precisely like the previous caveat, in case the main Window execution context destruction is ordered to happen before the DedicatedWorker (for example on tab closure, this can happen in Chromium/Blink since the ordering of context destruction is nondeterministic), then there will be short period of time when the DedicatedWorker MediaSource could throw DOMException (InvalidStateError). Would you prefer a distinct error in this scenario?

Responses would be helpful: either directly on this spec issue, or if there is an implementation issue, on the Chromium bug for this feature (https://crbug.com/878133).

• edited to add the questions.

[johnBartos]

Hey Matt, great work! I'm wondering how to best handle this new error: DOMException: Failed to read the 'buffered' property from 'SourceBuffer': Worker MediaSource attachment is closing. I noticed that the readyState is still "open" in this case. Is there a recommended way to handle this new error?

To add more context, I see this after getting a video error. I anticipate handling those will be tricky because passing the error from the main thread to the worker will be async.

[wolenetz]

@https://github.com/w3c/media-source/issues/175#issuecomment-742132101 -

Hi John. Thank you for trying out this feature. This new error condition that you're encountering is one of the three scenarios on which we have requested feedback; specifically, it is the second bulleted question in https://github.com/w3c/media-source/issues/175#issuecomment-721395481.

When handling this particular "Worker MediaSource attachment is closing" InvalidStateException scenario when readyState appears valid for the operation, the worker portion of the app that is exercising the MSE API should behave as if either:

• the main thread portion of the app has explicitly detached the MediaSource, initiating MSE closure that will happen soon in the worker,
• the main thread media element has experience a fatal error, initiating MSE closure that will happen soon in the worker, or
• the main thread media element's context is destructed, initiating MSE closure that will happen soon in the worker.

If it would help to have a distinct error (not an InvalidStateError DOMException) to help identify this scenario to the worker MSE API user, what kind of error would you prefer? Alternatively, or in addition to a distinct error, would a new readyState, such as "closing" be appropriate? I did not include such in the experimental implementation because such state would not be guaranteed consistent across multiple synchronous reads of the attribute by the worker MSE API user (the transition to "closing" is something that could happen at any time - except within carefully protected mutex regions within the respective MSE call handling in the worker).

I look forward to your responses.

[johnBartos]

@wolenetz Thanks for the detailed response.

the main thread portion of the app has explicitly detached the MediaSource, initiating MSE closure that will happen soon in the worker,

Yeah this is what happened, I detached the MediaSource on error, but the worker thread continued to act upon the MediaSource. At the time of video error I send a message to stop the worker MediaSource instance, but since that's async it's not guaranteed to work.

Alternatively, or in addition to a distinct error, would a new readyState, such as "closing" be appropriate?

This would be my preferred choice because I think the implementation would be the simplest. Before applying any SourceBuffer operation, I could just check this flag. But if it's not consistent across reads then I'd still want to wrap operations in a try/catch, and at that point there's not much value in the flag.

If it would help to have a distinct error (not an InvalidStateError DOMException) to help identify this scenario to the worker MSE API user, what kind of error would you prefer?

Right now I think the error is fine, but I haven't yet gotten around to detailed error handling. I don't think it'd help in this case.

I believe the correct way to do to this is to have the worker thread tell the main thread when to detach the source. This way the worker can clean up any pending operations before the detach. With this approach, is there a risk of getting errors from the MediaSource if an operation is applied to a media element in the errored state? For example, say I call enqueue after a decode error has been triggered.

[johnBartos]

The reasoning for this is that the MediaSource object is known to be closing soon by the implementation, and the implementation is already destructing the underlying demuxer and player, so proceeding without giving error would be worse in this case.

This won't always be true. Third party developers using a player SDK may bypass the proper APIs and remove the video source directly. Player vendors will therefore need to gracefully handle cases when the MediaSource is detached without advanced knowledge.

Even with a try/catch around the new exception, I'm noticing that the page still crashes after the error is handled. Is this expected in Canary?

[wolenetz]

Alternatively, or in addition to a distinct error, would a new readyState, such as "closing" be appropriate?

This would be my preferred choice because I think the implementation would be the simplest. Before applying any SourceBuffer operation, I could just check this flag. But if it's not consistent across reads then I'd still want to wrap operations in a try/catch, and at that point there's not much value in the flag.

It would not be consistent across reads (otherwise, the main thread would need to block itself until the attached worker thread reaches a stable state, before the main thread could continue and destruct the media player, etc.), so would we need to have a new 'closing' readyState then if try/catch is necessary already? In worst case, we might need to actually do that main thread blocking/delayed shutdown of media player if the approach to having a "closing at any moment" model in the experimental implementation is too onerous.

If it would help to have a distinct error (not an InvalidStateError DOMException) to help identify this scenario to the worker MSE API user, what kind of error would you prefer?

Right now I think the error is fine, but I haven't yet gotten around to detailed error handling. I don't think it'd help in this case.

I believe the correct way to do to this is to have the worker thread tell the main thread when to detach the source. This way the worker can clean up any pending operations before the detach. With this approach, is there a risk of getting errors from the MediaSource if an operation is applied to a media element in the errored state? For example, say I call enqueue after a decode error has been triggered.

Certainly, if the app could first prepare its worker thread for imminent main-thread detachment, and control the actual detachment sequence in the common case, this would help the app understand the attachment state better. However, there are also non-rare scenarios (such as decode error in the pipeline, or user closing the tab) whereby the main thread might begin detachment without being told to do that explicitly by the app.

I personally think a combination of a 'closing' readyState (to help apps confirm the reason for exception) and InvalidStateError Exception on operations on the worker MediaSource taken by the app when there is a pending detachment of the MediaSource, would be a good way forwards.

The reasoning for this is that the MediaSource object is known to be closing soon by the implementation, and the implementation is already destructing the underlying demuxer and player, so proceeding without giving error would be worse in this case.

This won't always be true. Third party developers using a player SDK may bypass the proper APIs and remove the video source directly. Player vendors will therefore need to gracefully handle cases when the MediaSource is detached without advanced knowledge.

Would the proposed 'closing' readyState and InvalidStateError Exception satisfy this concern?

Even with a try/catch around the new exception, I'm noticing that the page still crashes after the error is handled. Is this expected in Canary?

Crashing is not expected in Canary -- unless there is an attempt to access the audio/video/textTracks() of a worker-owned SourceBuffer (media tracks in MSE/HTMLMediaElement is also experimental, and incomplete in the MSE-in-Workers experimental implementation currently). Do you have any indications of what specifically is causing the crash? If you share a crash report ID with me, from your repro's chrome://crashes (direct email wolenetz@chromium.org), then I can investigate further.

[johnBartos]

Do you have any indications of what specifically is causing the crash?

Yeah, if you detach the MediaSource before closing it the tab will crash. Unfortunately it doesn't seem that I can enable non-local crash reporting in Chrome? It only gives me the local ID in chrome://crashes, and the toggle to enable crash reporting is unclickable for me.

My repro steps are simple though - start a MSE in workers instance and then call $('video.src') = ''

I personally think a combination of a 'closing' readyState (to help apps confirm the reason for exception) and InvalidStateError Exception on operations on the worker MediaSource taken by the app when there is a pending detachment of the MediaSource, would be a good way forwards.

I think this would be good. Restructuring our app to ensure MediaSource deletion wasn't onerous in my opinion, and having the closing state would help us handle the error correctly. I believe it addresses our concern as long as catching the error does not crash the tab 😁

To be clear, would this closing state would be reliable only after the exception is thrown? Meaning that developers should not use it to prevent errors, and that the recommendation for app devs is to ensure that interactions with the MediaSource are properly wrapped in a try/catch.

[wolenetz]

My repro steps are simple though - start a MSE in workers instance and then call $('video.src') = ''

@johnBartos - I could not get a crash to repro in my trunk build locally (using $(video).src='', to get it to parse...). Could you perhaps share a minimized small repro page+worker js with me (privately would be fine)?

[johnBartos]

@wolenetz Sorry for the delay, will do soon.

Another question - when MSEIW is released, will it be supported on every device which is on the supported Chrome version? I want to track its usage when we roll it out and I'm wondering if I can just go by the Chrome version.

[wolenetz]

@johnBartos - I think we discussed this on slack, but for all here, MSEIW will likely first go through origin trials in Chrome; potential API users will be able to sign up for experimental support for a period of releases to assist stabilization and eventual shipping. Unless plans change, it's likely the experimental feature will be in the origin trial across the Chrome products involved in the releases. Throughout, the proactive ability for apps to be able to detect if the feature exists and is available is expected to be available at runtime via MediaSource.canConstructInDedicatedWorker as described, above.

@johnBartos - regarding the crash you had reported, and I had been unable to repro, is it still occurring? Do you have more details on how I could possibly repro it?

[johnBartos]

@wolenetz I'll get you simple steps (created a ticket so I won't forget). It might be an issue with my machine since I'm getting other dev tools crashes.

[johnBartos]

Ok I have steps. I used your demo but modified it to append in 1-byte increments to give me plenty of time to repro the crash:

• Start the demo with append 1-byte increments
• In the console, set the video src of the worker element to ''
• Observe the following error:

Environment:

• Chrome 89.0.4389.90 (Official Build) (x86_64)

I think the key is to detatch the src and then have the MediaSource do an append. I can upload a fork with my modification if you'd like.

[wolenetz]

@johnBartos I'm trying to reproduce on trunk Chrome with a new radio button for 1-byte appends in a local version. So far, I haven't been able to get the tab to crash. Have you tried on recent Canary (maybe something unrelated, say in devtools, is causing the crash)?

[ Update: I have a repro - tracking this investigation and eventual fix at https://crbug.com/1195441 ] [ Update: Problem is understood. See https://bugs.chromium.org/p/chromium/issues/detail?id=1195441#c6 if curious. ] [ Update April 5: The fix has landed in Chromium and should be in a Chrome Canary soon. Follow https://chromiumdash.appspot.com/commit/734bae172772095a8a94337d994724264dacf405 to see which version will have it first. ]

[johnBartos]

Awesome, nice find Matt! I'll test out the fix once it lands.

Edit: Tested it out on Canary 91 and I can't repro in either the sample I sent or our player 👍

[wolenetz]

@johnBartos and any interested: I found and am fixing another issue in the experimental Chromium implementation: reading mediaSource.duration attribute from a worker context wasn't taking the right internal locks and was unsafe. https://chromium-review.googlesource.com/c/chromium/src/+/2909389 has more detail.

Also, I'm preparing a draft feature spec as part of getting MSE-in-Workers prepared for launching Chrome origin trials. Expect news on this soon.

[johnBartos]

Sounds great @wolenetz, thanks for the update! Looking forward to the origin trial

[karlt]

Having entangled objects across threads sets up an inter-thread cycle, which is a challenge for GC. I have trouble recalling all the possible scenarios, but I guess there are at least some where operations on either object may resume playback. e.g. if playback has stalled through lack of data, then it may resume either if more data is provided through a SourceBuffer or if a seek is performed on the HTMLMediaElement to a position which already has data. Neither thread can independently GC its object because the entangled object may want to use it again.

I assume this is a theoretically solvable problem, but complex enough that no browser solves it. MessagePort entanglement is similar and the workaround is an explicit close() method to disentangle the objects. Clearing the source resource on the HTMLMediaElement would provide a similar explicit disentengle function from the main thread. IIUC MediaSource#endOfStream() is not a permanent termination and so doesn't provide a worker-thread counterpart.

Has consideration been given to mitigating inter-thread GC difficulties?

Maybe MediaSource and SourceBuffer could be garbage collected if their events depend only actions on that thread, but the "sourceclose" event depends on HTMLMediaElement state, which may change in response to actions on its thread.

[wolenetz]

@https://github.com/w3c/media-source/issues/175#issuecomment-896355007 you raise good points that I have encountered when designing and implementing the current experimental prototype of MSE-in-Workers in Chromium.

For same-thread attachments, GC is prevented by any of:

• an active reference to the attached media element, MediaSource, or any of its SourceBuffers (e.g. from script or transitively from other objects back to a GC roots),
• the attached media element having some pending activity (e.g. seeking or an event is pending dispatch),
• or the attached MediaSource or its SourceBuffers having some pending activity (e.g. updating or an event is pending dispatch)
• or any of the objects that may not be traceable from a GC root but have pending activity themselves and have some reference, perhaps transitively, to the attached media element, MediaSource, or its SourceBuffers.

For cross-thread attachments (e.g., for MSE-in-Workers), Blink doesn't currently have (nor does it appear to be planned from what I recall) a notion of a "CrossThreadMember" that would automate the rules, above, similarly. So, yes, cross-thread attachments need explicit teardown by detachment from the media element to break the strong reference held by the internal CrossThreadMediaSourceAttachment GC-able object, which holds the references to the attached media element and MediaSource in CrossThreadPersistent members (which are treated themselves as GC roots on the thread owning the referenced object).

Some way of finding inactive cross-thread attachment cycles and automatically detaching them to enable their GC is not included in either the current draft specification for the feature, nor in the experimental Chromium implementation.

MediaSource.endOfStream(x), when x is either 'network' or 'decode' leads to closing the MediaSource itself but does not necessarily detach it from the media element. Perhaps a new method on MediaSource that causes detachment might help MSE-in-Workers apps clean up idle state?

-edited to correct type to be CrossThreadPersistent, above. -edited to suggest maybe a new MediaSource method to help worker-side initiate detachment

[wolenetz]

The experimental implementation in Chromium recently has taken a change to mitigate potential high-resolution timer security issues [1], so the communication of internal buffered, seekable MSE state and internal element error and recent media element time state across contexts is no longer optimized but instead has latency similar to the unoptimized postMessage() option described in the spec draft.

While this change hopefully shouldn't impact function or performance of the API significantly, please be aware that it might affect your use case.

Also, I'm working on enabling the Chromium origin trial for this feature right now and expect to be able to share more details very soon.

[1] https://chromium-review.googlesource.com/c/chromium/src/+/3095089

• edited to add link to [1]

[wolenetz]

Thanks to @johnBartos for discovering the intentional crash in the Chromium experimental implementation when the experimental AudioVideoTracks feature usage is attempted on a worker SourceBuffer. I've updated the demo's instructions for how to more narrowly enable just the MediaSourceInWorkers experimental feature until both the spec issue #280 is resolved and the implementation is updated.

[wolenetz]

The Chromium experimental implementation of MSE-in-Workers is currently in origin trials (as of M95).

[sterpe]

@wolenetz What is timeline (or current thoughts) for when MSE-in-Workers would be turned on in Chrome by default?

We (Facebook) are interested in starting some active development around this in 2022.

[wolenetz]

@https://github.com/w3c/media-source/issues/175#issuecomment-932483258 The current plan is to have the origin trial run from M95 through M99, and if data and feedback show that it is ready to launch, then it could be launched around the M99-M100 milestone (Q1 2022). I highly encourage you to engage in the origin trial as early as possible to help ensure both an ergonomic API shape and a stable implementation.

[wolenetz]

Note that using srcObject for MSE-in-Workers attachments is a change to the feature that the media wg arrived at last September [1] (following the previous objectURL-based attachment design getting more recent feedback [2]), and I'm working on getting that into both the feature spec and Chromium's experimental implementation currently.

[1] https://www.w3.org/2021/09/14-mediawg-minutes.html [2] https://github.com/mozilla/standards-positions/issues/547

[wolenetz]

With respect to switching to using srcObject-based attachment for MSE-in-Workers, I'm putting together a spec draft now. I've discussed approaches previously during the initial explainer [a], on the mozilla request for position [b], and directly with one of the current Chrome origin trial participants.

There's likely some flexibility in how that specification and implementation lands fully:

1. Should the transferable object be only constructible on a worker context? Or can it be constructed on the main context, attached before even a worker is spun up, and eventually transferred to a worker for use there in resolving the underlying MediaSource? (I prefer the former for this initial approach and for simplicity. The latter approach involves questions like, what should the state of the media element be if the worker doesn't yet have the underlying MediaSource. It is not precluded from being explored in the future though.)

2. Must we also (as part of this origin trial) enable main-thread MSE attachment using srcObject? (I prefer this be out-of-scope of the MSE-in-Worker OT, for simplicity and priority of getting MSE-in-Worker impl(s) updated and shipped.) Likewise, it is not precluded from being explored in the future.

I have received some private support for preferred options in 1 & 2 from a current Chrome origin trial participant, those preferences being: 1: worker constructs the handle, transfers it to main, main uses the handle to attach the worker MediaSource instance via setting the element's srcObject to be the handle object, and 2: main-thread srcObject attachment of some handle object or even just the MediaSource for MSE isn't required (for at least usage in MSE-in-Workers work). Existing objectURL attachment is working already.

@jernoble (and also anyone interested in this feature), would this approach to enabling srcObject usage for MSE-in-Workers attachment, as a replacement for using objectURLs for those attachments, have your support? If so, I'll get a spec draft and implementation update readied ASAP.

[a] (original explainer, not the current proposal): https://github.com/wicg/media-source/blob/mse-in-workers-using-handle/mse-in-workers-using-handle-explainer.md [b] https://github.com/mozilla/standards-positions/issues/547

-edited to add links a & b

[karlt]

• Should the transferable object be only constructible on a worker context? Or can it be constructed on the main context, attached before even a worker is spun up, and eventually transferred to a worker for use there in resolving the underlying MediaSource? (I prefer the former for this initial approach and for simplicity. The latter approach involves questions like, what should the state of the media element be if the worker doesn't yet have the underlying MediaSource. It is not precluded from being explored in the future though.)

The two options presented here are quite different approaches, requiring different APIs, so I'd be inclined to choose one or the other. The reasons you've given for the former approach sound good to me.

Perhaps, for consistency, the object could be constructed in Window context (as well as in Worker context) from a MediaSource in the same Window context for attachment to video element in the same or another Window context. But I don't feel this should block progress for Worker. It would not be precluded in the future.

• Must we also (as part of this origin trial) enable main-thread MSE attachment using srcObject? (I prefer this be out-of-scope of the MSE-in-Worker OT, for simplicity and priority of getting MSE-in-Worker impl(s) updated and shipped.) Likewise, it is not precluded from being explored in the future.

Similarly direct attachment of Window-context MediaSource to srcObject seems orthogonal to transferable object attachment, and need not block progress for Worker.

[johnBartos]

I'll chime in as an implementer whose currently running this feature in Chrome's origin trial. Before diving into specifics I just want to say that on Twitch we see that this feature has a very large positive impact on our topline metrics such as buffering and time to video. We'd love for all browsers to adopt MSE in workers 🙂

Should the transferable object be only constructible on a worker context? Or can it be constructed on the main context, attached before even a worker is spun up, and eventually transferred to a worker for use there in resolving the underlying MediaSource?

I also prefer the former. Maybe this is specific to the Twitch player, but a lot of logic lives in the worker and we create it very early in the player lifecycle. So even if the object was constructible in the main thread and passed to a worker once it instantiated (presumably saving some time), we likely would still prefer to construct in a worker and send to the main thread.

Must we also (as part of this origin trial) enable main-thread MSE attachment using srcObject?

I don't think this is really a sticking point for implementers because both options seem functionally equivalent from our point of view.

I'm happy to answer implementation questions to the best of my ability if anyone has them.

[wolenetz]

@jernoble - friendly ping to hopefully get your input on https://github.com/w3c/media-source/issues/175#issuecomment-1042395368

[wolenetz]

Chrome's MSE-in-Workers origin trial end milestone [1] has now been extended to M103 to accommodate time for the srcObject attachment mechanism change to this feature. I'll update here when the change to add srcObject for attachment has landed in Chromium, which will then shortly be followed by the removal of the objectURL-based attachment mechanism (ideally in distinct milestones to assist trial participants' implementations picking amongst the mechanisms).

[1] https://chromestatus.com/feature/5177263249162240

[wolenetz]

During Media WG meeting today [1], @jernoble affirmed the preferred approach in https://github.com/w3c/media-source/issues/175#issuecomment-1042395368 towards which I have optimistically been drafting the spec update. I'll continue my work on that.

[1] https://www.w3.org/2022/03/08-mediawg-minutes.html

[wolenetz]

Spec draft for the proposed srcObject way of attaching MSE-in-Worker (and disallowing MSE-in-Worker attachment via object URL) is uploaded and PR is pending review by @mwatson2 currently. See #305

[wolenetz]

Spec is updated. Chromium implementation and WPT have landed for it (as of 105.0.5180.0, behind experimental flags). Today, I:

• updated the demo to require handle+srcObject (instead of legacy object URL) for worker MediaSource attachment, and
• updated the original WICG explainer similarly to match the current specification and Chromium experimental implementation.

[wolenetz]

I have also sent for review by blink-dev (part of Chrome review launch process) intents to:

• ship MSE-in-Workers (using MediaSourceHandle as specified currently) in Chrome 105 (pending their approvals currently): https://groups.google.com/a/chromium.org/g/blink-dev/c/FRY3F1v6Two
• extend the origin-trial in Chrome through 104 (using the legacy worker MediaSource object URL attachment mechanism, but otherwise unchanged internally once attached versus the current MediaSourceHandle API shape): https://groups.google.com/a/chromium.org/g/blink-dev/c/WETrrUYLrTM

I'll update here when one or both of those intents are approved.

[wolenetz]

I'll update here when one or both of those intents are approved.

Both of the blink intents have been approved.

• MSE-in-Workers (using the spec's MediaSourceHandle and srcObject for attachment) is enabled in Chrome 106 by default (beginning with 106.0.5214.0) and I expect for it to be available soon in Chrome 105 once further approvals are received.

• MSE-in-Workers origin trial (using the earlier experimental API for this, where attachment of worker MediaSource is achievable by getting an object URL for it over to the main thread and assigning it to the media element's src attribute) is now extended to Chrome 104, ending August 30, 2022 (https://developer.chrome.com/origintrials/#/view_trial/3847199981681246209).

Note that the latter (origin trial) version of the API is NOT how the shipping version (and spec'ed version) works, so origin trial participants will need to expect to update their API usage accordingly based on which version of Chrome (OT through 104, shipping version beginning soon in 105) is being used. The presence of a 'handle' getter on the MediaSource prototype indicates the expected API that is shipping soon in 105. Likewise, the presence of a 'MediaSourceHandle' on the context indicates the same. Regardless, if the static method 'MediaSource.canConstructInDedicatedWorker' is missing or exists but returns false, then neither version of the API is available.

Some further points about this API transition from origin trial to the shipping version (again, expected to become enabled in Chrome 105 soon - I'll update here again when that happens):

• Though you can still obtain an "object URL" for a dedicated worker MediaSource, it'll fail with media element error if you try to use it to attach to the media element.
• Instead, obtain a "MediaSourceHandle" from the dedicated worker MediaSource (using a new readonly [SameObject] getter: "handle"), and transfer it (not just serialize it) to your main thread. Then assign it to the media element's srcObject attribute. Example of change in worker API usage:
  • const ms = new MediaSource(); const url = URL.createObjectURL(ms); postMessage(url); // OLD API shape (through 104)
  • const ms = new MediaSource(); const handle = ms.handle; postMessage(handle, [handle]); // New API shape (shipping in 105 soon)
• Important points about the new API shape.
  • A MediaSourceHandle instance is usable at most once for successfully attaching to a media element. A Worker MediaSource instance has precisely one MediaSourceHandle instance. Therefore, for multiple MediaSource attachments, you'll need to obtain new worker MediaSource instances and transfer their handles, for each attachment.
  • No worker MediaSource object URL revocation is necessary, since worker MediaSource now attaches via a MediaSourceHandle object and the implementation knows how to GC these objects when there remain no references to them and their attachment has been closed (for example, by assigning "null" to the media element's srcObject attribute.)
• There are other fine points. I recommend reviewing the current MSE spec working draft for edge cases not outlined above: https://www.w3.org/TR/media-source-2/

There is also a demo of this feature (using the spec'ed MediaSourceHandle+srcObject version shipping soon in Chrome 105):

• https://tinyurl.com/mse-in-workers-demo (which is short for: https://wolenetz.github.io/mse-in-workers-demo/mse-in-workers-demo.html)

[edited to add demo links]

[wolenetz]

...expected to become enabled in Chrome 105 soon - I'll update here again when that happens):

• If using Chrome 106.0.5214.0 or greater, the feature is enabled by default.
• If using Chrome 105.0.5195.18 or greater in the 105 milestone, the feature is enabled by default.

[Aholicknight]

Images

 

I am getting the same error with the latest version of Microsoft Edge Beta v105.0.1343.17 is there any way to fix this? (Please check images in the summary)

Thank you!

[dalecurtis]

This is tracked by https://bugs.chromium.org/p/chromium/issues/detail?id=1358542 it looks like an old version of video.js is trying to mirror all properties on the MediaSource object, so throwing an error upon access to handle is exploding in the main thread.

I think we'll want to return nullptr instead of throwing an exception when Handle is accessed from the main thread.

[dalecurtis]

Actually a more elegant solution may be to just tag "handle" as [Exposed=DedicatedWorker]" since it just throws an exception on the main thread anyways. We already have canConstructInDedicatedWorker for feature detection.

[Aholicknight]

This is tracked by bugs.chromium.org/p/chromium/issues/detail?id=1358542 it looks like an old version of video.js is trying to mirror all properties on the MediaSource object, so throwing an error upon access to handle is exploding in the main thread.

I think we'll want to return nullptr instead of throwing an exception when Handle is accessed from the main thread.

@dalecurtis so this is the sites problem and not mine?

[dalecurtis]

Yes, but ultimately it's our problem too since it's a compatibility issue. We'll pull MSE-In-Workers from M105 until this is resolved, we can continue discussion on the Chromium bug. Let's leave this issue for discussion of the API.

[wolenetz]

I'm finally back from vacation and illness recovery after it. I'll be looking at the MSE-in-Workers issue ASAP as I catch up.

[wolenetz]

For those interested in contributing to the discussion around how to update the spec for the MediaSource.handle getter, please see #316.

Also, I've updated the MSE in Workers demo today:

1. It no longer checks for the "handle" getter on the Window/Main context MediaSource object, since "handle" will only be needed by the demo on the dedicated worker context (https://github.com/wolenetz/mse-in-workers-demo/commit/056a416649a35a10e8a563e3189cfe4ef5f06f28).
2. Updated the "known-working" Chrome versions listed in the demo and in its README (https://github.com/wolenetz/mse-in-workers-demo/commit/9685c9470abe089f35d90e97e6dd05415e2e0083).

[johnBartos]

LGTM from our app's perspective, as long as typeof MediaSourceHandle === function on the main thread. We'd like some way to check if MSEIW is supported before we set up our worker.

[wolenetz]

@https://github.com/w3c/media-source/issues/175#issuecomment-1242390314 thanks for your input. We're currently considering two options: Change from exception to null value for the 'handle' attribute when accessed on a main thread MediaSource, or change visibility of that handle getter to only be visible on the dedicated worker context (for now at least). I presume we'll disallow exception even in the latter case.

Either way, it looks like, on main thread (or on worker thread), MediaSource.canConstructInDedicatedWorker === true will remain the overall feature detection for MSE-in-Workers, and, when it's true, both dedicated worker and main thread contexts should have typeof MediaSourceHandle === 'function' on conforming implementations.

[wolenetz]

From discussion during Media Working Group meeting at TPAC 2022 today, both @jernoble and @dontcallmedom also supported the visibility restriction option for the handle getter. I presented both options (null with custom spec replacement for [SameObject], or just exposed=dedicatedworker -- neither can throw exception).

I'll update (or just abandon/replace) the PR #316 to instead do visibility restriction of that getter, without exception throwing.

[wolenetz]

In Chrome, the feature has been fixed to behave per #317 and builds of Chrome >= 108.0.5334.0 should include it enabled by default. See also the Chromium feature tracking issue 878133.