search

w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.
https://w3c.github.io/webcodecs/
Other
951 stars 135 forks source link

Threading #51

Closed sandersdan closed 3 years ago

sandersdan commented 4 years ago

Chromium's WebCodecs implementation will offload decoding to a separate thread. We probably want to specify this behavior.

An alternative would be to specify a Worklet type for codec implementations, but this doesn't seem compatible with having a codec that uses many threads.

padenot commented 4 years ago

Both seem useful. Big apps ported from native or advanced users probably want a synchronous API and use their own threading synchronized via shared memory to keep caches hotter when doing something with the output afterwards. Other users probably want something a bit more ergonomic.

Parallelism inside Web Codec constructs will be some sort of fork-and-join, but that's useful for intra-frame parallelism, for encoding video.

chrishtr commented 3 years ago

I'm not sure it makes sense to expose the WebCodecs API to the main thread at all. Shouldn't it always be in a worklet or worker? Is there a compelling use case that requires the main thread?

chcunningham commented 3 years ago

The spec's processing model now makes the threading behavior explicit https://wicg.github.io/web-codecs/#codec-processing-model

Control thread: where you make the encoder/decoder (be it in a window or worker) Codec thread: where actual encoding/decoding is to occur. May actually be N threads, but may not be the control thread.

I'm not sure it makes sense to expose the WebCodecs API to the main thread at all. Shouldn't it always be in a worklet or worker? Is there a compelling use case that requires the main thread?

This question probably goes away with my comment above. Apps with lots of main thread activity and/or heavy UI (e.g. video conferencing) should still use WebCodecs in a worker to avoid contention with controlling the codec and painting their UI. But for other apps, Window is fine.

aboba commented 3 years ago

Some questions were raised about WebCodecs support for threading here.

chcunningham commented 3 years ago

Chromium's WebCodecs implementation will offload decoding to a separate thread. We probably want to specify this behavior.

This is now spelled out in the spec. https://w3c.github.io/webcodecs/#control-thread-and-codec-thread

Big apps ported from native or advanced users probably want a synchronous API...

We discussed this in the call w/ audio epxerts. It is true that many are coming from a synchronous API, but they concluded that this was not a requirement for them. We also discussed how our API's under the hood are often async and they stated that they did not desire for us to hide that from them at the JS layer. Please open a new issue for this if I've overlooked anything.

I'm not sure it makes sense to expose the WebCodecs API to the main thread at all.

This same question is under more active discussion in #211.