search

w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.
https://w3c.github.io/webcodecs/
Other
988 stars 136 forks source link

`desiredWidth`/`desiredHeight` as best effort #202

Closed padenot closed 3 years ago

padenot commented 3 years ago

Why are those marked as "best effort"? It's pretty standard to do resize-during-decode in browsers, and this allows saving quite a lot of memory and CPU for authors.

I'm told by the folks working on images in Gecko that it always supports this regardless of the image format, and that the benefits are significant.

dalecurtis commented 3 years ago

How does Gecko implement that? Not all formats allow inherent reduced resolution decoding. Certainly we can resize after the decode to full resolution (or nearest to desired size), but the motto for WebCodecs thus far has been to avoid implicit expensive operations.

We'd instead favor a post-decode type resize occur through some explicit operation on the VideoFrame. In this case the expectation would be that the author handles the resize during the drawImage/texImage/createImageBitmap operation.

padenot commented 3 years ago

How does Gecko implement that?

No idea, I'll get this information and report back.

the motto for WebCodecs thus far has been to avoid implicit expensive operations

I agree that Web Codecs needs to be low level and explicit, but specifying a target size directly in the decode steps allows for better performance in all cases (if only because of memory locality, at the very minimum).

Not all web codec uses are going to be about displaying images (but I agree that what you list are very good solutions for when this is the case), there is a need for a resize primitive. It's certainly possible via a Canvas today, but it's cumbersome and potentially wasteful. In addition, it requires a higher-level of abstraction object for doing an operation on a lower-level construct, which is indicative of a functionality gap.

dalecurtis commented 3 years ago

I agree that Web Codecs needs to be low level and explicit, but specifying a target size directly in the decode steps allows for better performance in all cases (if only because of memory locality, at the very minimum).

In the case where we have to decode to full resolution and then apply a scaling algorithm into a new block of memory, I don't follow how we could have any meaningfully better performance. E.g., it seems it would always be better if scaling was handled at the draw stage into a pooled and/or gpu buffer or by the encoder into potentially cross-process shared memory or by the hardware encoder itself.

Not all web codec uses are going to be about displaying images (but I agree that what you list are very good solutions for when this is the case), there is a need for a resize primitive. It's certainly possible via a Canvas today, but it's cumbersome and potentially wasteful. In addition, it requires a higher-level of abstraction object for doing an operation on a lower-level construct, which is indicative of a functionality gap.

I totally agree we eventually need a scaling and conversion API to handle a small set of non-display, non-encode use cases, but I don't think we need to bake that into the decoders themselves for cases where the format can't natively decode into a reduced resolution. I also feel that these use cases will be okay in the short term with going through canvas or ImageBitmap for scaling.

Another reason these attributes are best effort is that reduced resolution decoding doesn't guarantee an exact size. So even in cases where a codec does support a lower resolution decode, the user is effectively ending up with a nearest match to their requested size.

padenot commented 3 years ago

Another reason these attributes are best effort is that reduced resolution decoding doesn't guarantee an exact size. So even in cases where a codec does support a lower resolution decode, the user is effectively ending up with a nearest match to their requested size.

I buy this argument. I'm closing this issue, thanks for the clarifications!