search

w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.
https://w3c.github.io/webcodecs/
Other
978 stars 136 forks source link

Converting VideoFrame to canvas images or WebGL textures is very slow #421

Closed xfq closed 2 years ago

xfq commented 2 years ago

When using VideoFrame as the textures of Three.js, we found that it's very slow. When decoding a 3840x3840 video only, the frame rate is about 80fps; adding frame.clone() it is still about 80fps. But after adding drawImage or texSubImage2D, the frame rate dropped to about 10fps:

145173495-6d56f34f-8ff8-4894-886a-92cd9d90dfca

Can anyone point me in the right direction?

shaoboyan commented 2 years ago

(If it happens in chrome) I suspect that the VideoFrame you got from decoder is backed by CPU resources(Which means, the GPU decoder not works, because of profiles). One simple way to check this with chrome is that you could use HTMLVideoElement to play your video and use chrome://media-internals to see the decoder it uses.

kareljuricka commented 2 years ago

I think it can be related to my issue - https://bugs.chromium.org/p/chromium/issues/detail?id=1272288 , which is fixed already and will be released in newer Chrome versions. Try Chrome Canary to compare performance. They already applied fix there.

xfq commented 2 years ago

(If it happens in chrome) I suspect that the VideoFrame you got from decoder is backed by CPU resources(Which means, the GPU decoder not works, because of profiles). One simple way to check this with chrome is that you could use HTMLVideoElement to play your video and use chrome://media-internals to see the decoder it uses.

kVideoDecoderName is D3D11VideoDecoder.

xfq commented 2 years ago

I think it can be related to my issue - https://bugs.chromium.org/p/chromium/issues/detail?id=1272288 , which is fixed already and will be released in newer Chrome versions. Try Chrome Canary to compare performance. They already applied fix there.

Thanks. drawImage is faster in Chrome Canary, but texSubImage2D is the same as before (the frame rate is still very low).

shaoboyan commented 2 years ago

@xfq Would you mind to provide the use case and I can take a look at it. And you could file a chromium bug.

xfq commented 2 years ago

@shaoboyan The use case is volumetric video. See https://github.com/w3c/strategy/issues/286

Since it is possible to convert from Canvas 2D to CanvasTexture in three.js, and this is really an impl issue, not a spec issue, I'll close it.

If you think we need a Chromium bug to track the texSubImage2D performance issue, I can file one. Thank you for your help!

shaoboyan commented 2 years ago

@xfq Yes, pls help to file a bug in chromium to track texSubImage2D, thanks!

chcunningham commented 2 years ago

@xfq - just checking in on the chrome bug. Was it fixed as suspected by https://github.com/w3c/webcodecs/issues/421#issuecomment-990716303?

xfq commented 2 years ago

@chcunningham drawImage was fixed, but texSubImage2D is still slow. I have filed https://bugs.chromium.org/p/chromium/issues/detail?id=1279351