search

w3c / webrtc-nv-use-cases

Use cases for WebRTC NV
https://w3c.github.io/webrtc-nv-use-cases/
Other
32 stars 13 forks source link

Allocate RAW data capture buffer as shared GPU memory buffer #17

Closed astojilj closed 5 years ago

astojilj commented 6 years ago

Let's consider current data flow in processing video stream on GPU: getUserMedia() => WebGLRenderingContext.texImage2D(HTMLVideoElement) => processing in WebGL shaders.

An example for this is in TensorFlow.js - after video frame is uploaded to texture, processing is done on GPU under fenceSync - once the sync is signaled as ready, data can be read back to CPU without blocking.

While there is a security concern related to mapping buffers that GPU writes to, we can save a copy in WebGLRenderingContext.texImage2D(HTMLVideoElement) - if raw data buffer is shared GPU memory buffer.

I don't think that the engine should do this by default - it could be implemented by hinting to capture pipeline that the raw data will be used on GPU.

aboba commented 5 years ago

@astojilj Ability to use raw media for machine learning in an efficient manner seems important. Can we add something general to that effect as a requirement, or do we need to be specific?

aboba commented 5 years ago

@astojilj Can you prepare a PR for discussion at the June Virtual Interim?

aboba commented 5 years ago

@astojilj Can you review the changes to see if they are to your liking?

aboba commented 5 years ago

PR merged; if inadequate, please reopen.