search

mozilla / webxr-api

INACTIVE - http://mzl.la/ghe-archive - A proposal for WebXR, based on the WebVR extension
195 stars 23 forks source link

Supporting Computer Vision #10

Closed blairmacintyre closed 2 years ago

blairmacintyre commented 7 years ago

We will want to allow people to write their own CV code and run it synchronously with WebXR. They may want to parallelize, but we might be able to just keep it simple.

For a while, we were talking about possibly exposing something like OpenVX (https://www.khronos.org/openvx/) so that apps could initialize native "CV graphics" in JS, but they would run nativing, with the option of adding "nodes" in JS/WebAssembly. We might still want to do that, but this might actually be a separate thing that somehow synchronizes with WebXR.

The basic requirement is that whatever data the platform can provide (and that the UA/user allows access to) gets passed to a method written in JS/WebAssembly. I'm not sure the "best" way to implement this (in polyfill or natively), but the idea of "Vision Workers" was floated by some folks. If we could have efficient shared buffers and worker sync, we could let users set up a worker, WebXR could pass it data and then call the program's rAF, the program would sync up with the worker as late as it can (allowing parallelization) and then get the data.

An initialize implementation would have us put JSARToolkit in a worker and pass WebRTC video frames to it.

blairmacintyre commented 7 years ago

12 has some commentary on this now.

blairmacintyre commented 7 years ago

The biggest question is: API. The implementation (for the webxr.js) is pretty easy; esp since we know it won't be "super efficient".

huningxin commented 7 years ago

I am a big fan of this feature. For example, the OpenCV is being brought to Web platform via OpenCV.js in both asm.js and WebAssembly.