search

immersive-web / raw-camera-access

Spec draft: https://immersive-web.github.io/raw-camera-access/. Repository for experimentation around exposing raw camera access through WebXR Device API. Feature leads: Piotr Bialecki, Alex Turner, Nicholas Butko
Other
38 stars 12 forks source link

How to get access to the pixels #6

Open michaelvogt opened 2 years ago

michaelvogt commented 2 years ago

Hello.

I just worked on my code that used the camera-access as it was implemented until Chrome 91.

Access to the camera works it seems, as I can run the sample code provided at the Chromium site successfully. But I fail to get the pixels out of the WebGLTexture object returned by glBinding.getCameraImage() call now.

As WebGLTexture is described as an opaque texture object, I assume that accessing the pixels is not meant to work. But as computer vision is listed as one of the use cases in the explainer, I have the impression access to the pixels is necessary somehow.

Would be interested what the situation and plan is around this subject.

bialpio commented 2 years ago

Thanks for the report! I'd expect readback to be achievable with the API, even though we're defining the textures as opaque. Can you share a code snippet that'd demonstrate how you're attempting to access the pixels? Also, to make sure I understand: did the readback work for you in the past and no longer does?

michaelvogt commented 2 years ago

Hello Piotr.

Thanks for your answer.

Thanks for the report! I'd expect readback to be achievable with the API, even though we're defining the textures as opaque.

I checked again with the debugger, but couldn't find a way. Any suggestions?

Can you share a code snippet that'd demonstrate how you're attempting to access the pixels?

After lots of experimentation (I'm not that good with WebGL), this is the function that worked in the end: https://github.com/OpenArCloud/sparcl/blob/733c6a5a291b446e92b3ccf438eb58abafe93fbc/src/core/cameraCapture.js#L95

Also, to make sure I understand: did the readback work for you in the past and no longer does?

As it turned out, I had a difference between the old and new code for the camera access.

When trying to access the pixels with the code above in the same requestAnimationFrame as the glBinding.getCameraImage() call, the resulting image is all black. To actually get the camera image, I have to run the glBinding.getCameraImage() already in an earlier requestAnimationFrame().

Bugreport in Chromium tracker: https://bugs.chromium.org/p/chromium/issues/detail?id=1233001#c2

bialpio commented 2 years ago

I checked again with the debugger, but couldn't find a way. Any suggestions?

Apologies for confusing phrasing, what I meant was that even though we hand out opaque WebGLTextures, normal (WebGL) ways of reading pixels should work, for example the readPixels() API that you're using.

I took a look at the code snippet you link to and compared it to our internal test case (lines 173-181 & readCameraImageTexturePixels() function). The differences I'm seeing between the approaches we're taking is that in your case, the framebuffer gets recreated every time and we reuse it, and we also call checkFramebufferStatus() prior to calling readPixels().

I took over the bug in Chromium tracker to investigate this further.

michaelvogt commented 2 years ago

Thanks a lot.

Still I would appreciate when there could be a more straightforward way in this spec to get access to the pixels. Maybe getCameraImage() would return something directly usable for cv, and getCameraTexture() for texturing as is shown in the Chromium sample.

A related question if I may: It seems to me, that the dimensions of the returned image depend on the screen size of the device. Is that correct? Would be great to have some control over this.