Lack of access to image data per frame restricting use?

[CarinaMclane]

While I expect access to actual pixel data on a per Frame basis is deliberately excluded from the API rather than an oversight, it is currently blocking several avenues of use for us. Is there an intention for this to be accessible in any shape going forward? Or a way to get to this data, if accepting a performance hit?

[small-potatoes]

Hi Carina, you are correct that we did not expose the color camera image in the developer preview. Of course, there are ways to access the image through Unity (e.g. output camera to a RenderTexture) that may work depending on your exact need. Can you tell me a bit more about your use case?

[CarinaMclane]

Sorry - managed to do a finger slip and close the issue!

A simple use case would be your HelloAR example, but placing a different object based on the context containing a particular logo - the image data is already being processed into features for your SLAM, and to do further processing we would need to use similar data, but I believe I only have access to the resulting PointCloud and detected planes rather than the image or any features/descriptors, and the camera is presumably hi-jacked by the ARCore service.

I have played with RenderTextures already, for example with the HelloAR setup adjusted to:

• Render ARCore background to RenderTexture used in a RawImage in a background layer
• Take intermittent snapshots using ReadPixels to use for further processing
• Use a second camera with identical settings (minus the RenderTexture render, and layer filter) as the first person camera for the scene.

But even without the impact of ReadPixels, in the HelloAR example this seems to mess with the perspective expected for anchoring items to the planes as they "drift" across the surface when the camera moves rather than seeming anchored to it. It may of course be me missing some detail with regards to how this setup is matched internally - for example, with camera parameters potentially being adjusted at runtime?

I think it will be an obvious thing that others in our position will want access to as while the location and structure awareness is great, the obvious next thing is to continue to augment with other information!, (Though I appreciate the pulling of image data might have a performance impact).

[CarinaMclane]

So, after spotting the FOV adjustment in SessionComponent I've stopped the drift, but have ended up with a three camera setup (1 x AR Camera to allow image sample before rendering anything else onto it, 1 x background camera to re-draw the background at z: -1 to avoid geometry clipping, 1 x first person camera for 3D content and UI) which "works" but feels "non-optimal"...

So I suppose, I would still like to get behind my initial feature request :)

[momo-the-monster]

+1! Would love to read the color at a given pixel from the camera feed.

[nathanmartz]

Thanks for all the feedback. We're looking into this now.