Inquires on ARCore recording features

[Basel-Salahieh]

ARCore recording sample app by default records texture and raw depth and encodes them (apparently) as 640x480 yuv-420-888 into multiple video tracks in H264 mp4 file. I have the following questions that I appreciate getting answers on:

1. Since the original raw depth data is 13 bit, how exactly it is converted into yuv-420-888 format? How can I retrieve the metric depth during the playback from such format?
2. Can I switch the depth map to use the one by the original Depth API (with all complete pixels) instead of the raw Depth API? Where exactly should I change to get such results in the sample app?
3. The FOV of the raw depth (ToF) camera differs from the associated color image, so how can we get alignment on pixel level between the texture and depth?
4. How can I record a higher resolution color video (1920x1080) and add as a video track into the outputted mp4 file?

[Basel-Salahieh]

Just to clarify, this is not a bug but request for additional information... Sorry for the wrong label (as I cannot change it!).

[devbridie]

4. How can I record a higher resolution color video (1920x1080) and add as a video track into the outputted mp4 file?

You'll need to select a camera configuration that supports the desired size in getImageSize() using setCameraConfig.

Asking for somebody else to clarify the other points.

[devbridie]

Okay, got some clarification on the parts I wasn't too sure about:

1. Since the original raw depth data is 13 bit, how exactly it is converted into yuv-420-888 format? How can I retrieve the metric depth during the playback from such format?

Raw depth data isn't included in the recording, only the smoothed depth data is. To retrieve the metric depth, see the example snippet on the developer site.

2. Can I switch the depth map to use the one by the original Depth API (with all complete pixels) instead of the raw Depth API? Where exactly should I change to get such results in the sample app?

During playback, use e.g. acquireRawDepthImage to obtain the raw depth image instead of the smoothed image.

3. The FOV of the raw depth (ToF) camera differs from the associated color image, so how can we get alignment on pixel level between the texture and depth?

ARCore adjusts the FOV for the depth images such that it aligns with the display texture.

[Basel-Salahieh]

Thank you @devbridie for the useful info. Few more questions!

I was able to get the high resolution color video in addition to low resolution color one + depth but noticed that the two color videos are out of sync (e.g. differ in in number of frames possibly due to some frame drops in either channel) and frame rate dropped from 30fps to ~26.5 fps so is there a way to have them aligned and keep frame rate at 30fps (not that I'm using CameraConfig.TargetFps.TARGET_FPS_30 to setTargetFps)? Is ARCore pulling frames twice (at the two different resolution) from the camera or it is just pulling the high res version then resize that to the 480p version needed for the motion tracking?

During playback, use e.g. acquireRawDepthImage to obtain the raw depth image instead of the smoothed image.

Would this mean that the depth information written as yuv420p is sufficient to retrieve the raw depth, smoothed depth, and confidence map at the playback side?

Also, the metric depth retrieval shared above is to return the depth value when it is already stored as 16bit uint, but it is not yet clear how the 16bit depth map is being computed from the recorded yuv420 8bit depth. What ARCore functionality should be used for that?

ARCore adjusts the FOV for the depth images such that it aligns with the display texture.

Is there a way to modify that such that the recorded depth video has FOV that exactly matches the recorded color video (both at 640x480 resolution)? It will be beneficial to get the pixels perfectly aligned between the recorded depth and color video regardless of the display setting.

[devbridie]

and frame rate dropped from 30fps to ~26.5 fps

ARCore may be dropping frames when the device load is too high to record both streams at the same time at 30 fps. In order to keep the framerate at 30 you'll probably need to choose lower resolution streams. The CPU stream (lower resolution) is the camera stream resampled in order to save processing load, so no double frame pulling happening.

Would this mean that the depth information written as yuv420p is sufficient to retrieve the raw depth, smoothed depth, and confidence map at the playback side?

Yes, you are able to use all ARCore functionality during playback as if you weren't playing back a file at all. For the purposes of ARCore, it doesn't matter whether the session comes from a file (recording) or from a live moving devices with a real camera stream. So, to access the depth map (whether it's raw depth or smoothed depth), you would use this snippet to access metric depth from a depth image.

Is there a way to modify that such that the recorded depth video has FOV that exactly matches the recorded color video (both at 640x480 resolution)?

My understanding is that that is the case currently; ARCore will process the images such that the depth image matches the color image. Are you seeing different behavior?

[Basel-Salahieh]

ARCore will process the images such that the depth image matches the color image. Are you seeing different behavior?

Yes, below is the side by side color & depth at 640x480 recorded by Samsung S20+ phone (using the ARCore record and playback sample app). You can see that the depth map has actually smaller FOV compared to the color one making the pixels out of alignment.

Also I'm wondering of how to recover the edge (black) regions in the depth map given that this is the smoothed version and should be a more complete version (per Depth API).