search

kekeblom / StrayVisualizer

Visualize Data From Stray Scanner https://keke.dev/blog/2021/03/10/Stray-Scanner.html
MIT License
72 stars 13 forks source link

Asking about the registration of rgb and depth images, #12

Closed PyojinKim closed 2 years ago

PyojinKim commented 2 years ago

Hi Kenneth,

I have been looking for an app like this, to record RGB + depth images from the iPhone/iPad LiDAR sensor. Very cool!! :) Could I ask a few questions about the data acquisition process and data itself??

1) alignment (registration) between RGB and depth images It seems like the RGB, depth image resolutions are fixed as 1920x1440 / 256x192 in ARKit mode. (right?) If we downsample RGB as 1/7.5, then their sizes are exactly the same, showing good RGB and depth alignment.

I am very curious that the alignment (registration) between RGB and depth images is already done? (no need for camera extrinsic calibration parameters between the RGB and depth cameras?) RGB and depth images share the same camera intrinsic parameters stored in camera_matrix.csv? RGB's K will be the matrix in the camera_matrix.csv file, and depth's K will be RGB's K / 7.5,

Is this the correct understanding?

2) iPad Pro vs. iPhone 12 Pro Max It seems like the same RGB, depth image resolution is used in both iPad Pro and iPhone 12 Pro Max (AFAIK). Is there any difference in RGB, depth, and confidence data when recording datasets on both iPad Pro and iPhone 12 Pro Max?

3) adjustment of recoding FPS Is it possible to adjust the FPS setting of RGB, depth, and confidence images? It seems like the default FPS (60 FPS, right?) is too fast sometimes, so the data size grows too fast.... It would be really nice to have a setting that allows the user to set the FPS before data acquisition in the app.

4) camera_matrix.csv As far as I know, Apple ARKit estimates and provides K (camera matrix) for every frame. How is the K (camera matrix) determined? from the very first frame or the last frame??

5) depth image from depthMap As far as I know, we can access the depth map from the LiDAR sensor through didUpdate frame: ARFrame. (Swift) In this app, is the depth map to be saved with frame.sceneDepth?.depthMap or frame.smoothedSceneDepth?.depthMap ? Also, I am just curious what is the difference between them....

6) depth map format (npy) Is there any specific reason to save the depth map as .npy format? I think we can just directly save the depth map as .png format similar to confidence image..

I really appreciate your time and concerns for my several questions. Please let me know if you have any questions.

Thanks, Pyojin Kim.

kekeblom commented 2 years ago

Hi Pyojin,

  1. Yes the alignment between the depth frames is already done internally by ARKit. As you say, you can either downsample the RGB frame or upsample the depth frame with exact nearest neighbor interpolation, and you should get RGB-D frames. The intrinsics, should be shared across the depth and RGB frame.

  2. I haven't used the iPad pro myself, though from what I have heard and read, they seem to be running the exact same depth sensor. The iPhone 13 Pro, I think, had some minor update related to the range, though I am not sure about this and don't have hard data.

  3. The FPS thing is something I have been needing myself and have considered implementing. I hope to get to implement this soon.

  4. The camera matrix is saved from the last frame.

  5. It is using frame.sceneDepth?.depthMap, which is also post processed a little bit by internals of ARKit, but there is no temporal smoothing.

  6. Yes, the npy thing was an early decision made and indeed, I could have used png. The confidence maps were added in a later version and then I saved them as PNG without updating the depth maps, as I didn't want to break the code that wasn't using the confidence maps yet. I guess I should just change it to png in the next version. The change for any code reading the frames would be quite small and thus shoud be easy to upgrade. PNG would be more efficient.

Cool that you have gotten some use out of the app! Let me know what you build with it!

PyojinKim commented 2 years ago

Hi Kenneth,

I really appreciate your fast and detailed answers to my questions. :) Currently, I am testing some VO/SLAM algorithms, thus I will use this app for collecting some rgb-d datasets. At first, I thought of using Intel L515, but its effective depth range is about ~ 3 m.... too short... Definitely, iPhone/iPad LiDAR sensor shows much better results.

Anyway, thank you for developing such a valuable app!! I hope the FPS adjustment feature update will come soon. I will close this issue.

Thanks, Pyojin Kim.

kekeblom commented 2 years ago

@PyojinKim You'll be glad to hear that I just submitted a version to the App Store that allows setting the frame rate and saves depth frames as png.

The available settings are 60, 30, 15, 5 and 1. Would those work for you?

PyojinKim commented 2 years ago

@PyojinKim You'll be glad to hear that I just submitted a version to the App Store that allows setting the frame rate and saves depth frames as png.

The available settings are 60, 30, 15, 5 and 1. Would those work for you?

Wow, Thanks for the quick update on my feedback! :) I will check it out sometime next week.

Btw, I made a MATLAB parser and visualizer to work with the RGB-D data from your Stray Scanner app, https://github.com/PyojinKim/StrayScannerVisualizer Hope it will be helpful for the MATLAB user. :)

Thanks, Pyojin Kim.