[feature-request] also get the 6doF camera odometry

[soumith]

Great app, we were rolling something in-house for some research at NYU, but we might switch to using your app.

Currently, rgb, depth and intrinsic parameters are exposed. However, for full 3D reconstruction or SLAM applications, it would be essential to also stream the iphone extrinsics (i.e. XYZ locations, camera normal, etc.). The closest project that exposes this info (but for logging onto iphone local disk) is https://github.com/PyojinKim/ARKit-Data-Logger

Is this something that you have plans to expose? It would be really helpful for SLAM applications

[marek-simonik]

The next version of Record3D (1.7.2) will stream the ARKit world pose (quaternion + world position). It should be released by the end of the next week.

[soumith]

that is awesome, thank you!

[marek-simonik]

Please update Record3D to version 1.7.2, update the record3d to version 1.3.1; camera pose data for LiDAR are available.

For Python, see (read the comment) https://github.com/marek-simonik/record3d/blob/d712f4026d26a6cf3c7f697e4199ae5bf47395f1/demo-main.py#L53

And for C++, see https://github.com/marek-simonik/record3d/blob/d712f4026d26a6cf3c7f697e4199ae5bf47395f1/src/DemoMain.cpp#L120

[soumith]

Hi @marek-simonik I am running this on an iPhone 12 mini, which doesn't have LiDAR. So, it does report xyz to always be 0, 0, 0.

I am wondering why the requirement for LiDAR -- actually if you do LiDAR based, it will be more inferior than setting a WorldTracking session which does sensor fusion among camera, lidar and intertial sensors.

Wouldn't it be better and more widely supported (on the non-Pro models too) if you created a ARWorldTrackingConfiguration() session and then get the current ARFrame's frame.camera.transform?

[marek-simonik]

Record3D currently uses ARKit for access to LiDAR data (hence the availability of estimated pose data), but FaceID streaming does not use ARKit; the last time I checked, the ARKit API did not provide 640x480 px resolution option for FaceID depth maps.

[soumith]

I agree that streaming of phone's 6dof odometry is completely independent of FaceID depth maps.

Record3D currently uses ARKit for access to LiDAR data (hence the availability of estimated pose data)

What I am saying is, the estimated 6dof pose is available in ARKit even when there is no LiDAR. It fuses information from the back-camera in low-fps and the IMU sensor. That estimated 6dof pose wont correspond to the FaceID depth maps exactly, but we can compute a fixed transform from the ARKit's ARWorldTrackingConfiguration's frame.camera.transform and the FaceID depth transform.

[marek-simonik]

I'm not sure if it's possible to have 640x480 px TrueDepth camera stream running via AVCaptureSession (which is what Record3D does now) and also having an ARKit ARWorldTrackingConfiguration session running at the same time.

If I'm not mistaken, then ARKit does internally use the AVCaptureSession too and since iOS doesn't allow more than 1 AVCaptureSession running at the same time, the TrueDepth camera's AVCaptureSession would automatically stop as soon as I would start an ARKit session.

I currently cannot think of a way how to get 640x480 px TrueDepth depth map stream and ARKit's pose estimate (estimated using the rear cameras with IMU sensor fusion, as you say) on all devices with the TrueDepth camera — am I missing something obvious?

[soumith]

I'm not sure if it's possible to have 640x480 px TrueDepth camera stream running via AVCaptureSession (which is what Record3D does now) and also having an ARKit ARWorldTrackingConfiguration session running at the same time.

Got it. I did not know this. Thanks a lot for the information.