[feature request] Rear RGB+Odometry

[soumith]

Record3D successfully delivers on some parts of the following matrix:

Device and Sensor	RGB	Depth	Odometry
Pro FaceID	Y	Y	N
Pro LiDAR+Rear Camera	Y	Y	Y
non-Pro Face ID	Y	Y	N
non-Pro Rear Camera	N	N	N

The Pro models expose RGB + Depth + Odometry. The non-Pro models expose only RGB + Depth (via FaceID cam).

This feature request is to request the enablement of the last line of this matrix, i.e. enable RGB + Odometry on non-Pro models. The motivation here is to use the much cheaper non-Pro models on a variety of robotics research settings. Mixing and matching the iphones can provide us with a more cost-effective research. There are many settings in which we don't need to use the depth, but need RGB and Odometry. We think this feature would be useful not just for us but for many others.

Thank you!

[marek-simonik]

Would it be enough for your purposes if I added non-Pro rear RGB + Odometry only for USB streaming? I.e. without the ability to save videos in this configuration (the app relies on depth data for the 3D video player and export features). Do you have a deadline for this feature request?

[soumith]

There is no deadline for this feature request :)

it would be sufficient if you enable USB and Wifi streaming. But ideally, the ability to record is there. We are developing some 3D depth recovery algorithms, for which humans are walking around collecting trajectories (right now with Pro models). In the recording mode, even if the depth is mocked out to all ones (i.e. just all points at 1m) or something like that, we are fine. We use / read .r3d files on our computer in Python. We don't use other formats.

[marek-simonik]

If you goal is to collect trajectories with as low drift as possible, then I would suggest to keep using the Pro models with LiDAR. The camera poses provided by Record3D are taken from ARKit, which uses LiDAR for SLAM on the Pro models.

I expect trajectories produced by ARKit on non-Pro models to have worse drift, but I will check this and will report back. (Would you still be interested in using the non-Pro models if the camera poses had significantly worse drift than the Pro models?)

Thanks to LiDAR, Apple's APIs deliver almost driftless tracking — I think they internally use the precise, raw & sparse LiDAR point clouds for their SLAM. Unfortunately, there is no public API for 3rd-party developers to access the accurate, raw & sparse LiDAR data; we only get low-quality depth maps produced by Apple's neural net that fuses the accurate sparse LiDAR data and the RGB camera images to produce dense (pseudo)depth maps.

P.S. Currently, Record3D does not stream pose data during Wi-Fi streaming (it uses WebRTC for Wi-Fi streaming, for which — as far as I know — there is no easy and widely-supported way of embedding metadata to each frame).

[soumith]

sorry for the late reply.

I expect trajectories produced by ARKit on non-Pro models to have worse drift, but I will check this and will report back. (Would you still be interested in using the non-Pro models if the camera poses had significantly worse drift than the Pro models?)

I think the answer is relative. If the drift is really bad, it's a problem, but we have not see it to be too bad in practice. Additionally, the SLAM for the non-Pro models also seem to have loop closure, so the drift doesn't always compound.

P.S. Currently, Record3D does not stream pose data during Wi-Fi streaming (it uses WebRTC for Wi-Fi streaming, for which — as far as I know — there is no easy and widely-supported way of embedding metadata to each frame).

Thanks, we have been using USB streaming mostly, so, we didn't even realize this. Just a wild thought (just for thought's sake, i dont expect this to be implemented) -- one can be clever and add in artifacts into the image or depth that are preserved under compression, and you recover + delete the artifacts on the client side as you are reading frames -- kinda like steganography without the overkill of obfuscation -- simple wavelet patterns will probably do.

[marek-simonik]

In the upcoming version of Record3D, it will be possible to optionally enable rear camera recording and USB streaming on non-LiDAR devices. But since I am still working on a few minor features that I want to include in the upcoming update, let me give you access to the pre-release version today, so that you don't have to wait. I sent you an email.

Thanks, we have been using USB streaming mostly, so, we didn't even realize this. Just a wild thought (just for thought's sake, i dont expect this to be implemented) -- one can be clever and add in artifacts into the image or depth that are preserved under compression, and you recover + delete the artifacts on the client side as you are reading frames -- kinda like steganography without the overkill of obfuscation -- simple wavelet patterns will probably do.

Thanks for the tip! I'll definitely look into this as I'm curious how well it will be resilient against WebRTC compression.

[marek-simonik]

Record3D 1.8 lets you enable rear-camera streaming even on non-LiDAR devices. If you want to enable this feature, go to the iOS Settings app > Record3D > enable Rear Camera recording on nonLiDAR devices and force-kill Record3D. The depth data will always be 1.0 meters and you will be able to record 3D videos and also stream. Please let me know if this is what you wanted. Thanks.

[soumith]

thank you. Sorry I could not try things yet due to some paper deadlines. We will check it out soon.