search

marek-simonik / record3d

Accompanying library for the Record3D iOS app (https://record3d.app/). Allows you to receive RGBD stream from iOS devices with TrueDepth camera(s).
https://record3d.app/
GNU Lesser General Public License v2.1
379 stars 55 forks source link

Intrinsic matrix in .r3d recordings #83

Closed tszolar closed 2 months ago

tszolar commented 3 months ago

Hi, is there a specific reason that in the internal recording format (.r3d), there is only one intrinsic matrix stored per recording and not for each frame? Are those always values from the first frame?

I am doing some 3D reconstruction experiments with iPhone 15 Pro LIDAR where I need accurate intrinsic parameters for each frame and one intrinsic matrix per whole recording doesn't seem to be sufficient. I did also some recordings via the USB streaming mode, where the intrinsics are provided for each frame, and the principal point seems to be moving around quite a lot (presumably because of active image stabilization) during the recording.

image

Here is a visualization of the principal point positions for all frames in 4 short USB streaming mode recordings. In 2 recordings the phone is static and in the other 2 it is hand-held and moving. It seems to be only changing significantly in case the camera is also moving, which would further confirm the image stabilization hypothesis. Auto-focus option does not seem to have much impact on it.

marek-simonik commented 3 months ago

Hi,

I am sorry for the late reply. As you correctly said, the K key in the metadata file stores (some of the) coefficients of the intrinsic matrix of the first frame of the 3D video.

Back when I first released Record3D, every frame had the same intrinsic matrix, which is why I opted for storing just a single instance of intrinsic camera coefficients.

As this assumption is no longer valid, I will introduce a new key into the metadata file, which will store per-frame intrinsic coefficients. This will be available in the next Record3D update.

Thanks for suggesting this.

marek-simonik commented 2 months ago

This feature has been implemented in Record3D 1.10, please update.

Beginning with Record3D 1.10, the metadata file of a 3D video contains a new field called perFrameIntrinsicCoeffs, which holds per-frame intrinsic camera coefficients.

j96w commented 1 month ago

Hi @marek-simonik , Thanks for sharing the updates. However, I find the length of perFrameIntrinsicCoeffs does not match the number of frames in the video. For other metadata like poses and frameTimestamps, their length do match the number of frames. How to get the accurate intrinsic for each frame if the data length is different?

j96w commented 1 month ago

After upgrading the app, now it works