Support for ARKit Pose during the video session?

[y22ma]

I'm a computer vision researcher and would love to get poses for frames we record during dataset collection

[marek-simonik]

You can already obtain pose data for pre-recorded LiDAR videos. To get access to them, export your video into .r3d, unzip the .r3d file and open the "metadata" file (it is a JSON file).

The pose data for each frame are stored as an array of 7-element arrays. The first 4 elements represent the quaternion and the rest is the world pose.

Please let me know if this answers your question.

[y22ma]

That's perfect! Thanks for the tip!

[areiner222]

Thanks for the awesome app and ease of data export!

I'm using the app on ios and, If I'm not mistaken, I notice that the pose quaternion is measured relative to the starting camera orientation (starts at "0" rotation).

Is it possible to extract the initial pitch / roll relative to gravity based on the phone accelerometer from the exported app data?

[marek-simonik]

Yes, it is true that the initial pose is "zero" and that there is no way to tell the "true" initial orientation. The reason why I chose the app to behave like this is because I think this is more natural for the primary purpose of Record3D — i.e. for making 3D videos and using them e.g. in music videos.

[areiner222]

Appreciate your response.

That makes sense to me. Would you be open to including the initial orientation via accelerometer reading in the metadata? That way, one could switch between the relative pose and "true" orientation over the whole trajectory of poses.

It would greatly help me with some of the 3D scene modeling I am doing.

Thanks again!

[marek-simonik]

That shouldn't be a problem — I will include the "true" initial device orientation into the metadata in the next Record3D update.

[areiner222]

Any info on when the next update might be? Thank you!

[marek-simonik]

The next update should be released by the end of the next week, I apologize for the delay.

[marek-simonik]

Any info on when the next update might be? Thank you!

I included the initial frame's pose into the initPose field of the metadata file (format: qx, qy, qz, qw, x, y, z). This feature is a part of Record3D 1.6.12, which should be live on the App Store within ~2 days.

[areiner222]

@marek-simonik the world pose information has been working great for me this week!

I have one follow up question, is there a straightforward way to convert the camera translation trajectory from being relative to the initial frame of reference to always being relative to the camera coordinate system? I believe there should be enough information to do this currently, but just looking for a simple methodology.

For e.g., I'd want movement forward in a straight line followed by a 180 degree turn followed by forward motion again to map to a monotonically decreasing trajectory in only the camera-z direction instead of a decreasing then increasing trajectory.

Thanks again for all your help.

[marek-simonik]

I'm not sure if I understood your question correctly, but if you want to transform a point P so that it's expressed relative to a specific coordinate system (represented by a transformation matrix T), then you can multiply P by the inverse of T (i.e. T^{-1}·P).