search

marek-simonik / record3d

Accompanying library for the Record3D iOS app (https://record3d.app/). Allows you to receive RGBD stream from iOS devices with TrueDepth camera(s).
https://record3d.app/
GNU Lesser General Public License v2.1
387 stars 57 forks source link

How can obtain absolute coordinates? #100

Open jeonhuhuhu opened 1 week ago

jeonhuhuhu commented 1 week ago

The Record3D app you developed is very useful and I appreciate it.

I have two questions while using your Record3D app.

First, I wonder if the USB streaming function can fix the camera's initial position when recording and the final position when it ends.

In other words, when using the Record3D app you developed, it seems to have been created with relative coordinates, but instead, I want to force the initial and final positions to be fixed on the same coordinate line.

The reason why I had to do this was because when I recorded (A) and (B) in different locations, there was a problem where the environments created were not combined due to relative coordinates.

So in the end, I created absolute physical coordinates by moving the iPad to location (A) before starting recording, starting recording, and ending recording at location (B).

Therefore, I am wondering if it is possible to forcefully determine the initial and final position coordinates in the code.

If possible, please share the method.

Second, I wonder if USB streaming recording is possible using the app on devices other than iPhone or iPad, such as the D435, a depth camera.

In other words, I wonder if regular Depth cameras other than Apple devices can be used without the app.

Thank you for your valuable and quick replies every time.

marek-simonik commented 5 days ago

I apologize for the late reply.

Answering your first question:

TL;DR: It sounds like as if for some frame of the USB stream, you want to say "assume this camera frame defines the identity pose from now on" so that all subsequent camera poses can be expressed in relative to the coordinate system of that selected frame.

I am not completely sure if I correctly understood, but let me try to answer anyway.

First, I want to explain the logic behind the coordinate system which Record3D uses during USB streaming: imagine the Record3D app is not running in the background (e.g. you have just force-closed it). The pose (orientation and position) your iPhone/iPad is at when you first open Record3D (while in the LiDAR mode) OR when you switch from FaceID to LiDAR, is assumed to be the world coordinate system. Even if you stop the USB stream by pressing the red toggle button and then press the button after a moment again to start the USB stream, the camera pose reported by Record3D will be relative to the same coordinate system. You have to either force-close and reopen Record3D OR switch from the "Record" tab to the Library or "Settings" tabs and then back to the "Record" tab OR switch from LiDAR to FaceID and back to LiDAR to cause a reset of the coordinate system.

In other words, when using the Record3D app you developed, it seems to have been created with relative coordinates

What do you mean by "relative coordinates"? Relative to what? I think the long paragraph above might be useful in answering that question.

I want to force the initial and final positions to be fixed on the same coordinate line

You can select arbitrary frame of the USB stream as the frame in whose coordinate system will the camera poses of all other frames be be in.

I am wondering if it is possible to forcefully determine the initial and final position coordinates in the code

Not sure if I understood what you meant by forcefully determining the final position of a camera. But if you want to use a known 4x4 transformation matrix T_init as the camera-to-world transform of the first frame of your recorded requence, then this is what you could do (pseudocode):

T_init = <the camera-to-world pose you want to use as the pose of the first frame of your recorded sequence>
frame_A_raw_pose = get4x4_transform_matrix( self.session.get_camera_pose() )
inv_frame_A_raw_pose = invert( frame_A_raw_pose )

for frame in <your recorded sequence>:
    curr_frame_relative_pose = inv_frame_A_raw_pose * frame.pose  # Multiply the 4x4 pose matrices to get the current frame's pose relative to the frame A's pose
    curr_frame_final_pose = T_init * curr_frame_relative_pose  # Pose of the current frame in the coordinate system defined by T_init

It sounds like this is your workflow:

  1. You opened the Record3D app and pressed the red toggle button to start the USB stream.
  2. You perhaps moved your iPad so that it is in a position (A).
  3. Once your iPad was physically in the position (A), then in your own Python app, you started recording the USB stream (using your own script) AND you want the first frame of the recorded sequence to have the identity camera pose, so when you move the iPad by 5 centimeters to along the X axis (the X axis determined by the iPad's orientation when it was in the position (A)), you want to see the increment by 5 centimeters just and only along the X axis (and not projected along the other two axes too).

Answering your second question:

Unfortunately, Record3D works just with the built-in cameras of the iPhone/iPad. There is no support for external (depth) cameras. In other words, it is not possible to connect an external depth camera to you iPhone/iPad and use Record3D as a camera app for the external depth camera.

I don't think the first question you asked is related to a bug/issue of Record3D. It sounds like this is a problem related to your own program (which merely uses the Record3D library). But if I am wrong, then please let me know and I'll try to fix a potential problem.