Transforming Aruco marker pose estimation to unity world coordinate

[Metacadx]

Hello , i used client stream pv to perform an Aruco code detection and pose estimation and i managed to send the data to unity however i can't find a way to superpose a 3D model onto the Aruco marker , do you have any suggestion ? Thank you for your help !

[jdibenes]

Hello, both the pv pose and aruco pose are expressed in right handed coordinates while unity uses left handed coordinates. Are you performing right hand to left conversion before applying the final transform to the 3D model?

[Metacadx]

yes of course but i 'am not sure if i am doing thinks correctly when applying the transformation, I ve been looking for a solution for a while but seems like there is a lot of matrix transformation , so i am looking for a simpler and less expensive solution for the hololens 2 and the way i am doing , i feel like i am reinventing the wheel , is there any build in method or a simple equation to get the absolute position of aruco code using the transform of the camera ?

[jdibenes]

The equation is aruco_point_world_h = aruco_point_h @ aruco_pose^T @ pv_extrinsics^-1 @ pv_pose where _h is for homogeneous 3D coordinates, aruco_pose is from solvePnP converted to a homogeneous 4x4 matrix, pv_extrinsics is a 4x4 constant, and pv_pose is the 4x4 transform of the PV frame. We have included a demo that demonstrates this conversion and the right to left hand conversion required for Unity (lines 100-119).

[Metacadx]

Thank you for you answer , there is something strange happening i get this error message ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host , the IP adress is correct do you know where this error comes from ?

[Metacadx]

Also there is a weird windows appearing and disappearing real quick in the HoloLens application i can't tell what is it and this was not happening in the older build I am using unity 2022.3.20f1

[jdibenes]

For Unity 2022 enable Sync Capabilities in Project Settings > Player > Publishing Settings. The library and binaries were updated recently, check that you have the latest version of the files (hl2ss.py, hl2ss.dll in the unity folder, etc.). Since the dll was updated, the unity project has to be rebuilt from scratch (delete the old build folder and create a new one).

[Metacadx]

I just updated to the newest version of the project but i can't find the Sync Capabilities in unity

[jdibenes]

My bad, I was using unity 2022.3.24f1. I see now there is no sync capabilities in 2022.3.20f1, so the capabilities are set correctly already.

[Metacadx]

Okay i will try to upgrade the unity version ! Do you have any plan on creating unity sample client ?

[jdibenes]

do you mean a prebuilt version like the main app? If so, I have added it to the latest release.

[Metacadx]

No i mean an actual unity scene that runs as a c# client just like the python client ( not in c++ )

[jdibenes]

No. There are no plans for a unity c# client.