ArUco Marker Tracker without Research Mode

[verenoiu]

Hi!

I started writing an ArUco based marker tracker for the HoloLens, using the HoloLensForCV project as a base and came across a few issues that maybe you could help me with. In contrast to the ArUcoMarkerTracker, I only want to access the PhotoVideo SensorType. So instead of triangulation between the sensors and the detected corner locations of the marker, I'm using the camera intrinsics provided by SensorFrame and the void cv::aruco::estimatePoseSingleMarkers(InputArrayOfArrays corners, float markerLength, InputArray cameraMatrix, InputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs ) to get the estimated position and orientation of the tracked marker. On this pose I'm applying different transformations to get the position in the right coordinate system:

_markerSlateRenderer->SetMarkerMatrix(markerTransformationCV * (frameToUsedCoordinateSystem * headCoordinates));

with

• markerSlateRenderer: SlateRenderer, expanded by the SetMarkerMatrix, setting position and rotation of the hologram
• markerTransformationCV: Windows::Foundation::Numerics::float4x4 representation of the rvec and tvec provided by estimatePoseSingleMarkers (with the coordinate system already adapted and the Rodrigues vector already converted into a rotation matrix)
• frameToUsedCoordinateSystem: result of latestFrame->FrameToOrigin (with latestFrame of type HoloLensForCV::SensorFrame)
• headCoordinates: result of float4x4 headCoordinates = make_float4x4_world(poseOriginPos, poseOriginForward, poseOriginUp) with the head pose information coming from Windows::UI::Input::Spatial::SpatialPointerPose^ poseOrigin = Windows::UI::Input::Spatial::SpatialPointerPose::TryGetAtTimestamp(_spatialPerception->GetCoordinateSystemToUse(), holographicFrame->CurrentPrediction->Timestamp);

While the orientation of the virtual (hologram) marker looks fine and aligns with the real world marker, the position seems a bit off (a small, but significant offset). I also noticed that when reducing the frame resolution (and with that increasing the fps) the positioning gets a little bit more accurate. Do you have any idea where this offset might come from? Is there a mistake in those transformations or could this inaccuracy be caused by only using one camera frame for position prediction? I'm relatively new to this marker tracking for HoloLens topic, so any suggestions might be really helpful. Thank you in advance!

[nmsmith]

I'm about to embark on a similar project, so I'd like to hear about whether this is an actual bug or if it has/can be resolved.

[nmsmith]

I think the same problem with the RGB camera projection is mentioned in this issue? https://github.com/Microsoft/HoloLensForCV/issues/64#issuecomment-432298029

[franckies]

I'm the only one getting an FPS of 7 instead of 60 running the application? Reducing frame resolution solved for you?