Alignment of Hologram with QRSpace Pin

[aless98]

https://drive.google.com/file/d/1Ck88Drm9YNpY-RdZjuyQayRUV48rcloX/view?usp=sharing

I will attach directly a video so you can understand my doubt. As you can see i am aligning a cube to a real one. The alignment seems quite ok when i am close but if i move around the object or far away from it, the hologram seems to drift from the original position. As you can see actually world locking tool is implemented as i move around the room pins are placed along the way. I have also tried to place anchors directly on the Cube Gameobject whit some vocal input but still same result. What i would understand is, is it normal that hololens2 experiences this behavior (so there is nothing much to do about it) or can i improve this kind of drft? I have really tried everything is suggested online but nothing seems to improve it.

To give you more info: to align the hologram on the real cube i am tracking the sensor inserted in the cube as you can see. Then I apply a transformation chain to get that coordinates in the Hololens space (spongy space). In the picture, you can see the transformed chain. T_spongy_QR is the transform of the QRspace pin scanned (top left corner) with respect to the spongy space.

Transform .pdf Thanks in advance really. I am getting mad with this one, it would be great if you could give me some hints!

[fast-slow-still]

Okay, your results look very promising.

I think you are introducing a couple of sources of error that you might be able to (fairly) easily correct. These errors are from the QR code scanning, they are not directly related to World Locking Tools.

The Pose scanned from a QR code is not uniform in its accuracy. Understanding which parts of the Pose are more and less accurate can help you use them better.

First, the position scanned from a QR code is much more accurate than the rotation. When you think about it, this is not surprising. Also, if your scanned position has a 1cm error (just an example number), then positions based on it will be 1cm off at the QR code, and also 1cm off 10 meters away from the QR code.

However, if the rotation scanned from the QR code is off by 1 degree, that will cause a positional error of 0cm at the QR code, but 2cm at 1 meter away, and 17cm at 10 meters away. This is called "the lever arm effect." So where possible, you should use multiple positions to imply orientation, rather than the rotation scanned from the QR code.

Second, the position isn't uniformly accurate either. The position along the face of the QR code (the scanned local X & Y axes) is much more accurate than the position perpendicular to the face (the scanned Z axis). Again, this isn't surprising. What is surprising is that the scanned depth is accurate at all.

Looking at your video, it looks like the depth scanned from the QR code was actually under the surface slightly (about 1cm?). This offset all of your holograms down by about 1cm. The rotation also looks slightly off. When you back away from the QR code, visual inaccuracies become apparent because the system is incorrect about where the camera is (the lever arm effect).

It sounds like you really want to push the limits of achievable accuracy. If so, then you need to redesign your QR code layout.

1. Use multiple QR codes to set multiple SpacePins. When you do that, the scanned rotation is discarded, and orientation is inferred from relative positions.
2. Keep them spaced >= 1m apart.
3. Surround the region of interest (ROI) with QR codes, e.g. use 3 QR codes to form a triangle around the ROI, or 4 QR codes to form a square around the ROI.
4. Prefer vertical QR codes to horizontal, and don't have them all facing the same direction.

To really get the maximum positional accuracy out of scanned QR codes, you would want to use pairs of QR printouts perpendicular to each other, discarding the dimension out of the page, and combining along the two faces to get a point in 3D space. But that's outside the scope of World Locking Tools.

[fast-slow-still]

@aless98 also, if you could upload a UnityPlayer.log, I could verify that the systems are all working correctly. But guessing from the video, I think the errors are coming in as I described above.

Thanks!

[genereddick]

As I'm exploring the same issues I'll add re:

Looking at your video, it looks like the depth scanned from the QR code was actually under the surface slightly (about 1cm?). This offset all of your holograms down by about 1cm.

With ARKit image recognition, similar in most respects to the MR QR Code tools though less accurate, it uses the image size property to help estimate the z position from the camera.

A small discrepancy -- even a few mm -- between the image size value you set and the actual size leads to the position being placed above or below the surface, as well as increasing the number of re-estimations as you move around and it tries to fit the mis-sized image from each different angle.

[aless98]

UnityPlayer.log

Here it is @fast-slow-still ! Thanks again. Have a good day.

[fast-slow-still]

Okay, I can see from your log that there are serious issues in the setup of your Unity project. If you are interested in help fixing it, please give the following information:

1. Unity version
2. WLT version
3. WLT source (github source code, MRFeatureTool, github release)?
4. MRTK version
5. Target platform(s) (e.g. HoloLens, HoloLens2, Android, iOS)
6. Anchor subsystem (in WorldLockingManager/WorldLockingContext/Anchor subsystem)
7. VR/XR provider (in Unity Player settings and/or Unity XR Plugin Management settings)

[aless98]

1. Unity version 2019.4.36f1
2. WLT version 1.5.8
3. WLT package source is your GitHub repo (the QRSpace pin project)
4. MRTK version 2.7.2.0
5. Platform Hololens2, in the unity settings though I just selected any device
6. anchor subsystem WSA
7. XR setting in the player setting. Depth format 16-bit and Depth sharing buffer enabled.

[fast-slow-still]

Did you install the Frozen World Engine? https://docs.microsoft.com/en-us/mixed-reality/world-locking-tools/documentation/howtos/initialsetup#manual-frozen-world-engine-dll-installation

[aless98]

![Uploading Screenshot (50).png…]() Yes it is installed.

[aless98]

Microsoft.MixedReality.Unity.FrozenWorld.Engine is installed as a NuGet package

[fast-slow-still]

I'm looking at the following line from your UnityPlayer.log:

Creating PluginNoop, is this intentional?

The only condition I know of which could cause that would be that the Frozen World Engine isn't installed. I can't seem to download/preview your screenshot. Are you using NuGet For Unity, or the command line nuget?

Also, there are the many many InvalidOperationExceptions found later in the log:

InvalidOperationException: Nullable object must have a value. at System.Nullable`1[T].get_Value () [0x00000] in <00000000000000000000000000000000>:0 at Update_posit.Compute () [0x00000] in <00000000000000000000000000000000>:0 at Update_posit.FixedUpdate () [0x00000] in <00000000000000000000000000000000>:0

Unfortunately, the InvalidOperationException doesn't give enough information to guess where the problem is.

[aless98]

I am using Nuget from Unity.

Maybe the Update_posit.Compute () error showing is due to the fact that for example if I have not scanned my QR code yet the transform between the head user and the QR is null so basically in that script I use that transform to Multiply another one and that's the error, i am multiplying something for a null value. Btw I don't think it is not a problem because as soon I scan the QR code, THe position of the cube is eventually correct.

Also, I was trying to think about this:

To really get the maximum positional accuracy out of scanned QR codes, you would want to use pairs of QR printouts perpendicular to each other, discarding the dimension out of the page, and combining along the two faces to get a point in 3D space. But that's outside the scope of World Locking Tools.

The algorithm takes the pose of the frame on the top left corner of the QR codes with respect to the user's head, so the idea is to hypothetically get the line of intersection between them and place a spatial anchor in that location(?).

[fast-slow-still]

The algorithm takes the pose of the frame on the top left corner of the QR codes with respect to the user's head, so the idea is to hypothetically get the line of intersection between them and place a spatial anchor in that location(?).

Yes, if you know the physical configuration of the two QR codes, then the two pairs of XY coordinates (along the two faces) gives enough information to establish a point in 3D space, without needing the less reliable Z coordinates (out of the face).

[fast-slow-still]

I am using Nuget from Unity.

Can you give a screenshot of your Frozen World install in NuGet for Unity? It should look like this:

I find it very troubling that your UnityPlayer.log says you are using the no-op Plugin. And that UnityPlayer.log was from deploying to HoloLens2 and running on device?

[aless98]

That's it, maybe I provided you with the wrong UnityLog file. Here is another one.

UnityPlayer.log

Yes, if you know the physical configuration of the two QR codes, then the two pairs of XY coordinates (along with the two faces) give enough information to establish a point in 3D space, without needing the less reliable Z coordinates (out of the face).

So ideally I would get the pose of the 2 QR codes and discard the z coordinates of both and use just the x and y values of both frames to intersect and place an anchor there. But what about the orientation? Again you are really helpful thanks for this discussion.

Alessandro

[aless98]

yes the UNityPlayer was done by running on the holo the app and then downloading it from the Windows Device portal in the File explorer section _[User Folders]() \ [LocalAppData]() \ [OpenIGTlink_1.0.0.0_arm64_sc9m1gdp5g340]() \ [TempState]() \

[fast-slow-still]

That's it, maybe I provided you with the wrong UnityLog file. Here is another one.

Ah, great, looks like that was just a bad log file, the second UnityPlayer.log you posted looks fine (except for the InvalidOperationExceptions, but as you point out, those go away once you start scanning QR codes).

[fast-slow-still]

So ideally I would get the pose of the 2 QR codes and discard the z coordinates of both and use just the x and y values of both frames to intersect and place an anchor there. But what about the orientation?

There are a number of ways you could approach this. What I had in mind was to use the scanned orientation to determine a face normal. As I mentioned, the orientation is less accurate than the position, but since we would only be using it to project along the normal a couple of millimeters, it should be accurate enough.

After scanning both faces A and B, you have 3D positions for each (P_a & P_b), and a face normal for each (N_a & N_b). (You also know that the face normals should be perpendicular to each other, so you could check that now to validate).

You know the distance along N_a from P_a to P_b. The untrusted dimension is along N_a, so you project P_a along N_a to be the correct distance along N_a from P_b. That correction should be a small number of millimeters or less.

Then you do the same with P_b along N_b.

Full disclosure here, I haven't tried doing this. Your mileage may vary.

[aless98]

Very interesting. I will try and dig into that. In case it would be great if we could keep the discussion about that in the future. For the moment again thanks for the answer and your time. In case you agree, should i open a new issue for that or can we keep this one open?

[fast-slow-still]

Let's close this issue, and start a new one. I'm getting confused about what this issue is about. 😆

[GiovanniTurri]

Thank you everyone for all the insights of this thread!

@aless98 , I'm working on this example too, maybe in two we can test and improve it faster and then open a new issue (or continue this). Feel free to coordinate with me through giovanni.turri[at)edu.unife.it