Recommended way to get the frozen pose of new QR Codes?

[zantiu]

What is the correct way to get the frozen pose of newly scanned QR Codes (So QR codes that are not attached to spacepins)?

Currently I have following code:

private static Pose GetPoseFromNewQRCode(QRCode qrCode, out bool success)
{
    success = false;
    var pose = new Pose();

#if WINDOWS_UWP

    SpatialCoordinateSystem CoordinateSystem = Windows.Perception.Spatial.Preview.SpatialGraphInteropPreview.CreateCoordinateSystemForNode(qrCode.SpatialGraphNodeId);

    // check to verify the coordinatesystem for this node is not null
    if (CoordinateSystem != null)
    {
        // create identity/zero rotation and translation
        Quaternion rotation = Quaternion.identity;
        Vector3 translation = new Vector3(0.0f, 0.0f, 0.0f);

        // GetNativeISpatialCoordinateSystemPtr is obsolete!
        SpatialCoordinateSystem rootSpatialCoordinateSystem = (SpatialCoordinateSystem)System.Runtime.InteropServices.Marshal.GetObjectForIUnknown(UnityEngine.XR.WSA.WorldManager.GetNativeISpatialCoordinateSystemPtr());

        // Get the relative transform from the unity origin
        System.Numerics.Matrix4x4? relativePose = CoordinateSystem.TryGetTransformTo(rootSpatialCoordinateSystem);

        if (relativePose != null)
        {

            System.Numerics.Vector3 scale;
            System.Numerics.Quaternion rotation1;
            System.Numerics.Vector3 translation1;

            System.Numerics.Matrix4x4 newMatrix = relativePose.Value;

            // Platform coordinates are all right handed and unity uses left handed matrices. so we convert the matrix
            // from rhs-rhs to lhs-lhs 
            // Convert from right to left coordinate system
            newMatrix.M13 = -newMatrix.M13;
            newMatrix.M23 = -newMatrix.M23;
            newMatrix.M43 = -newMatrix.M43;

            newMatrix.M31 = -newMatrix.M31;
            newMatrix.M32 = -newMatrix.M32;
            newMatrix.M34 = -newMatrix.M34;

            System.Numerics.Matrix4x4.Decompose(newMatrix, out scale, out rotation1, out translation1);
            translation = new Vector3(translation1.X, translation1.Y, translation1.Z);
            rotation = new Quaternion(rotation1.X, rotation1.Y, rotation1.Z, rotation1.W);
            pose = new Pose(translation, rotation);

            // If there is a parent to the camera that means we are using teleport and we should not apply the teleport
            // to these objects so apply the inverse
            if (CameraCache.Main.transform.parent != null)
            {
                pose = pose.GetTransformedBy(CameraCache.Main.transform.parent);
            }

            success = true;

        }
    }

#endif

    return pose;

}

The positions from the resulting pose are correct, however the rotations are often shifted by 180 degrees. For example in a test:

1. add two spacepins to qr-codes positioned on the x-axis with coordinates (0,0,0) and (2,0,0) (qr-codes in the XY-plane)
2. scan a new qr code on the x-axis (in the XY-plane) with coordinate (1,0,0)

Multiple scans give following results (Euclidian rotations in degrees):

• 359.9492, 2.909461, 0.903221
• 357.6101, 182.9058, 180.9385
• 357.1973, 182.2521, 180.7275
• 357.0967, 181.965, 180.6811
• 356.9806, 181.8914, 180.6653
• 358.108, 1.143362, 359.558
• 358.3949, 1.034422, 359.475
• 358.8199, 1.068542, 359.5049
• 358.8633, 1.850759, 359.7042
• 358.5123, 2.797868, 359.7931
• 358.9621, 183.0922, 179.9445
• 359.0522, 182.5009, 179.8562
• 358.8767, 181.697, 179.6391
• 358.7914, 181.4219, 179.6022
• 357.2258, 181.7257, 179.5746

So about half of the time the y and z rotation are both offset by 180 degrees.

What is the recommended way to get the pose that will always be correct?

[fast-slow-still]

Your code looks correct, it looks basically like mine, which was copied from the QR code Samples. I have not seen this behavior myself, but there are reasons I might not notice it.

I will need to do some tests, and then ask around. I am curious whether the difference is in the matrix returned by TryGetTransform or in the decomposition.

I'm afraid I am unfamiliar with the internals of that code, but will see what I can find out.

[zantiu]

Maybe because there's a situation that creates a singularity, and/or two solutions. For example here there's some discussion about a 3x3 matrix decomposition and some exceptions: https://nghiaho.com/?page_id=846

Things that could cause it:

• scanning the barcode from straight ahead (should be a common situation, so probably not this)
• having the barcode in the XY-plane

The last one will not be common, because the coordinate system in a normal project will be random, and maybe even continuously shifting. However with WorldLockingTools we have a fixed coordinate system, where having the barcode in the XY-plane becomes a high probability.

But still it makes no sense. Edge cases should happen at exact number, for example 90.00 or 180.00 degrees. The moment you have 90.01 or 180.01 any possible edge cases should be gone.

I'll try do some more tests later with other positions.

[fast-slow-still]

Agreed on all your analysis.

Again, I don't think I have seen that behavior, and I certainly don't see it 50% of the time. Perhaps you could share your QR code with instructions for printing (i.e. size) and layout (I assume on a wall, indicate which side up?). It might help me repro the issue, which could either help me resolve it or channel it to the right team.

[zantiu]

I put the files here: https://gofile.io/d/661rAi Or I can also upload somewhere else if you prefer.

Print the qr-codes on A4 and position them on a wall with the coordinates described in the layout. Markers 1 and 2 should be connected to spacepins with the relevant coordinates.

Then keep scanning marker 10 and log the rotations. Maybe you need to scan from different directions to see the effect.

And I just tried to scan some other qrcodes in different positions and rotations. That was even weirder: they all consistently had a 180 degree offset on the y and z rotation axis. Each marker sampled for 20 times.

[zantiu]

Giving it some more thought: this is maybe the issue:

CoordinateSystem.TryGetTransformTo(rootSpatialCoordinateSystem)

Because calculation of a transform to a position/rotation probably has more than one solution.

Let's take a simple example of one rotation, and you want to calculate the transform of 30deg to 0deg. That give two solutions: -30 deg and 150 deg.

So if you start from 30 degrees and apply any of these transforms then you'll arrive at 0deg.

Now if you look at the QR Code sample: https://github.com/chgatla-microsoft/QRTracking/blob/master/SampleQRCodes/Assets/Scripts/SpatialGraphCoordinateSystem.cs

Here the resulting transform is also used to calculate a new position of an object:

gameObject.transform.SetPositionAndRotation(pose.position, pose.rotation);

But in the WLT example the transform is used as a pose, which might be the correct one, or not...

[fast-slow-still]

Interesting, what format of QR code is that? I haven't seen it before. Is it some variant of Micro QR? Or a new type I haven't heard of yet?

[zantiu]

MicroQRM1. Officially supported by HL2.

[fast-slow-still]

Let's take a simple example of one rotation, and you want to calculate the transform of 30deg to 0deg. That give two solutions: -30 deg and 150 deg.

So if you start from 30 degrees and apply any of these transforms then you'll arrive at 0deg.

I don't follow that. Don't you arrive at 0 with -30deg and 180 with 150deg?

CoordinateSystem.TryGetTransformTo(rootSpatialCoordinateSystem)

Because calculation of a transform to a position/rotation probably has more than one solution.

I'm afraid I don't follow that either. TryGetTransformTo returns a matrix, doesn't it? I don't believe that's underspecified.

[zantiu]

I don't follow that. Don't you arrive at 0 with -30deg and 180 with 150deg?

yes right I mean -30 and 330 deg lead to the same solution.

I'm afraid I don't follow that either. TryGetTransformTo returns a matrix, doesn't it? I don't believe that's underspecified.

Yes: it returns a transform matrix, not a pose. The mathematics behind it probably have multiple solutions but the implementation of the method only returns one solution. No matter which solution it will lead to the same result when the transform is applied. But in the WLT example the transform is interpreted as a pose, which happens to be sometimes correct. At least that's my theory ...

[fast-slow-still]

Ah, thank you, I understand now.

I would understand if the rotations returned wound up approximately the same place, like (180,0,0) == (0,180,180), but the differences you are seeing (if I am doing the math correctly in my head which is unlikely this time on a Friday) would seem to flip the QR surface normal into the page and flip it vertically, leaving only the width dimension unchanged.

Again, I haven't seen that. It is possible it is a bug with scanning of that type of QR code. I will investigate further this weekend.

But I don't believe that the problem is the way they are being used. The rotation is not equivalent to itself flipped about its width, as is visible from the symmetry of the microqrm1 code samples you sent.

I'll let you know what I find (or if I find nothing).

[zantiu]

Some more data from a different orientation (each time scanning the same code in the same position):

• 6.027079, 21.48151, 0.3555717
• 5.149407, 21.06489, 0.2040969
• 354.2695, 166.2822, 181.9895
• 354.2703, 166.2823, 181.9883
• 354.1355, 168.8752, 181.6687

So it's not related to being almost aligned with one axis as I thought before.

But I don't believe that the problem is the way they are being used. The rotation is not equivalent to itself flipped about its width, as is visible from the symmetry of the microqrm1 code samples you sent.

The sample I sent show how to position the markers to reproduce the issue. It doesn't show how the flip happens. In fact I do visualize the flip and instead of having a square on the marker, it for sometimes floats just on top of the marker which corresponds to two rotations of 180deg around two axis

[fast-slow-still]

Yes, exactly. I don't think that the two position/rotaton pairs (that's a Pose, right?), one which displays the marker visualization aligned with the physical marker, and the other which displays the visualization flipped, can be considered equivalent. The code returning those two interchangeably is incorrect. And that is probably the TryGetTransform, although it might be the decomposition.

[fast-slow-still]

I have been trying to repro what you are seeing, but have not been able to. This has led me to some more questions:

1. Where in the capture of the QR scan's rotation are you seeing the anomalous 180degree rotations? 1.a Is that in the function that you posted, or after more processing? 1.b Is that before or after applying the camera parent's transform?
2. The raw transformation returned by TryGetTransformTo is in what we call Spongy space, that is the raw unstabilized space. You want the pose in Frozen space, that is the WLT stabilized Unity global space. Have you considered using WorldLockingManager.GetInstance().FrozenFromSpongy.Multiply(spongyPose) instead of the camera parent?
3. Do you have a simple shareable test scene that shows the behavior?
4. Have you looked at QRSpatialCoord.cs in QRSpacePins project?

[zantiu]

Where in the capture of the QR scan's rotation are you seeing the anomalous 180degree rotations? 1.a Is that in the function that you posted, or after more processing?

I collect the poses returned by the function. After collecting 10 poses I print out the rotations of each pose for a quick compare. There's no further processing done.

1.b Is that before or after applying the camera parent's transform?

I print the poses returned from the GetPoseFromNewQRCode() method above.

Have you looked at QRSpatialCoord.cs in QRSpacePins project?

So I wrote the method in a different way using QRSpatialCoord:

private static Pose GetPoseFromQRCode3(QRCode qrCode, out bool success)
        {
            success = true;

            // make a new helper coordinate system for this qrCode
            var coordinateSystem = new QRSpatialCoord();

            // set the SpatialNodeId in the helper coordinate system
            coordinateSystem.SpatialNodeId = qrCode.SpatialGraphNodeId;

            // try to get the spongyPose
            Pose spongyPose;
            if (!coordinateSystem.ComputePose(out spongyPose))
            {
                success = false;
                return new Pose(); // no success, just return any pose
            }

            // get the frozenPose
            Pose frozenPose = WorldLockingManager.GetInstance().FrozenFromSpongy.Multiply(spongyPose);

            var wltMgr = WorldLockingManager.GetInstance();
            //Pose lockedPose = wltMgr.LockedFromFrozen.Multiply(frozenPose);

            return frozenPose;

        }

But the result is the same.

3. Do you have a simple shareable test scene that shows the behavior?

No it's part of a tightly integrated server-client solution. It would take some time to make a simple project only showing this.

Is it an option to share your repro project instead? It's maybe quicker to test from that direction.

[fast-slow-still]

I would suggest that you examine the pose before and after the camera parent transform is applied. For completeness, you might want to also print out the raw matrix returned from TryGetTransformTo as well.

Again, I haven't managed to repro.

[zantiu]

So I found some time to start from the QRSpacePins sample app and try to reproduce.

What I did:

• Put QR code 1 and 2 in the same positions as my test (1, 0, 0) and (1.65, 10, 0) and all rotations at (0, 0, 0)
• Ignore QR code 3 and 4 (don't print them)
• Add QR code 10 right between 1 and 2 and print the rotations

I could not reproduce the major swings between 0 and 180 degrees. So I need to figure out the difference between both apps.

However the resulting rotations are now always (~0, ~180, ~180) I don't understand why, I would expect (~0, ~0, ~0) ...

[zantiu]

I did another test: Same as above, except hide QR code 2 and 10 from view. So a test on only QR code 1.

This is what was logged:

"ROTATION: 356.7958, 332.5505, 179.8412" "ROTATION: 359.9156, 0.1240982, 359.9925" "ROTATION: 359.8754, 0.547743, 359.9479" "ROTATION: 359.9176, 0.7982327, 359.9043" "ROTATION: 359.9243, 1.160291, 359.8937" "ROTATION: 359.3513, 3.52732, 0.3978981" "ROTATION: 359.3623, 3.528701, 0.4297442" "ROTATION: 359.7415, 5.721841, 0.2642708" "ROTATION: 0.1230932, 0.4597737, 359.9537" "ROTATION: 0.02560534, 0.5731812, 359.9371" "ROTATION: 359.9034, 1.034507, 359.9039" "ROTATION: 359.9106, 1.033908, 359.898" "ROTATION: 359.4315, 1.635164, 359.851" ...

However the green square was at all times displayed correctly on the qr code. And that was not the case with the previous test.

I can upload the code if you want.

The changes are only in QRSpacePinGroup.cs:

        /// <summary>
        /// Process a newly added QR code.
        /// </summary>
        /// <param name="qrCode">The qr code to process.</param>
        private void OnQRCodeAdded(QRCode qrCode)
        {
            SimpleConsole.AddLine(trace, $"OnAdded {qrCode.Data}, enumerated {enumerationFinished}");
            if (enumerationFinished)
            {
                DebugQrCode(qrCode);
                int idx = ExtractIndex(qrCode);
                if (!QRCodeIndexValid(idx))
                {
                    return;
                }
                spacePins[idx].Update(qrCode);
            }
        }

        /// <summary>
        /// Process a newly updated QR code.
        /// </summary>
        /// <param name="qrCode">The qr code to process.</param>
        private void OnQRCodeUpdated(QRCode qrCode)
        {
            SimpleConsole.AddLine(trace, $"OnAdded {qrCode.Data}, enumerated {enumerationFinished}");
            if (enumerationFinished)
            {
                DebugQrCode(qrCode);
                int idx = ExtractIndex(qrCode);
                if (!QRCodeIndexValid(idx))
                {
                    return;
                }
                spacePins[idx].Update(qrCode);
            }
        }

        // test
        private void DebugQrCode(QRCode qrCode)
        {
            bool success;
            Pose frozenPose = GetPoseFromQRCode3(qrCode, out success);
            SimpleConsole.AddLine(trace, $"POSITION: {frozenPose.position.x}, {frozenPose.position.y}, {frozenPose.position.z}");
            SimpleConsole.AddLine(trace, $"ROTATION: {frozenPose.rotation.eulerAngles.x}, {frozenPose.rotation.eulerAngles.y}, {frozenPose.rotation.eulerAngles.z}");
        }

        // test
        private static Pose GetPoseFromQRCode3(QRCode qrCode, out bool success)
        {
            success = true;

            // make a new helper coordinate system for this qrCode
            var coordinateSystem = new QRSpatialCoord();

            // set the SpatialNodeId in the helper coordinate system
            coordinateSystem.SpatialNodeId = qrCode.SpatialGraphNodeId;

            // try to get the spongyPose
            Pose spongyPose;
            if (!coordinateSystem.ComputePose(out spongyPose))
            {
                success = false;
                return new Pose(); // no success, just return any pose
            }

            // get the frozenPose
            Pose frozenPose = WorldLockingManager.GetInstance().FrozenFromSpongy.Multiply(spongyPose);

            var wltMgr = WorldLockingManager.GetInstance();
            //Pose lockedPose = wltMgr.LockedFromFrozen.Multiply(frozenPose);

            return frozenPose;

        }

[zantiu]

And some more logging comparing the Frozen and Spongy poses:

"SPONGY ROTATION: 357.5991, 356.8183, 180.3161", "FROZEN ROTATION: 357.5991, 356.8183, 180.3161",

"SPONGY ROTATION: 357.4779, 356.4985, 180.3077", "FROZEN ROTATION: 0.09881838, 0.3281521, 359.9731",

"SPONGY ROTATION: 357.4779, 356.4985, 180.3077", "FROZEN ROTATION: 0.09881838, 0.3281521, 359.9731",

...

Why is spongy rotation almost aligned with the frozen? Maybe because I accidentally start up facing the wall at 90 deg?

So the first measurement has frozen and spongy aligned, but offset 180 deg around z axis from where it should be.

Then from 2nd and further measurements the frozen rotation becomes 0, so actually the last test is correct behaviour. The first scan is used to align the frozen coordinates.

[zantiu]

I found a problem in the example that you should be able to reproduce:

• set up qr #1 at (1, 0, 0)
• set up qr #2 at (1.65, 0, 0)
• hide qr #2 temporary so it cannot be scanned (put something over it)
• scan qr #1

Result: as above: first scan will follow spongy pose rotations, then all succeeding scans will correctly have rotation (~0, ~0, ~0)

• next unhide qr #2 and rescan both qr #1 and qr #2

Result: First scan will correctly have rotation (~0, ~0, ~0) Then all resulting scans will have rotation (~0, ~180, ~180) (both on qr#1 and qr#2)

During all the scans the spongy rotations stay stable.

So something happens during the alignment of the frozen coordinate system on more than one anchor that offsets the succeeding calculations to have an offset of 180 deg on both Y and X axis.

Since spongyPose is stable something seems to be wrong inside the data used to calculate FrozenFromSpongy once more than one spacepin is attached.