Moving space pins after scaling results in constant offset

[demcdonald2000]

Some background

I am an undergraduate research assistant working on an MR bridge inspection project. I am developing an application for the HoloLens 2 to perform bridge inspections. The application would overlay a virtual model (a digital twin, if you will) accurately over the physical bridge. This is the main feature I have been working on, as accurate alignment of such as large model has proved to be a challenge. The application would also allow a normal bridge inspection to take place, with all inspection data (condition of each component, photos, measurements, etc.) being recorded and linked via the virtual model. This model would then be able to be analyzed at a smaller scale( say, on an office table) and with the linked inspection data visible in some form or fashion.

Before starting this project in November 2021, I had little to no previous experience in MR, Unity (including WLT and MRTK), C#, and developing in the Windows environment. I am majoring in Computer and Electrical Engineering. I take classes and work (<15 hrs/wk), so things are slow going. I appreciate any feedback and help filling the gaps in my knowledge.

My Issue

I am trying to use space pins to align a bridge model at variable scale levels that can be changed at runtime. I have a bridge model and space pins under the same Unity hierarchy so that when the bridge model is scaled, the pre-placed space pins will scale with it and stay put relative to the model features. The issue I am having follows this general procedure:

1. I load up my Unity-created application in the HoloLens 2. My bridge model and corresponding space pins are loaded in at around a couple meters in length.
2. I manipulate the space pins (before any manual scaling) and they accurately are placed. Everything works as it should (as far as my understanding goes) and model placement shifts to reflect my pin placement as I move around.
3. I scale the model up (including the space pins).
4. With the now larger model, I grab a space pin and attempt to reposition it. I am able to move it to a location and it accurately tracks my grab. When I release the space pin to end manipulation, it appears to jump a certain distance away from where I released it (I think proportional to how much I scaled up).

It seems to me like WLT is registering my new adjustment of space pins differently after I scale them up. To be clear, I have tested this inside Unity and the transform position of the space pins is not moving but rather the adjustment object containing the main camera. As far as I understand, this is the correct behavior and the fundamental way WLT keeps objects aligned.

I did some testing, and simply grabbing the space pin and releasing it (not moving it at all, or at least a negligible amount) still results in the space pin "jumping" some constant offset. Also, it seems to offset the space pin a constant amount no matter how much I move it. Simply grabbing and releasing the space pin results in the same offset as if I moved it 1 meters.

The tools I am using are:

• Unity 2020.3.30f1
• MRTK v2.7.3 (for all packages)
• WLT v1.5.8 (Core and Samples)
• HoloLens 2

Additional Notes

• I am using the 'SpacePinOrientableManipulation.cs' from the example namespace. I plan to change this in the future but have been using it for simplicity's sake.
• I have tried moving the space pins into their own parent object and having a script update their scale to match the scale of the bridge model. The same behavior resulted.
• The space pins work at any scale as long as that scale is set in Unity before the project is built or tested. A potential strategy to avoid this issue could be just having two bridge models at different scales and/or only activating the space pins when the model is at a 1:1 scale. This would be possible but could create issues down the road. I haven't been able to test such as setup yet, but am confident it could work.
• I can submit photos and files as needed, but wasn't sure what would be most helpful. I am working out of a private GitHub repo right now, but could make it public or share the project files if that would be helpful.

Thank you!!

[fast-slow-still]

Thank you for a most excellently detailed report. This is an area in which I am very interested. In particular, scale in the transform chain has been under-tested.

Sadly, as happy as I am to receive your excellent issue report, the timing is a tad off, as I am about to take some time off. This will be top of my agenda when I am back to work.

Many thanks again for the excellent issue details!

[demcdonald2000]

Thank you for your fast reply! I'm glad the detail wasn't too much.

I will be off all next week on spring break and won't be working anyways, so the timing is fortunate. I'll look forward to discussing this issue with you when you come back to work!

[fast-slow-still]

I've made some progress here. I was able to repro the issue.

So far, the root cause is eluding me.

[demcdonald2000]

I haven't had much time to focus on this issue lately, but I was able to do some thinking and came to the same realization. I hadn't had time to test it yet, so I'm glad to hear it worked for you.

When I get time, I will look into the AlignmentControl.cs script and see if I can apply the space pin reset everytime I resize or reposition my model. It shouldn't take too much work. I'll let you know how it goes.

---

From: Mark Finch @.> Sent: Friday, April 22, 2022, 10:10 AM To: microsoft/MixedReality-WorldLockingTools-Unity @.> Cc: Daniel McDonald @.>; Author @.> Subject: Re: [microsoft/MixedReality-WorldLockingTools-Unity] Moving space pins after scaling results in constant offset (Issue #275)

I've made some progress here. I was able to repro the issue, and even fix it (or workaround it, I'm not sure yet).

After scaling the scene up or down, resetting the space pins makes subsequent space pin manipulations behave correctly.

You can find sample code for resetting the space pins in the AlignmentControl.cs script in the WLT Examples/Scripts.

It makes sense, after rescaling the space pins, their old poses are meaningless, and the system needs to be notified to discard them. But I want to make sure I'm fully understanding what's going on.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/MixedReality-WorldLockingTools-Unity/issues/275#issuecomment-1106610309, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AULF2IESQARMMYDOEC6VWCTVGK6P7ANCNFSM5RU5FJOA. You are receiving this because you authored the thread.Message ID: @.***>

[fast-slow-still]

Hi @demcdonald2000 , I spent some more time on this over the weekend. I'm afraid I was wrong in my initial evaluation.

I was failing to take scale into account in my transform chain caching. So when the scale changes after the modeling pose is captured in Start(), well, as you have seen, things go wrong.

The good news is that I have a fix. It needs a lot of testing but seems safe. I'm confident it will be in the next release.

The even better news is that you can workaround the issue by calling ResetModelingPose() on the SpacePins that get moved because of a rescaling up the scene graph hierarchy. I also suggest calling Reset() on them. So you would collect all of your affected SpacePins, and then iterate over them like so:

foreach( var pin in myPins)
{
    pin.Reset();
    pin.ResetModelingPose();
} 

This same sequence is suitable, but unnecessary, for SpacePins that have been moved by the translation or rotation of a hierarchy.

I'll note that it is also necessary if the LOCAL position or rotation of a SpacePin is changed after Start().

So to summarize:

• SpacePin moved by translating or rotating hierarchy - no action necessary, Reset/ResetModelingPose okay
• SpacePin moved by scaling hierarchy - Reset/ResetModelingPose needed (for now) because of bug.
• SpacePin moved by local transform - Reset/ResetModelingPose will always be needed.

I'll leave this issue open until the fix gets merged in, but I hope that's enough to unblock you.

Thanks again for your detailed report!