NearMenu/RadialView/Solvers break in ARM64 Release mode

[ei2kpi-ptc]

Describe the bug

In the NearMenuExampleScene, there seems to be a race condition causing an issue only seen in Release mode ARM64 where the menus disappear from view when they are unpinned.

During investigations, I noticed that the scale of the menu would go to (0,0,0.3) instead of 1,1,1 causing the menu to disappear completely or sometimes, the menu would be stuck at a weird angle for a bit and then disappear completely.

It looks like the issue is related to RadialView since I found it when I was upgrading a project from v2.0.0 to v2.3.3 and my menu bar was never showing up since it had RadialView on it. So, it does not seem related to the NearMenu prefabs in any way. After a ton of debugging, I realized it was happening in the stock MRTK repo itself.

The issue seems to repro only with ARM64, Release and when the app is built in Unity without development mode on. It does not repro when built for ARM, or when built for Debug or when built with Unity Development Mode + Script debugging.

To reproduce

A) Clone the repo, open and build the NearMenuExamples scene for ARM64, Release and deploy to a HL2. Try to unpin one or more of the menus displayed in the scene

B) Create a new project in Unity using the instructions in the Getting Started guide and import the MRTK Foundation and MRTK Examples packages. Then, build for ARM64, Release and deploy to HL2. Try to unpin one or more of the menus displayed in the scene.

Expected behavior

See videos below

Videos

REPRO: Arm64, Release, Vanilla Project with MRTK unitypackages: https://youtu.be/WAaUk4ChLkk REPRO: Arm64, Release, MRTK Repo checked out at v2.3.0: https://youtu.be/tMJBf6YXvLM

NO REPRO: Arm64, Debug, Vanilla Project with MRTK unitypackages: https://youtu.be/XlW5iN6SEgM NO REPRO: Arm64, Debug, MRTK Repo checked out at v2.3.0: https://youtu.be/BoffJwdd0X4

Your setup (please complete the following information)

• Unity Version 2019.3.7f1
• MRTK Version v2.3.0

Target platform (please complete the following information)

• HoloLens 2, ARM64, Release

[keveleigh]

Interesting...the combination of repro scenarios you lay out makes me think there's some race condition somewhere? In slower configurations (ARM vs ARM64, debug vs release, development mode vs not) it doesn't repro, but does in ARM64 + release.

[ei2kpi-ptc]

Agree...

My initial hunch was that it might have something to do with the RadialView Maintain initial scale option but as you can see in the repro videos, sometimes it is not just the scale that gets zeroed out, sometimes it is the orientation that gets locked in a weird way.

Another (probably lower chance) possibility is that some code optimization between release / debug or casting between 64-bit and 32-bit might be causing things to go awry?

[ei2kpi-ptc]

BTW, I upgraded my project from MRTK v2.0.0 to v2.3.0 when this started happening and the RadialView.cs script has very minimal changes between those two versions. So, this is something a bit more complicated I'm afraid.

[keveleigh]

BTW, I upgraded my project from MRTK v2.0.0 to v2.3.0 when this started happening

Did you update Unity versions as well, or just MRTK around that time?

[ei2kpi-ptc]

I did go from Unity 2019.2.13f1 to Unity 2019.3.7f1 as well.

[cefoot]

I can confirm this issue for different menus we have on our app.. For me the simpliest way to reproduce is:

• empty project (Unity 2019.2.15 )
• import Microsoft.MixedReality.Toolkit.Unity.Foundation.2.3.0.unitypackage
• configure Sample scene and project(just using MRTK provided)
• adding "Windows Mixed Reality" under Virtual Reality SDKs (this is another problem ;) )
• adding "simple hand palm menu" only to display a sphere when the palm is tracked (I've marked everything I've changed after adding the Scripts)

When deploying Release|ARM64 the Sphere does not show up (as described above) when using "slower" build settings it does show up.

Hope this helps to track this down! :)

[cefoot]

Sorry, I just noticed that for now the issue was only mentioning radial view but my guess is, that my comment is still valid as this seems to infect more than radial view only.

[flaidxu]

我也碰到这个问题。

[c11epm]

Hi, I think this was what I experienced with my project for Hololens 2. And when having a solver with smoothing enabled (eg. HandMenu, HandConstraintPalmUp) the menu would "disappear" prob. going to scale 0 as described. When I disabled the smoothing the hand menu did not disappear any more and followed the hand as expected, although a little choppy without the smoothing.

Had the Hololens 2 for a couple of weeks to try it out and can't test on it anymore. But the smoothing worked fine in the editor play mode, but not when deployed to the HL2. Built with Master|ARM64 I think...

I also recall that there were some other component that had similar problem, I think it was the surface magnetism as well (the object disappeared when the magnetism was activated), and when disabling the smoothing then it worked as expected again. This can maybe help you a little further.

Cheers,

[cefoot]

yes @c11epm that solves it for me (with different solvers I use) thanks! If I've time later I'll try to track this down!

[tank-t-bird]

@cefoot and @c11epm , I unchecked the" Smoothed Eye Tracking"setting in the Eye Gaze Provider section of the Configuration Profile and that fixed it. Even though I did not change the smoothing setting in my Radial View components.

I think this also happened on a different occasion when I had RadialView components both on the parent AND a child object simultaneusly, but that seemed like a reasonable outcome.

[matware]

I confirmed this in 2019.3.10f1.

This looks like and IL2CPP bug (or god forbid an ARM64 compiler bug) when generating code for Solver.UpdateWorkingScaleToGoal or UpdateWorkingPositionToGoal. The 'GoalPosition' value from the previous call to UpdateWorkingPositionToGoal appears to be reused/overwriting the 'GoalScale' value being passed into the call to the SmoothTo method. This is super strange, the generated IL2CPP code looks as sane as it ever does, and I can't see how it's blatting or re-using values from the previous call.

In my case, the the position update was setting a GoalPositon of (0.0, 0.0, 0.1) which was then flattening out some objects when it was applied to the GoalScale.

The work around suggested above works.

If there is anybody left in Seattle I'd suggest trying to work out what the heck, as SmoothTo is used in a few places and might produce weird behavior elsewhere.

[tank-t-bird]

From the HoloDevelopers Slack mixed-reality-toolkit channel:

[HerrKatzen]

What I found out, that if I build the exact same VS solution from Visual studio 2019 16.4.5 -> hand menu works, Visual studio 2019 16.5.4 -> hand menu dissapears. Unity version 2019.2.21f1, MRTK: 2.2.0

So I think it's more of a visuals studio bug than an MRTK bug.

[matware]

I can confirm I was using VS2019.5.4, well snap ARM64 compiler bug? I'll try and test ARM tomorrow (but I Unity 2019.3 ARM builds were broken last time I checked). edit: ARM is still broken in Unity 2019.3 so can't check.

[srinjoym]

@keveleigh Did we find any more information about this bug? I hit this bug in an app that was upgraded to MRTK 2.3. Created a new scene with just a Slate: It scales to zero as soon as the "Follow me" button is hit. The hand menu solvers also scale to zero.

Building with ARM solved the issue. I had also tried reinstalling MRTK from Nuget and that seemed to fix the issue for some reason...

[keveleigh]

@srinjoym Unfortunately, I haven't had time to really dig in yet.

upgraded to MRTK 2.3

Which version were you upgrading from? And did you happen to change Unity versions at the same time?

I had also tried reinstalling MRTK from Nuget and that seemed to fix the issue for some reason...

Huh...

[david-c-kline]

@matware, for ARM builds with Unity 2019.3, can you try following the steps documented in #7888 (disable Graphics Jobs in the player settings)?

Thanks!

[david-c-kline]

I have yet to be able to repro this with Unity 2018.4.x and Visual Studio 16.5.5 (granted, this was targeting ARM not ARM64). I built both Release and Master flavors.

I will try again using 2019.3.x and ARM64.

[david-c-kline]

I can confirm I was using VS2019.5.4, well snap ARM64 compiler bug?

@matware, can you try installing the 16.5.5 release of VS and let us know if you see the same issue? I would recommend a clean build to ensure that if this is a compiler issue, there is nothing left over :)

[david-c-kline]

I can confirm this with Release | ARM64 on current (16.5.5) Visual Studio

[david-c-kline]

I am also not seeing hand menus when building ARM64 for master.

[tank-t-bird]

@ei2kpi-ptc did you try to unpin every single menu? Because....

Out of curiosity I updated to VS 16.6.0 Preview 6 and followed procedure A as described in the original issue. Meaning that I built the NearMenuExamples scene as Release ARM64 without Development Build checked.

To my surprise, the two leftmost menus behave as expected, that is, I can pin and unpin them and they follow me around when unpinned. The remaining 4 on the right disappear just as described in the issue. 2 out of 6... why? I didn't have time to compare the different Prefabs to each other... Did everyone unpin every single Menu in teh scene?

I also tried starting with unpinning menus in a different order, using the HandRay and the hand touch... same consistent experience.

[ei2kpi-ptc]

I have had them disappear when doing all the menus and only one or a few of them. I've also seen cases where it doesn't disappear completely but just gets placed in a very weird position (I think the first video I posted has an example of that)

[tank-t-bird]

@ei2kpi-ptc Got it, very strange! I can also confirm that in my own scene, a HandMenu attached to the palm only appears with Developer Mode enabled.

[waynebaby]

same here not working in 2019.2.21f1 Arm64 i have not tried arm yet. trying now

yes this is totally happening in ARM64 and ARM is bug free,

You can just check with the Solver Example scene. it have the same bug. the coffee cup is disappear in 3 seconds and leaving a phantom in somewhere.
20200520_180941_HoloLens.zip

[david-c-kline]

Thanks for all of the additional input folks! This is super helpful as we try to get to the bottom of this issue.

[brookman]

Just in case: For me it helped to disable "Smoothing" in the Hand menu (#7931 ) as well as SurfaceMagnetismSpatialAwarenessExample (#8003 ) to get both things working with ARM64 and Release:

[FreakTheMighty]

Oooph, our team hit this like a 🧱 wall. After updating VS and building to release, our menu vanished. As it was release build, it was very difficult to track down the root cause.

We found then menu was present in our CI builds and build from my colleague. My colleague was running Visual Studio 16.4.6 I am running 16.6.0.

It finally occurred to me that it could be an issue with the solver which led me to this issue and the "Smoothing" fix.

[XPav]

2019.4.1 LTS has the following "improvement": Physics: Make the MeshCollider fast midphase work on all platforms. (1213433)

[FreakTheMighty]

@XPav do you think that error is related? I have that physics error with working menus using an earlier version of Visual Studio.

[jasonhbartlett]

Just hit this myself. Man it's killer. Brand new project in Unity 2019.3.15f1 and MRTK2.4.0 and Release and ARM64. So is the best workaround to just disable smoothing in the inspector for all solvers in the scene?

[FreakTheMighty]

@jasonhbartlett I think the best workaround is to downgrade visual studio. See my comment above for the version that works for my team.

Disabling smoothing looks pretty bad, so not a great work around in my opinion.

[zantiu]

Same problem here: Visual Studio 16.6.2 Unity 2019.3.15f1 MRTK2.4.0 Release ARM64

Solved by disabling smoothing

[wiwei]

FWIW this is something we've been looking at - this looks like it's probably an ARM64 compiler bug.

We haven't figured out the min repro for hitting this (i.e. we'd need to come up with a min repro in order to the file the compiler bug)

[tank-t-bird]

@jasonhbartlett I think the best workaround is to downgrade visual studio. See my comment above for the version that works for my team.

Disabling smoothing looks pretty bad, so not a great work around in my opinion.

I believe you can alternatively leave smoothing on and instead check the box "developer mode"and build in Release. Not 100% sure but I think this is what I did later on.

[Silloty1]

Hi This bug wasn't supposed to be fixed on the MRTK 2.4 ? I am still having the issue when releasing in ARM64, the HandConstraintPalmUp.cs with Smoothing activated doesn't work at all. When I am removing the smoothing, it works, but not very well

[BewaAutomatisierung-RD]

Same problem with the following toolset: Visual Studio 16.6.4 Unity 2019.4.3f1 MRTK2.4.0 Release ARM64

With smoothing disabled it works. I wasted several hours because of this bug before stumbling over this bug-report.

[Magiel1975]

Same here. I can only see a white dot where the Handmenu supposed to be, moving along with my hand. The dot is in the middle of where the handmenu previously was and this dot even can jump from one hand to the other hand. Putting a finger close to the dot makes the grab sound play. But I cannot do anymore with the this dot.

[torynfarr]

I can reproduce this problem as well:

Visual Studio 16.6.4 Unity 2019.4.1f1 MRTK2.4.0 Release ARM64 (HoloLens 2)

The problem occurs with any near menu the moment the RadialView component is activated (either by tapping the pin button or by auto-follow kicking in). It happens regardless of whether you're using Unity's legacy XR system or the new XR Plugin Management.

[laultman]

I have a brand new HoloLens 2 7/15/20 and fully clean setup of Unity 2019.4.4f1 and VS 2019 16.6.4 with MRTK v2.4.0. Basically nothing works. Trying to work through the getting started and tutorials. Can't get any app to work with either ARM64 or ARM using debug or release. I have completely unloaded Unity and reinstalled. Completely deleted all references and still it will not create a simple out of the box HL2 app. Very frustrating. I can't tell if the documentation is bad or if the software is bad - nothing works to even be able to begin to troubleshoot.

[torynfarr]

I have a brand new HoloLens 2 7/15/20 and fully clean setup of Unity 2019.4.4f1 and VS 2019 16.6.4 with MRTK v2.4.0. Basically nothing works. Trying to work through the getting started and tutorials. Can't get any app to work with either ARM64 or ARM using debug or release. I have completely unloaded Unity and reinstalled. Completely deleted all references and still it will not create a simple out of the box HL2 app. Very frustrating. I can't tell if the documentation is bad or if the software is bad - nothing works to even be able to begin to troubleshoot.

There could be a few things going on there, depending on what you're experiencing. The documentation / software isn't bad, but it's not perfect.

Can't Deploy Your App to HL2?

Make sure you're working with the built Unity project here and that you haven't just opened your C# scripts by double clicking on them in Unity. First you build the project in Unity, then navigate to the folder where you stored the build, and open the solution in Visual Studio. After you open the solution in Visual Studio, make sure to set the "Solution Configuration" dropdown to release and the "Solutions Platforms" dropdown to ARM64.

You'll notice that there are two projects in the solution. One for Windows Store 10 and another labeled Universal Windows. Right click on the Universal Windows project and select Properties. Click on Debugging and make sure the "Debug to Launch" dropdown is set to Remote Machine. In the "Machine Name" field, enter the IP address of your HoloLens 2. At this point, Visual Studio will probably prompt you for a pairing PIN.

Back on your HoloLens 2, go to Settings > Update & Security > For developers and make sure developer mode is turned on. Tap Pair to get the device pairing PIN that Visual Studio is prompting you for.

Finally, in Visual Studio (after entering the pairing PIN), you should be able to build and then deploy to your HoloLens 2.

You can deploy, but the app runs in a window on HL2 or doesn't run at all?

Make sure your Unity project platform is set to UWP and that you've installed at least the Microsoft.MixedReality.Toolkit.Unity.Foundation.2.4.0 package. At this point, you need to make a decision or at least be aware of which Unity XR management system you want to use. The XR management system you're using will dictate what profile you need to set MRTK to use in your scene.

By default, Unity 2019.4 will be set up to use the legacy XR management system. You can tell which one you're using by going to your project settings in Unity and clicking on XR Plugin Management. If you see a button to install the XR Plugin Management, that means you're using the legacy XR system. If you don't see that button, that means you've installed the XR Plugin Management package and are now using the new XR system.

If you're using the legacy XR system, click on MixedRealityToolkit in your scene hierarchy and then go to the Inspector and set the MixedRealityToolkit component on that object to use DefaultHoloLens2ConfigurationProfile.

If you're using the new XR system, you'll need to set that profile to DefaultXRSDKConfigurationProfile. You'll want to clone that profile and clone the camera profile and then change the background under the camera's display settings to black (switch "Clear Flags" to Color and set the background color to black.) You'll also want to go into the package manager and make sure that you have these four packages installed:

• Windows XR Plugin
• XR Plugin Management
• XR Legacy Input Helpers
• XR Interaction Subsystems

Hope that helps!

[RichLogan]

Also ran into this issue on 2.4.0 (specifically when using the new Dialog boxes), and saw from other team members they'd hit issues with solvers. As suggested, disabling smoothing is a workaround for now. ARM64/Release/HL2

[jasonhbartlett]

Just FYI - The issue still occurs in Visual Studio version 16.6.5 - Requires disabling smoothing to get the hand menu to appear.

[wiwei]

This also reproes on the latest 16.7 preview as well.

I spent some time yesterday/today digging into this deeper, and went through a crest of highs and lows.

I ended up filing a VS bug here: https://developercommunity.visualstudio.com/content/problem/1128346/arm64-c-164-165-regression.html

This covers the investigation that I did, along with what I'm seeing in the compiler output that's leading to the issue.

A few things:

1. If you look at the actual generated Lerp function assembly, it seems right - assuming the s0 to s6 registers are properly filled, it returns the right value.
2. The issue is that some of the callers have assembly that is putting the parameters into the totally wrong register - for example, a value that should be going into s6 is getting stuffed into s3 (which means garbage in == garbage out).
3. I tried doing some work here to write our our Lerp function (or do some other arbitrary code restructuring) and unfortunately it was really hit or miss - it actually did help with one or two of the solvers, but ultimately did nothing for others. There was no reliable way that I saw which would actually work around this issue.

Obviously this isn't a problem with Lerp in general (plenty of other places Lerp just fine), so there is really something specific to the way the generated IL2CPP code + ARM64 compiler are playing together.

Note that the same generated IL2CPP code builds and deploys and works fine on 16.4.11, but fails on 16.5+

Unfortunately right now the best thing is just to disable smoothing when building for ARM64 (in solvers that you use) until the ARM64 issue is fixed. There may be some way to restructure the c# code arbitrarily to trigger different IL2CPP, but even that's a crapshoot unfortunately.

[wiwei]

Woops, also forgot to mention, the alternative option is to just build ARM (not 64) with smoothing enabled.

[cre8ivepark]

Verified - using ARM instead of ARM64 resolved the issue with any Solvers - Hand Menu, Radial View, Surface Magnetism, InBetween Solver, etc...

[JimWest]

I had the same problem, I've updated now to the newest Visual Studio version (16.7.1) and everything's working again (using Unity 2019.3.0f6, ARM64 Release Build).

Edit: only works for one app, the other apps which are using the solvers still have that problem.

[Scanlia]

I also have this issue on my HoloLens 2. Visual Studio 16.7.2 Unity 2020.2.1 MRTK 2.4.0 Master, ARM64

[matware]

We're still seeing the issue in 16.7.2. I ended up writing a smoothenator as a monobehavviour that implements the positional and rotation smoothing. I disable Smoothing and enable UpdateLinkedTransform, then just smooth the WorkingPosition and WorkingRotation (same logic as Solver.cs) and update the transform. I think you won't run in the same issue as the built in version of you don't update scale, or use the System.Numerics Lerp and Slerp. As a work around it's the least terrible thing I can think of at the moment.