Comments on "Using ARMeshManager for Spatial Awareness with MRTK3 on HoloLens 2"

[LocalJoost]

Original article: https://localjoost.github.io/Using-ARMeshManager-for-Spatial-Awareness-with-MRTK3-on-HoloLens-2/

[pkk1113]

Thanks LocalJoost! Your post save my life.👍

[LocalJoost]

You are welcome. Great to hear back from someone who appreciates my work!

[AlmarvDiessen]

Hello LocalJoost I ran into the issue with the XRMeshSubsystem not being recognized in the ARMeshManager.cs and there not being a correct loader. I checked the project settings, tried diffrent setting but to no avail. Do you have any idea why this might be?

[LocalJoost]

I am not quite sure I understand your problem. Have you checked out my sample project? Do you have a repro project I can look at?

[StrPython]

Hi, I downloaded and deployed your project on my Hololens 2 but it crashed every time I opened the app. I followed your instructions in the original article and tried to add ARMeshManager.cs to a new project but it did not generate the meshes (nothing seems to happen). Can you please tell me if you are aware of the issue I am currently facing?

[LocalJoost]

@StrPython what Unity version do you use? I have used this successfully in 2021 versions

[StrPython]

I use Unity version 2021.3.21f1

[StrPython]

I attempted to deploy the MRTK3 sample project that you pointed out in the article, but the world surface was not detected, although I can interact with the 3D object using hand gestures.

[LocalJoost]

@StrPython I seemed to have forgotten to push one setting, the MRTK3 profile, that I just corrected. It's one of those annoying things that gets saved only when you close Unity. But as you can't even compile or deploy without it, I assume you had already found that. However, if I then pull the repro freshly, and deploy it using the Unity version I made it with, it runs and shows the Spatial Map. The Spatial Map also works fine in another app, that is using Unity 2021.3.23f1. You do compile and deploy it for Master?

Also, I Unity I tend to make these settings before generating the C++ app:

Although the first one does not really matter, as Visual Studio decides what it should be

[StrPython]

I have deployed it for release.

[LocalJoost]

@StrPython I always deploy for Master. That seems to be the thing for runtime configurations on HoloLens 2

[StrPython]

Do you have missing prefabs in your project, I see those errors:

[LocalJoost]

@StrPython this is because I did not add a gltf importer. It is not important. It should compile and run.

[StrPython]

I tried your settings and deployed the app for Hololens 2 in Visual Studio for Master. The app detected my hands and showed a far ray, it did not generate meshes. I also used Unity 2021.3.23f1. I'm not sure what I missed.

[LocalJoost]

When I tried it with a newer Unity version (2021.3.31f1) I had the same problem. I am at a different machine now and I don't have the 2021.3.16f1 version available I tried it with. Also, I seem my sample uses quite an outdated MRKT Pre version still. I will update it later, and see if that has any issues with newer versions of Unity

[LocalJoost]

@StrPython I upgraded the app to MRKT3 GA (actually, even a bit beyond that) and Unity 2021.3.23f1. This definitely works.

I pushed the update to the sample.

However, if I learned one thing about HoloLens development is that you can't 'just' upgrade a Unity version and expect everything to work. Even between minor version you can get weird effects or things not working. However, I am convinced the MRKT3 GA/Unity 2021.3.23f1 works. At least for Spatial Mapping ;)

[StrPython]

Do you know what caused the issue?

[StrPython]

I think the problem is that I haven't turned on camera access for the app on the Hololens 2 :v. After turning it on in Settings → Privacy → App Permissions → Camera, it works as expected. I really appreciate your support. Thank you very much.

[ldzung]

Hi, your demo is very useful for me. Btw, I would like to ask you a question about getting the position of handray on Spatial Mesh. Could you share your experience on Spatial Mesh? Thank you!

[LocalJoost]

Does this help? Freshly written. I had the idea, just had to find the time. Sorry it took so long. Getting the hand ray end position with MRTK3 - DotNetByExample - The Next Generation (localjoost.github.io)https://localjoost.github.io/Getting-the-hand-ray-end-position-with-MRTK3/

Joost van Schaik

Lead Unity Developer

Mixed Reality MVPhttps://mvp.microsoft.com/en-us/PublicProfile/4034925

@.***

LinkedInhttps://linkedin.com/in/joostvanschaik Bloghttps://dotnetbyexample.blogspot.com/ Twitterhttps://twitter.com/localjoost

Sent from mail for Windows 11 on my over-the-top-specced HoloLens 2 development beast

"Where we go we don’t need… roads"

From: @.> Sent: 08 November 2023 23:05 To: @.> Cc: Joost van @.>; @.> Subject: Re: [LocalJoost/BlogComments] Comments on "Using ARMeshManager for Spatial Awareness with MRTK3 on HoloLens 2" (Issue #446)

Hi, your demo is very useful for me. Btw, I would like to ask you a question about getting the position of handray on Spatial Mesh. Could you share your experience on Spatial Mesh? Thank you!

— Reply to this email directly, view it on GitHubhttps://github.com/LocalJoost/BlogComments/issues/446#issuecomment-1802752222, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA7QDH2AZ4D7FZCORXAQJJLYDP6ZDAVCNFSM6AAAAAAY2T7XX2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBSG42TEMRSGI. You are receiving this because you authored the thread.Message ID: @.***>

[tonyi22]

Hi and thanks for this cool tutorial! I am trying to recognise tables and platforms somehow to then place a hologram on top of it. Is this the right approach? Do I also need the ARMeshManager? I am really confused right now with MRTK3 and unfortunately there is not much online. Would you have any advices?

[LocalJoost]

Hi @tonyi22 you are welcome. For trying to recognize platforms you might want to have a look at this article https://localjoost.github.io/Using-ARPlaneManager-as-an-alternative-to-Scene-Understanding-with-MRTK3-on-HoloLens-2/

[mikael-bergstrom-ntisthlm]

You're a real life saver! Seems like you're the only one on the web, including Microsoft, actually writing clear documentation for how to do things with MRTK3.

One question: Is there a way to test the ARMeshManager properly in-editor? It works fine on the Hololens 2 for me, but I haven't found a way to properly test mesh generation without the lengthy build-and-run-process. In MRTK2 I could just drop in a 3D model and have the observers use it as their mesh, thereby testing my apps properly without having to use the headset.

I seem to have read somewhere that the AR Foundation Simulated XR Environments should be usable for this kind of thing, but as yet I have been unable to get the ARMeshManager to care about those whatsoever. I can open the XR Environments window and choose an office or something, but it all just goes away when I press Play.

So… any ideas?

[LocalJoost]

Thank you 😊. I try to do my best ;) As a matter of fact, I do have an idea. Make the planes appear in a certain layer. Add to your scene a quad in the same layer that only appears when you are in the editor. Thus it can be a stand-in for the plane you need to test against . This is more or less my standard approach to things like this. I usually add Quad for the 'floor'. The fun thing is, this not only works for the editor, but can also be extended to use in head sets that don't support spatial maps, like the Quest 2

[DimitryD]

Hi @LocalJoost I followed your example to have Spatial Awareness on MRTK3 and Hololens 2. But I also want to go further and calculate distance from the hololens to the mesh and somehow struggle with it. Below is code sample I use. For some reason it always goes into else section -> there is no hit with the mesh. Maybe you have an idea what could be wrong here?

Btw, hand rays end up on the mesh -> there is a hit between hand rays and the mesh.

Ray ray = raycastCamera.ScreenPointToRay(raycastPoint);
LayerMask layerMask  = 31;

RaycastHit hit;
if (Physics.Raycast(ray, out hit, 15.0f, layerMask)) // Assuming a max distance of 15.0f for the ray
{
    // Draw the ray to the hit point
    lineRenderer.SetPosition(0, ray.origin);
    lineRenderer.SetPosition(1, hit.point);
}
else 
{
    // Draw the ray up to the maximum distance
    lineRenderer.SetPosition(0, adjustedOrigin);
    lineRenderer.SetPosition(1, adjustedOrigin + ray.direction * 15.0f);
}

[LocalJoost]

@DimitryD don't use the raycastCamera. That's meant for like translating a tap on a screen into a ray. You should cast a ray from themain camera's position in the direction of main camera's forward: Physics.Raycast(Camera.main.transform.position,Camera.main.transform.forward, out hit, 15.0f)

You see, the main camera is always the position of the HoloLens. Good luck!

[DimitryD]

@LocalJoost thank you for your reply! I used your suggestion and now I get hits into game objects I created e.g cubes or spheres, but the rays aimed just into mesh still don't hit it.

[LocalJoost]

@DimitryD I think your layer mask might be wrong. AFAIK it's bit mask. In stead of LayerMask layerMask = 31; try int layerMask = 1 << 31

I hope that works out for you.