Sharing holograms precisely in a large space

[Luckypon]

Hi ! I'm using the Hololens 2 with World Locking Tools (and MRTK, Unity 2019.4.9).

The goal : I’m developing an app where, in a large space (ideally 100m 100m but if it's too big it can be 25m25m), the users may position, create, update different holograms. They need to keep the same position between different sessions and different hololens, users. Theses holograms are saved in a JSON to be loaded for a future session. If I put this JSON to another hololens, the new user will se the same holograms.

To do this, in the app you can scan multiples QR codes to align the space to the physical world. I’m using Space Pins from WLT to align a part of my scene containing my holograms to the markers (like in the scene QRSubScene in https://microsoft.github.io/MixedReality-WorldLockingTools-Samples/Advanced/QRSpacePins/QRSpacePins.html ) It works well, but when I totally close the application and then go back and scan again my QR code, the holograms close to the QR code (less than 1m) are well placed, but the ones which are farther (5m) are not well positioned... I'm saving the position of the hologram relative to the QR code(which is at the root of the sub scene)

I don’t understand why it does this, because in a different test I made (with only WLT, without QR codes or Space Pins), if I create holograms everywhere, close my app, restart it, they are well positioned… but I’m not sure if they will be positioned at the same place for all the users/hololens when I share my JSON..

Do you know a better way to fix this ? Do I need to put a lot of QR codes everywhere (it will be a bit tedious to scan all of them everytime) ? Or do this without any QR codes at all ? The ideal would be to just scan one QR code to position the origin of the scene and then use WLT to precisely position all of the holograms.

Thanks in advance !!

[fast-slow-still]

The first question I have to ask is, why did you model after QRSubScene scene, rather than QRSpacePins scene? Either can be correct, but the QRSubScene involves more complications. I'd like to understand what the motivation is there.

Also, if at any point you think this would be more efficiently handled over email, that is always an option. Just let me know.

[fast-slow-still]

The second question, why do you need to go back and scan the QR code when you restart the application? The persistence should load the scene back the same as it was relative to the physical world as when you shut down the previous session. Is this not the behavior you are seeing?

[fast-slow-still]

Third question, @Luckypon, are you using WLT from a clone or fork of the github repo? Or are you importing the release .unitypackage files? What version?

[fast-slow-still]

By the way, if you have a shareable version that I could try and see what's going on, that would be incredibly enlightening. Doubly so if you have a github fork I could debug. But I understand how that difficult that can be. Just thought I would ask. :D

[Luckypon]

1. At first I did the QRSpacePins scene, but there was a bug when I scanned my first QR code : I had holograms that were following the player (like an info window, or UI buttons) and they moved away or behaved weirdly at that moment, so I switched to the QRSubScene where there was no bug.

2. When I go back to the home button and then click again on my app windows, yes the persistance is working. But I force quit the application to do like if it was another hololens trying the application, trying to load the holograms with the same shared JSON data.

3. I import the release .unitypackage files ! v0.8.6.

4. Sadly, I can't share my work because of confidentiality issue, sorry :( .

[fast-slow-still]

Great answers. No problem on the confidentiality issue, I totally understand. But still had to ask.

That sounds like there may be a problem in the organization of your scene. The relevant part is the hierarchy of your AlignSubtree node. Very important are where your:

0) AlignSubtree node 1) QR SpacePin node 2) objects whose positions are stored in the JSON file. 3) Other WLT related nodes 4) Camera hierachy

Can you share either a screenshot of your hierarchy showing all of those in Unity, or a text description, e.g.:

Scene

• AlignSubtree -- QR SpacePin -- Object 1 -- ... -- Object N
• AdjustmentNode -- Playspace -- main camera

Or any other way that would be convenient for you to share.

BTW, WLT's persistence should start the application, including the QR code driven SpacePin, in the same state after totally closing the app and restarting it as at the end of the last session. To clear the state as if it were a second device, you need to either uninstall the app and redeploy, or manually delete some key save files off the device. It's usually fastest and easiest to just uninstall/redeploy. So there is something very wrong in the behavior you are seeing.

Have you tried adding the AnchorGraphVisual to your scene? It is just for debugging, but can be especially helpful in seeing what's going on with persistence. With persistence enabled, the BLUE frozen space axis should be in the same place in the world after totally quitting and restarting the app. The GREEN spongy space axis will be wherever the head was at startup (native HL behavior).

[Luckypon]

Okay, my hierarchy is :

Then

• game objects that scan the scene, update my JSON, UI,etc
• Root. (with component AlignSubtree. CollectFromTree is uncheck. SaveFileName is Persistence/SpacePins.fwb. At runtime I dynamically add the owned pins that I create from another JSON file.) Located at 0,0,0 -- MarkersParent (empty transform) Located at 0,0,0 -- HologramsParent (empty transform) Located at 0,0,0

Under MarkersParent, there are my instantiated prefab Marker.

• Marker (with a manager component). Located at 0,0,0 -- VirtualMarker/SpacePin (with a SpacePin component). Its location varies. --- Mesh (just a proxy to see the instantiated marker at scan)

Under HologramsParent , there are my instantiated prefab Hologram.

• Hologram (with a manager component and other specific scripts). Located at 0,0,0 -- MainHologramToMove (you can grab it). Its location varies.

In the JSON, I save the localPosition of MainHologramToMove. (because the parent is at 0,0,0)

With the AnchorGraphVisual, what I see is : When I start the app, the blue and green axis are at the same position (my head). If I move a lot, the blue axis will move away from the green. If I hit the home button and click again on the app, nothing as changed. The blue is still a little away. If I close the app and start it again (I see the Unity splashscreen), the blue and green axis are at the same place where my head is. (like in step one)

[fast-slow-still]

Okay, something is wrong there. When you close the app and start it again (see Unity splashscreen), the blue and green axis should be like when you last ran, with blue a little away. This is lower level than the SpacePins.

Can you send a screenshot of your WorldLockingContext settings?

Also, can you send your UnityPlayer.log?

[fast-slow-still]

Also, can you describe your camera hierarchy? The description you gave for the rest of the scene is great!

[fast-slow-still]

Also, note that for your SpacePins, you need to manage its Save/Load yourself. The reason is obvious. If WLT tried to load your AlignSubtree at start, nothing would happen, because you haven't dynamically created your SpacePins yet. So you should create your SpacePins, and add them to your AlignSubtree, then call Load(). It's not an error if there is nothing to load, just nothing will happen.

You don't have to worry about the AutoSave, that will work for your situation.

However, the immediate problem is that you aren't seeing the WLT core AutoSave/AutoLoad working, so that is definitely the first mystery to solve.

[fast-slow-still]

By the way, when you close the app and start it again (see Unity splashscreen), do you see the trail of markers that you left from the last session?

[Luckypon]

To be sure, I reverted the prefab values, but nothing changed. Everytime I launch the app, the trail of markers have disappeared and the blue and green axis are revert ! To be sure, I show on a textmeshpro the result of WorldLockingManager.GetInstance().AutoLoad and it was true.

For the camera : -Adjustment is an empty gameObject. -- MixedRealityPlayspace is an empty gameObject. --- Main Camera (Components : Camera, AudioListener, EventSystem, MixedRealityInputModule, GazeProvider) ---- Below are gameObjects deleted if not in the Unity Editor (they are not important)

UnityPlayer.log

For the space pins, on AlignSubtree, I call AddOwnedPin(), then ClaimPinOwnership(). But I didn't call Load(), I'll try to call it after ClaimPinOwnership().

[Luckypon]

I don't know if it's related, but sometimes when I start again the app I have the following error at each update :

And I tried to compile a WLT sample (AlignSubScene - TwoIndependantSpaces) and it's working well, there is persistance... So there is definitely something wrong with my project

[fast-slow-still]

The AlignSubtree persistence definitely won't work until we get the core system persistence working.

That's a great test you did, trying the TwoIndependentSubspaces for comparison.

I recognize that error. It usually means that a degenerate (garbage) set of SpacePins has been sent to the AlignmentManager. Especially, two SpacePins with the same virtual or frozen space positions. I don't remember if it's same virtual or same frozen, it's one or the other, but not both.

I'm trying to think back to exact conditions leading to that. I'll try to think about that and maybe repro here. I'll post back if I remember something important.

Could you check the existence of the following files?

/LocalState/frozenWorldState.hkw /LocalState/frozenWorldState.hkw.old /LocalState/Persistence/SpacePins.fwb If they aren't there, then the failure is on save. If they are there, then the failure is probably on load. I'll go through your UnityPlayer.log now and see if there's anything interesting. One last thing, I want to correct what I said earlier about the expected behavior with persistence. On restarting the app (see Unity logo), the BLUE axis should be where it was at the end of last session. The GREEN axis should be at the head position when restarting the app. So on restarting the app with persistence enabled, it is very unlikely for the blue and green axes to be in the same place. I think you have already realized this yourself, but I wanted to correct what I said earlier about the axes being same relative to each other at start of new session vs end of old session. Thanks!

[Luckypon]

I have an update ! I checked the files frozenWorldState.hkw, frozenWorldState.hkw.old, Persistence/SpacePins.fwb and they were here. But then I uninstall and re install the app, and the persistance now works ! (I see the trail of markers and the green axis and blue axis are like you said)

[fast-slow-still]

Okay, that is fantastic. My guess is one (or all) of those files got corrupted. I'll need to put more rigorous checks in, and if anything goes wrong on save or load, make a lot of noise in the log file.

Next step, before getting back to the persistence of your SpacePins, is to check behavior when you scan 2 QR codes after restarting.

If you are wondering why 2 QR codes, rather than just 1 for the origin, the reason is that the orientation data coming from a QR scan is not very reliable. So all of your objects in the scene would appear the correct distance from the origin, but depending on the error in rotation when scanning a single QR code, they would be displaced perpendicular to the radial direction, and proportionate to the radial distance. For a meter or so, that wouldn't be much, but at 10 meters, a 5 degree error would be almost a meter displacement.

When you scan 2 QR codes with a decent separation, then we can use the more accurate positions to infer a rotation for the scene. You don't need a lot of QR codes, just at least 2, in order to get the higher accuracy orientation.

[fast-slow-still]

Filed separate issue on better error checking on save/load, #76

[Luckypon]

Ok ! I tried with 2 virtual QR codes (created in my script at runtime). If I scan one of them and then restart my application, I need to scan it again. (the Root gameObject is back at the same location as the Blue axis). If I call Load() on my AlignSubtree after having done ClaimPinOwnership(), I have the error show before.

[fast-slow-still]

Great progress.

I believe that the error you see comes from the SpacePins virtual position (that is, the transform's Pose) being unset and defaulting to (0,0,0).

Your code being custom dynamically generated SpacePins leaves me at a disadvantage in debugging, but I might be able to help.

The restore of the SpacePin happens in a callback from the AlignmentManager's Load(), in the member function SpacePin.RestoreOnLoad(). If you could add logging info to that function, it might shine some light on what is going on.

I am ahead on this machine on an unreleased upcoming version of WLT. Let me drop back to 0.8.6 and I can give more specific details on the logging. Just a few minutes.

[fast-slow-still]

Okay, I would add the Debug.Log line to the SpacePin.RestoreOnLoad function in SpacePin.cs as follows:

        protected virtual void RestoreOnLoad()
        {
            CheckDependencies();

            AnchorId = AlignmentManager.RestoreAlignmentAnchor(AnchorName, ModelingPoseGlobal);
            if (PinActive)
            {
                Pose restorePose;
                bool found = AlignmentManager.GetAlignmentPose(AnchorId, out restorePose);
                Debug.Assert(found);
                lockedPose = restorePose;
            }
            Debug.Log($"PinActive={PinActive}: n={AnchorName}, mpg={ModelingPoseGlobal}, lp={lockedPose}");
            CheckAttachment();
        }

capturing those four critical values. Then your UnityPlayer.log file should tell us what state got loaded and fed into the AlignmentManager.

[Luckypon]

(Thank you for your reactivity) Ok, I see that lp is different the second time ! The logs (I removed no required infos) : The first time, when there are no errors : UnityPlayer(1).log The second time, when there is the error : UnityPlayer(2).log

[fast-slow-still]

Okay, I definitely see one issue. The SpacePins need to have unique names, but yours are all named "VirtualMarker/SpacePinSpacePin". The second "SpacePin" in the name is internally appended. But your SpacePin GameObjects' all appear to have the name "VirtualMarker/SpacePin". When you instantiate your SpacePin prefab, can you assign a unique name? Append an autoincremented index, or random string, or the time, it doesn't matter, just give them unique names. This is necessary both for the regular runtime and for the persistence.

You should probably delete those save files before retrying with unique names.

Let me know how that proceeds, this feels like progress!

[Luckypon]

There is progress !! Now, my space pins have an unique id so there is no more errors. Plus, there is persistance, my Root gameobject is correctly aligned between sessions ! Thank you, it's better :)

The problem now is : I have a QR code which is very near the root of the scene (like 0.1f, 0, 0). I scan it, then I create new holograms at a precise place a few meters away. If I move like ten meters away from all of my holograms and go back, they are still at the same position (which is great). But when I totally close the application and then go back, the holograms close to the QR code (almost less than 1m) are well placed, but the ones which are farther (5m) are not well positioned... (I'm saving the position of the hologram relative to the Root gameObject).

Is it normal ?

[fast-slow-still]

Are the ones which are farther away systematically displaced? Are the correct distance but offset along the circle centered at the origin?

Or can you characterize the error in some other way?

[fast-slow-still]

Also, when you restart the application, are the placed objects put back in the AlignSubtree hierarchy (i.e. attached to HologramsParent)?

Also, how many QR codes are you scanning in your setup phase? And what are their relative locations to each other, and to the problem objects 5m away?

[fast-slow-still]

Oh, but to answer your question, that's not normal or expected. It sounds like there is still a minor issue.

[Luckypon]

Ok ! For now, I only scan 1QR code. I'm going to try to scan 2 of them (I just need to mesurate precisely their relative distance in the physical world).. The distance seems correct, but they are often (not always) right positioned on the Y axis on the next start. All of my holograms are located under HologramsParent. When I close my app, I save their local position relative to the Root gameObject. And when I open my app, I set their local position.

[fast-slow-still]

That is odd. You will want the 2 QR codes scanned for better accuracy across two devices, but you are correct: On a single device, even with just 1 QR code, your objects should show up the same place as last session.

Could you share your JSON file? I would like to look at the storage format, and see if anything strikes me. If the entire JSON file is too sensitive, then if you could cut and paste the Pose data for one object, that would be helpful.

[Luckypon]

It's something like : marker : {"Items":[{"Id":"AQRCode_1","Position":{"x":0,"y":0,"z":0},"Rotation":{"x":1.0,"y":0.0,"z":0.0,"w":-4.371138828673793e-8}}]}

Hologram : {"Items":[{"Id":"id1","Type":0,"Position":{"x":0.27584877610206606,"y":-0.0554458424448967,"z":-0.15838003158569337},"Rotation":{"x":0.0,"y":1.0,"z":0.0,"w":0.0},"Scale":{"x":1.0,"y":1.0,"z":1.0}},{"Id":"id2","Type":1,"Position":{"x":-0.27000001072883608,"y":0.0,"z":-0.2800000011920929},"Rotation":{"x":0.0,"y":1.0,"z":0.0,"w":0.0},"Scale":{"x":1.0,"y":1.0,"z":1.0}}

[fast-slow-still]

One more test, when you get a chance. If you could log the placed object's position in Unity global space, on the first run where it is behaving correctly when you walk 10m and back, and then on the second run when it is not in the correct position. I'm thinking here of the object away from the origin, but of course it would be great to have all objects, both those staying in correct position and those displaced.

Obviously, the global position should be the same for both runs, even though the objects 5m away from origin are displaying in offset in the physical world.

Also, to clarify, on the second run, when the 5m object is in the wrong place, then on startup you are seeing the marker trail from the previous session?

[Luckypon]

Okay ! To be sure, Unity global space is juste transform.position ?

[fast-slow-still]

Actually, while you are doing that, if you could record the "locked" position too, that would be great. Locked pose is just:

WorldLockingManager.GetInstance().LockedFromFrozen.Multiply(transform.GetGlobalPose())

[Luckypon]

Ok I've done my tests : Under HologramsParents, I've instantiated Holograms (from my JSON), and dummy fixed holograms (from my scene). They have the same "drift".

In all of my sessions : Locked LocalPosition is always equals to LocalPosition Locked WorldPosition is always equals to WorldPosition

LocalPosition is always the same. But, WorldPosition (and Locked WorldPosition) changes at every session. A few centimeters every time. And I don't think it's related to where I started the app (I moved 5-10 meters away from the previous start position everytime)

Another test I've done is that I instantiated an hologram on the blue axis, and everytime I restart, it moves a little away (a few cm) on different direction

[fast-slow-still]

Okay, that is clearly wrong. That says that your Root node is moving every time you start a session. If I remember your hierarchy correctly the Root node (which has the AlignSubtree component?), is the only non-identity transform between your objects and the scene level. So the Root node's transform is the only difference between LocalPosition and WorldPosition.

When you store your marker position, what space is that in?

[fast-slow-still]

Do you still have the Debug.Log you added to SpacePin.RestoreOnLoad? Could you send two consecutive UnityPlayer.log files? I want to compare the values printed there. They should be the same, of course, but I suspect they are not.

[Luckypon]

I've changed the base location of my QR code (in the physical and virtual world). But between two sessions, I have :

PinActive=True: n=VirtualMarker/SpacePinAQRCode_1SpacePin, mpg=((0.0, 0.0, 0.0), (1.0, 0.0, 0.0, 0.0)), lp=((0.9, 0.5, -2.4), (0.9, 0.0, -0.5, 0.0)) PinActive=True: n=VirtualMarker/SpacePinAQRCode_1SpacePin, mpg=((0.0, 0.0, 0.0), (1.0, 0.0, 0.0, 0.0)), lp=((0.9, 0.4, -2.4), (0.9, 0.0, -0.5, 0.0))

You can see that lp has changed

[fast-slow-still]

It might also be helpful to log the local pose of the Root (AlignSubtree) node, to verify that it is really responsible for the shift.

[Luckypon]

When I store my marker data, I store : spacePin.transform.localPosition and spacePin.transform.localRotation

[Luckypon]

When I set a marker from the JSON :

_spacePin.transform.SetGlobalPose(_positionMarkerHelper.GetGlobalPose());
_spacePin.ResetModelingPose();

and _positionMarkerHelper is a empty transform set under Root:

Vector3 position= data.Position; Quaternion rotation = data.Rotation; _positionMarkerHelper.MoveLocalPose(position, rotation);

The marker is loaded but not yet scanned

When I scan my marker, I do : _coordinateSystem.SpatialNodeId = qrCode.SpatialGraphNodeId; _sizeMeters = qrCode.PhysicalSideLength; if (!_coordinateSystem.ComputePose(out Pose spongyPose)) { frozenPose = new Pose(); return false; } frozenPose = WorldLockingManager.GetInstance().FrozenFromSpongy.Multiply(spongyPose); return true;

and if it's true :

Pose lockedPose = WorldLockingManager.GetInstance().LockedFromFrozen.Multiply(frozenPose); if (NeedCommit(lockedPose)) { _spacePin.SetFrozenPose(frozenPose); ShowHighlightProxy(true); // just a visual helper _isDetectedInWorld = true; _lastLockedPose = lockedPose; _positionMarkerHelper.SetGlobalPose(_spacePin.transform.GetGlobalPose()); // just a visual helper }

[fast-slow-still]

That all looks great.

And so when you create your SpacePin object, you set its localPosition and localRotation to the values stored in JSON?

The output of the two UnityPlayer.log files would be great to verify nothing is going wrong there. As well as the Root node's transform pose. But whenever you get a chance would be great. It feels like we are homing in on the problem.

[Luckypon]

I've changed the base location of my QR code (in the physical and virtual world). But between two sessions, I have :

PinActive=True: n=VirtualMarker/SpacePinAQRCode_1SpacePin, mpg=((0.0, 0.0, 0.0), (1.0, 0.0, 0.0, 0.0)), lp=((0.9, 0.5, -2.4), (0.9, 0.0, -0.5, 0.0)) PinActive=True: n=VirtualMarker/SpacePinAQRCode_1SpacePin, mpg=((0.0, 0.0, 0.0), (1.0, 0.0, 0.0, 0.0)), lp=((0.9, 0.4, -2.4), (0.9, 0.0, -0.5, 0.0))

You can see that lp has changed

I don't know if you have seen this. Or do you want more info on the Player.Log ?

[Luckypon]

I've tested root. LockedLocal, LockedWorld, Local and World have the same values. There is no difference between 2 sessions

But, in 1 session, if I move like 10m away from root an then go back, its values have a little changed. For example, z is -2.42 (meters) and before it was -2.41. And it switches between these values.

[fast-slow-still]

That is very strange, the root moving. Moving 1 cm is a lot. Especially since I don't know where the change would come from at all. Any change in the output of AlignmentManager.ComputePinnedPose() should come from interpolation between pins, but you only have one pin, so should be getting back the identical same pose every frame, no matter where you are.

Likewise the change in locked pose lp from Y=0.5=>Y=0.4. That value is just written to binary and restored from file, it should not change value between sessions. It should have no way to change value between sessions.

I will need to dig deeper. I will try to make a simple app that does the essence of what you are trying to do, and see what behavior I get. I will let you know what I find.

[fast-slow-still]

Also, if you could supply your save file Persistence/SpacePins.fwb, I would like to examine its contents.

Also, after sending it, you should change the name of your SaveFileName on your AlignSubtree node. "Persistence/SpacePins.fwb" is the file name that the global alignment uses. You want your local alignment (AlignSubtree) to use a different file. If you set the filename to empty, it will default to the name of your Root (AlignSubtree) object, with the .fwb extension.

[Luckypon]

Hi, sorry for the delay in my answer. I managed to create another project by removing confidential stuff and share it on GitHub : https://github.com/Luckypon/SharingHolograms There are very simple holograms (simple spheres) on there.

[Luckypon]

Here is my .fwb (with the new project). I forgot to tell you that I already named it differently before my tests, sorry.

RootSpacePins.zip

[fast-slow-still]

I finally managed to get the sample built and deployed. (I had to update Unity to 2019.4.9f1, and my internet connection is very slow.)

What is the best set of steps to repro the behavior that you are seeing? Also, do I need a printed QR code? If so, does it need anything particular embedded? How should it be mounted for scanning?

[Luckypon]

Okay awesome ! So you can open AQRCode_1.png directly on your computer (no need to print but don't move the window so its location stays the same). The marker is at the root of my project. It's virtual local position is at (0,0,0). If you need to change it, you can update ImportExportMarkers.cs

When you start the app, you need to scan the marker to make all the holograms appear. (I've haven't changed this behaviour yet). You will see blue spheres, that you can't manipulate. You can add red spheres . Click on ToggleButton, then select Simple. A red sphere will appear on your right index finger. With your other hand click on add. You can move the sphere where you want. Put on spheres close to the Root and other further away. Close the app and restart it (unity splashscreen). You will see the trail of markers. Scan the QR code (which must not have moved between the sessions) You will see that some of the farther red spheres have moved from their previous location.

Click on ResetScan to reset the scan.

I have a functionnality that is not working. If you scan another QR code after scanning one which is in the database, it will be added to the list of known QR code. But it's not working well and the holograms bug if you do it, so if it happens, click on DefaultMarkers to reset the markers :)

Thanks !

[fast-slow-still]

Question: When the application is restarted, it restores the SpacePin from the previous session, but doesn't restore the placed red spheres. But when I scan the QR code again, then the placed spheres show up. Is there a way to get the spheres from the previous session to show up without re-scanning the QR code?

[fast-slow-still]

I'm having difficulty reproducing the behavior you are seeing. The simplified app seems to be working well for me. I would like to try it without re-scanning the QR code (see above Question).

But when I restart the app, and re-scan the QR code, even though the displayed position changes each scan by a few millimeters, the positions of the placed spheres remains constant (after being restored by re-scanning QR code, see above Question).

I placed one red sphere about 1m from origin, and one about 4m from origin. Spongy origin moved each startup (starting from different place in the room). Both spheres showed same behavior, staying on reference corners of tables.

I tried moving away and coming back, including going upstairs, losing tracking, regaining tracking, and coming back down stairs. But reference spheres remained locked.

This suggests that either I am doing something different, or there is a difference in environment. Does your test environment have enough features to ensure good tracking? One problem(?) with my test environment (which is my home) is that it has a great deal of non-repeating features, so it provides very good tracking quality. Perhaps you could post an MRC or photo of your test space?

For comparison, this is my test space. (Ignore the incorrect display displacement in the MRC. I'm trying to file an issue on that now.) The thing to note is the richness of non-repeating details for the feature tracking.

But that's just a thought, I really don't have a good idea why I would be seeing different behavior. I think key might be getting the spheres to appear without re-scanning the QR code.