Request to make Microsoft.Psi.MixedReality.StereoKitTransforms properties public

[Abdul-Mukit]

Microsoft.Psi.MixedReality.StereoKitTransforms has very useful properties like the

internal static CoordinateSystem StereoKitToWorld internal static CoordinateSystem WorldToStereoKit

Similarly, inside the PsiInput class, the private static CoordinateSystem ToPsi(this Pose pose) is also very useful.

However, due to being private or internal, they can't be used when developing applications.

While using Psi, if I have to render something simple, I currently have to make a pipeline to be able to use the StereoKitRenderer. (if we have done the MixedReality.InitializeAsync() ) For instance, a simple UI menu that I want to display in front of the user's head. Making that UI inherit from StereoKitRenderer means that I have to pass in a pipeline for the contractor. That feels unnecessary.

If I didn't use Psi and didn't initialize the world coordinate system, I could have used StereKit.Input.Head.Forward and the Quat.LookAt() to set the menu directly facing the user with zero roll. But when I do use psi and don't use the StereoKitRenderer, I will have to calculate the StereoKit.Input.Head.Forward (vec3) myself from the PsiInput.Head. That is because PsiInput class doesn't contain the Head.Forward property. Moreover, the PsiInput.ToPsi() is also private funciton. Similarly, I can't implement a local function either, as StereoKitTransforms.StereoKitToWorld is internal.

private static CoordinateSystem ToPsi(this Pose pose) => pose.ToCoordinateSystem().TransformBy(StereoKitTransforms.StereoKitToWorld);

So I can't find an easy way to calculate custom poses with respect to the MathNet's mixed-reatliy world coordinate system. The useful functions are either private or internal. If the PsiInput.ToPsi() had a vec3/Point3D version and was public I could have used something like StereoKit.Input.Head.Forward.ToPsi().ToVec3()

I currently implement them in my code like this: private Vec3 ToPsi(Vec3 pt) { var poseSK = Matrix.T(pt).ToCoordinateSystem(); var SK2World = StereoKitTransforms.WorldHierarchy.Inverse.ToCoordinateSystem(); var posePsiWorld = poseSK.TransformBy(SK2World); pt = posePsiWorld.ToStereoKitMatrix().Translation; return pt; }

So, if some of these functions or properties could be made public that would be very helpful. Please let me know if I misunderstood something.

[sandrist]

We could consider opening up some of these APIs to be public, but I'd like to understand a bit better what you'd like to achieve. A couple points given what you described:

1. It should still be possible to do pure StereoKit input and rendering, even if you are using Psi as well. Positioning a UI menu using StereoKit.Input.Head.Forward and Quat.LookAt should work as normal if you do not want to use a Psi renderer component. Does this seem to not be true for you?
2. The PsiInput.Head CoordinateSystem uses its XAxis as "Forward". Does that help?

In general, PsiInput simply gets you the StereoKit inputs in the Psi "world". If you would like to compute StereoKit things, you should simply use the StereoKit inputs. There shouldn't be a reason to convert StereoKit -> Psi -> back to StereoKit, unless I'm missing something.

[Abdul-Mukit]

Thank you very much for the comment. That motivated me to test my understanding of StereoKit again.

I did some very simple tests. Only SK.Input-based rendering works with Psi without pushing world hierarchy or inheriting from StereoKitRenderer. I have figured out why we couldn't render things with Input.Head.Forward while using Psi. It has nothing to do with Psi. Our understanding of Input.Head.Forward was incorrect.

Input.Head.Forward (or Pose.Forward) only calculates the direction vector. It doesn't provide the actual location wrt to head position in that forward direction. The direction is always calculated wrt to (0,0,0) and the direction vector's values are always bound between -1 to 1. Checking Input.Head.Forward, I saw that the values always stayed in the -1 to 1 range no matter where I stood in the room. So when we were trying to display a menu using: _menuPose = new Pose(Input.Head.Forward, Quat.LookAt(Input.Head.Forward, Input.Head.position));

It didn't work once we walked away wearing the headset from the initial position where we started the app. The reason it worked if we didn't walk away was that the (0,0,0) of SK basis was initialized at the initial position where the device was. Psi also mentions this in the documentation.

The correct solution is: _menuPose = new Pose(Input.Head.position + Input.Head.Forward, Quat.LookAt(Input.Head.position + Input.Head.Forward, Input.Head.position));

As a result, we don't need the PsiInput.Head or WordHiarchy anymore. We can directly render simple stuff using pure SK even if we had initialized Psi's MixedReality.InitializeAsync().
I tried this out on today and it worked. The following is a simple code snipped that I tried.

internal class Program
    {
        static void Main(string[] args)
        {
            // Initialize StereoKit
            SKSettings settings = new SKSettings
            {
                appName = "StereoKitPsiMenuRendering",
                assetsFolder = "Assets",
            };
            if (!SK.Initialize(settings))
                Environment.Exit(1);

            MixedReality.InitializeAsync().GetAwaiter().GetResult(); // Initialize Psi MixedReality statics

            Matrix floorTransform = Matrix.TS(0, -1.5f, 0, new Vec3(30, 0.1f, 30));
            Material floorMaterial = new Material(Shader.FromFile("floor.hlsl"));
            floorMaterial.Transparency = Transparency.Blend;

            Pose menuPose = new Pose(Input.Head.Forward, Quat.LookAt(Input.Head.Forward, Input.Head.position));

            // Core application loop
            while (SK.Step(() =>
            {
                if (SK.System.displayType == Display.Opaque)
                    Default.MeshCube.Draw(floorMaterial, floorTransform);

                //Lines.AddAxis(new Pose(Input.Head.Forward, Quat.Identity) , 0.2f);  // Uncomment to see the problem
                Lines.AddAxis(new Pose(Input.Head.position + Input.Head.Forward, Quat.Identity) , 0.1f);
                Lines.AddAxis(Pose.Identity, 0.1f);

                UI.WindowBegin("Input.Head.Forward", ref menuPose);
                UI.Label($"Head: {Input.Head.position.ToString()}");
                UI.Label($"Head forward: {Input.Head.Forward.ToString()}");
                UI.WindowEnd();

            })) ;
            SK.Shutdown();
        }
    }

[sandrist]

Great, glad you got it working! I'll mark this issue as closed for now, but feel free to open it back up if you have any follow-up questions.