Add Lynx loader support

[rodolpheh]

This MR adds support for the Lynx loader and the Lynx R-1 headset.

Tested features:

• Head tracking :heavy_check_mark:
• Hand tracking :heavy_check_mark:
• I have some issues with displaying the hand model and OpenGL, need to investigate
• Vulkan kind of works but is upside down and spam the console with errors

This MR is currently missing the LICENSE file. Merging this MR should close #22

[m4gr3d]

Looks good!

Can you update the CHANGES.md file, and it should be good to merge.

[rodolpheh]

I've updated the CHANGES.md, feel free to merge. I am a little bit conflicted regarding the missing LICENSE file. On one hand I suppose it's important, on the other hand I can't find any LICENSE file in Lynx's tooling for Unity.

[m4gr3d]

@rodolpheh Thanks!

@BastiaanOlij @akien-mga Do we have a contact at Lynx we can ping for access to their license?

[BastiaanOlij]

Sigh, is Lynx not using the standard Khronos loader?

edit yeah I'm confused, I was under the impression Lynx based it's implementation on Monado which uses the standard KHR loader, so isn't this a matter of adjusting the KHR option and just adding the missing manifest entries? I don't think its much of a problem to have the extra entries on platforms where it's not used.

Though I guess there is an argument to have separate entries for the different targets even if they use the same loader, it's easier for someone to tick the "Lynx R1" tickbox then to know they need to tick the "Khronos Loader" one.

Though the end goal is to one day is to only have the khronos loader.

[BastiaanOlij]

I've updated the CHANGES.md, feel free to merge. I am a little bit conflicted regarding the missing LICENSE file. On one hand I suppose it's important, on the other hand I can't find any LICENSE file in Lynx's tooling for Unity.

Any chance you can ping Stan Larroque on this? I don't mind nudging him on Twitter but it sounded like you guys have a more personal relationship with him?

[rodolpheh]

I'll get in contact with Lynx to see if a LICENSE file is necessary. Unfortunately it seems that their loader is slightly different from Khronos' loader.

[BastiaanOlij]

I'll get in contact with Lynx to see if a LICENSE file is necessary. Unfortunately it seems that their loader is slightly different from Khronos' loader.

Bah, alas :)

[GeorgeS2019]

@rodolpheh

Have u tried passthrough with Lynx using Godot?

What other features YET-To-Implement for Godot on Lynx?

[rodolpheh]

@BastiaanOlij Oops just received and pushed a LICENSE file and a NOTICE.txt file. Lot of thanks to Lynx for providing it ! Should I open a new PR ?

@GeorgeS2019 I haven't tried passthrough yet but this should definitely be a feature to check out. Hand tracking seems to work but there is rendering issues preventing them from showing. They do show with Vulkan but Vulkan display is upside down. Let me know if you're aware of any feature I should check for.

[m4gr3d]

@BastiaanOlij Oops just received and pushed a LICENSE file and a NOTICE.txt file. Lot of thanks to Lynx for providing it ! Should I open a new PR ?

@GeorgeS2019 I haven't tried passthrough yet but this should definitely be a feature to check out. Hand tracking seems to work but there is rendering issues preventing them from showing. They do show with Vulkan but Vulkan display is upside down. Let me know if you're aware of any feature I should check for.

@rodolpheh yes can you open a new PR.

[GeorgeS2019]

@rodolpheh

Mandy social reviews believe Lynx-R1 has better passthrough then e.g. Quest Pro.

It is definitely better than Pico ( One Camera non-stereo passthrough).

WHY the priority to show Godot supports the Lynx-R1 passthrough with clear RECENT (less than 6 months old) improvement in color calibration and background color video alignment of the Lynx TWO stereo color camera!

[rodolpheh]

Just to close this topic, I haven't tried the passthrough on the Quest Pro but I've tried it on Lynx and the Pico 4. Stereo camera really improves the experience and gives depth perception which is priceless. A simple test I tried which I think is very much telling about the comfort and quality of passthrough is to juggle with the headset on my head, and it is possible to juggle while wearing the Lynx (impossible on Pico 4 due to lack of depth perception).

[rpavlik]

@rodolpheh can you get in touch with me about the loader situation? I'd like to follow up with the vendor as well as the WG because my understanding is that we have solved most of the blockers to adoption of the generic loader, but I don't have a Lynx contact. My email is my handle at collabora dot com.

[GeorgeS2019]

To ALL,

This discussion is about making Lynx R1 SDK available for Godot developers.

Citing how Lynx R1's passthrough has potential advantages over others is strictly my personal opinion. I am not here to change anyone's opinion against other headsets. I just hope for "full support" of Lynx in Godot.

[BastiaanOlij]

@GeorgeS2019 We can't embed Lynx R1 proprietary logic into an Open Source project like Godot, this is one of the reasons this loader plugin is not part of the core repository and why I express my frustration that there are more vendors who are creating their own non-standard loaders on Android, than there are vendors who fully support Khronos' Open Source loader that is part of the OpenXR spec.

If the only option for pass through on Lynx is to use a proprietary SDK, than this logic needs to be implemented as a separate plugin, I really hope they are not going down this path because its a royal PITA to do. We'd need access to the hardware and it becomes a distraction to other important work we need to do.

OpenXR has a standard for supporting AR solutions as implemented by MS and as far as I am aware Magic Leap that involves setting the correct blend mode on the provided image, Godot fully supports this.

OpenXR also has a vendor extension by Meta for pass through that could be adopted by Lynx (and PICO) though I hope OpenXR soon adds a core extension for pass through all vendors can adhere to so we as a game engine developer don't need to implement a plethora of different solutions. Again Godot fully supports the Meta pass through extension.

In the end, this is up to the hardware vendors to do properly, to adopt and support the standard and join in with the discussions the OpenXR working group is having, instead of re-inventing the wheel and making it harder for a FOSS engine like Godot to support them.

Disclaimer here, I don't have a Lynx R1, I've not looked at their SDK, I'm basing my opinion here on the discussion above. For all I know Lynx is taking all the right steps but just need to have a little guidance to ensure conformity to the standard, so indeed @rodolpheh if you could get @rpavlik in touch with the right people at Lynx that would be awesome.

[BastiaanOlij]

Hey @rodolpheh

Hand tracking seems to work but there is rendering issues preventing them from showing.

Be sure to report this on the Godot issue tracker. Clay and myself are currently investigating an issue where on AMD hardware we're having geometry not rendering correctly when multiview is enabled, though this was on Vulkan, not OpenGL. Still the cause may be similar.

They do show with Vulkan but Vulkan display is upside down. Let me know if you're aware of any feature I should check for.

I'm not aware of anything in the OpenXR spec where we can communicate the orientation of the image we're submitting, this would seem to be an issue on the Lynx itself where they are expecting an orientation other then is standard.

Just to close this topic, I haven't tried the passthrough on the Quest Pro but I've tried it on Lynx and the Pico 4. Stereo camera really improves the experience and gives depth perception which is priceless. A simple test I tried which I think is very much telling about the comfort and quality of passthrough is to juggle with the headset on my head, and it is possible to juggle while wearing the Lynx (impossible on Pico 4 due to lack of depth perception).

Ah that explains why the PICO doesn't have the same sort of distortion the Quest 2 has when attempting re-projection of the stereo image. I also haven't tried the Quest Pro so can't comment on that.

For the use case of picking up your controllers and drawing your boundary I didn't feel the lack of depth perception was an issue, but I can see that it would be for more proper AR experiences.

[kisg]

FYI @rodolpheh @GeorgeS2019 , for proprietary extensions (like the scene capture extension of Meta Quest devices) we (mostly my colleague @konczg) created a PR that allows such OpenXR extensions packaged as GDExtensions: https://github.com/godotengine/godot/pull/68259

It is mostly reviewed already by @BastiaanOlij and @m4gr3d, we just need to rebase it to the current master. We will see if we can do that in the next few days, our work on this has been preempted by other projects in the last few weeks.

This solution could be used to support the Lynx passthrough API in Godot as well.

[rodolpheh]

As far as I can tell, Lynx's tooling for Unity sets the XrEnvironmentBlendMode to be Alpha blend or Opaque to enable or disable AR passthrough. It should be using standard OpenXR feature if I'm not mistaken. I am not able to try it now as I am pretty busy but will try to come back quickly with something. Please keep in mind I'm still in the process of learning about Godot and OpenXR.

@BastiaanOlij I'll try to capture more precisely the issue with the rendering and report it to the main Godot repository. It doesn't seem to happen on Vulkan but Vulkan has other issues. I understand your frustration regarding the proliferation of different OpenXR loader (and sometimes the lack of distribution of them, or the inaccessibility of them). As for depth perception, I do believe it's important for AR experience, specially when using hands as the main controllers and interacting with virtual object with direct touch (as opposed to far field interaction e.g. laser pointer).

@rpavlik I'll get in touch with you and see if I can arrange some introductions.

[rodolpheh]

Need more investigation but I'm struggling to find how to enable passthrough. Checking if AR or passthrough is supported in Godot returns false. I'll probably need more time to figure this out and I think it'd be better to create an issue to track the efforts into this. I can see that the blend modes are detected according to these lines and they are the ones expected for the Lynx:

02-23 17:55:23.923  4554  4611 I godot   : OpenXR: Found environmental blend mode XR_ENVIRONMENT_BLEND_MODE_OPAQUE
02-23 17:55:23.923  4554  4611 I godot   : OpenXR: Found environmental blend mode XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND

Regarding the OpenXR runtime, on start of my application I have this line showing up:

02-23 17:55:23.923  4554  4611 I godot   : OpenXR: Running on OpenXR runtime:  Snapdragon OpenXR 'v1.6.1' based on Monado(XRT) by Collabora   13.1.1

@rpavlik would this message explains why their OpenXR runtime doesn't work with KHR's loader?

@BastiaanOlij regarding the OpenGL issue, this is the error I get. Just checking if you've seen it before or if it is the one you're investigating. If not I'll open a ticket on Godot's repository:

02-23 17:55:52.261  4554  4611 E godot   : USER ERROR: GL ERROR: Source: OpenGL Type: Error ID: 2147483647  Severity: High  Message: Mismatch between shader number of views and FBO number of views
02-23 17:55:52.261  4554  4611 E godot   :    at: _gl_debug_print (drivers/gles3/rasterizer_gles3.cpp:167)

Double checking the logs, I see that there may be an issue with the version of OpenGL requested:

02-23 17:55:24.068  4554  4611 I godot   : OpenXR: Requested OpenGL version exceeds the maximum version this runtime has been tested on and is known to support.
02-23 17:55:24.068  4554  4611 I godot   : - desired_version  3.3.0
02-23 17:55:24.068  4554  4611 I godot   : - minApiVersionSupported  2.0.0
02-23 17:55:24.068  4554  4611 I godot   : - maxApiVersionSupported  3.2.1023

[BastiaanOlij]

Hey @rodolpheh

As far as I can tell, Lynx's tooling for Unity sets the XrEnvironmentBlendMode to be Alpha blend or Opaque to enable or disable AR passthrough. It should be using standard OpenXR feature if I'm not mistaken. I am not able to try it now as I am pretty busy but will try to come back quickly with something. Please keep in mind I'm still in the process of learning about Godot and OpenXR.

Indeed, for me similarly, I don't know enough about the Lynx and where it differs. This actually should work, from what I understand from the OpenXR specs setting the blend mode to alpha is what you should be doing on AR devices, and i think it's kind of inferred that on passthrough this should enable the pass through. Quest is the odd ball out here, I guess in part because Meta had their existing VrApi solution they needed to make available in OpenXR but also because Quest offers a bunch more options with how passthrough is applied.

It's kind of counter intuitive for Godot now because we have a passthrough mode that you don't need for passthrough on several devices :)

Need more investigation but I'm struggling to find how to enable passthrough. Checking if AR or passthrough is supported in Godot returns false. I'll probably need more time to figure this out and I think it'd be better to create an issue to track the efforts into this.

Like I said above, the passthrough extension is a Meta thing, nice and confusing if Lynx just uses the blend mode.

I can see that the blend modes are detected according to these lines and they are the ones expected for the Lynx:

There should be a property on the OpenXRInterface to change the blend mode. So you should be able to do something like:

func _ready():
  var xr_interface : OpenXRInterface = XRServer.find_interface("OpenXR")
  if xr_interface and xr_interface.is_initialized():
    get_viewport().use_xr = true
    xr_interface.set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND)

@rpavlik would this message explains why their OpenXR runtime doesn't work with KHR's loader?

Ah! Ryan was talking about that yesterday, yes if they are using the Snapdragon OpenXR loader that would explain it. So basically the Lynx entry in loaders really is the Snapdragon entry as it would theoretically support any Snapdragon based HMD.

Double checking the logs, I see that there may be an issue with the version of OpenGL requested:

This can be mostly ignored, the official stance as I understand is that we need to at least support the minimum, and that the maximum is sort of a "things are likely downwards compatible but up to you to test". On desktop it's annoying because most OpenXR runtimes report a minimum of OpenGL 4.0.0, while Godot is a GLES3 renderer and thus uses a 3.3 context. Hasn't given us problems yet (other than possibly some annoying issues with linear v.s. sRGB).

[rodolpheh]

@BastiaanOlij I've already had set up the blend mode. Here is my current function to initialize OpenXR:

func _ready():
    interface = XRServer.find_interface("OpenXR")
    if interface and interface.is_initialized():
        print("OpenXR initialised successfully")
        var modes = interface.get_supported_environment_blend_modes()

        for mode in modes:
            print(env_mode_to_str(mode))

        get_viewport().use_xr = true
        interface.set_environment_blend_mode(XRInterface.XR_ENV_BLEND_MODE_ALPHA_BLEND)
    else:
        print("OpenXR not initialised, please check if your headset is connected")

I might be missing something on the scene side. I've set up the environment background to either Clear Color or Custom Color with an alpha of 0, to no effect.

[BastiaanOlij]

@rodolpheh DUH! I'm so stupid!

Yeah just add this:

get_viewport().transparent_bg = true

That's needed to ensure the background has a proper alpha channel and it disables most of the background code.

I think that will do the trick fingers crossed.

[rpavlik]

Yes alpha environment blend mode is how you do video pass thru (and possibly some optical pass thru with dimmers) in the standard. Additive is for "normal" optical pass thru where you can add light but not remove it (black is transparent). Some vendors have added extensions for fancier video see thru, but (at least in my opinion) those should be in addition to the standard blend mode. Hopefully Meta (management) will agree at some point, I know the engineers have heard about it. Hoping the wg also can come to some agreement on augmenting the alpha blend mode with custom effects like the vendor extensions since there seems to be demand, apparently.

If you support multiple blend modes, the recommended algorithm is pick the first one in the list as provided by the runtime (they are in runtime preference order) that you support.

Iirc the custom snapdragon spaces loader is there for broker and layer reasons. I'm in touch with the right folks to help them fix that.

[rodolpheh]

Seems so close but I might be missing something. At first launch, it doesn't switch to passthrough, but there is a very small moment where the view goes black for one frame (could be the switch to the alpha blend mode ?). If I relaunch the app from Godot (while it is running), it will show one frame from the RGB camera then freeze completely and displays the following:

OpenVR Head pose now tracking with high confidence
OpenXR: failed to release swapchain image! [ XR_ERROR_INSTANCE_LOST ]
OpenXR: failed to end frame! [ XR_ERROR_LAYER_INVALID ]
OpenXR head space location not valid (check tracking?)
OpenXR: xrWaitFrame() was not successful [ XR_ERROR_INSTANCE_LOST ]
OpenXR: failed to end frame! [ XR_ERROR_TIME_INVALID ]

It also happens if I switch between applications and go back to the Godot application. One frame from the passthrough is shown before it freezes completely, showing the above in the logs.

[BenMcLean]

I really want to thank everybody working on this and I hope my question isn't a stupid one but here goes: What versions of Godot will I be able to build VR/AR apps for Lynx on? Will this work on Godot 3 or is it going to only work in Godot 4?

Answered on Discord: Godot 4 only. Good to know.

[rodolpheh]

Got an update from Lynx, and now the AR passthrough works in Godot :tada: It still works only one time on two, but as so does Unity made applications. There will surely be more fixes to come. The hands still not display either, this time it completely crashes when I show my hands.

[m4gr3d]

@rodolpheh Glad they're addressing the issue for passthrough!

Can you start a separate issue / thread for it. This will allow us to better track ongoing issues and their status.

[rodolpheh]

I don't think they tried to explicitly fix the passthrough for Godot, but they have provided a new OpenXR runtime which improves the performances and fixes a few issues. Here is the new version number of the OpenXR runtime:

OpenXR: Running on OpenXR runtime:  Snapdragon OpenXR 'v1.7.1' based on Monado(XRT) by Collabora   14.0.2