GodotVR / godot_openxr_vendors

Godot 4 wrapper for OpenXR vendors loaders and extensions
MIT License
97 stars 22 forks source link

Add automated tests using Meta XR Simulator #150

Closed dsnopek closed 3 months ago

dsnopek commented 4 months ago

The goal of the PR is to add the structure for automated tests running on GitHub Actions using Meta XR Simulator.

With the XR Simulator, it's possible to record inputs and periodically take screenshots, which both (the inputs and the images) get saved to a .vrs file. Then we can have the XR Simulator play back those inputs, taking screenshots at the same moments, recording into a new .vrs file. Then using pyvrs and a Python script provided with the XR Simulator, we can compare the screenshots, generate diffs, and decide whether they match or not based on a threshold of similarity.

Ultimately, I'd like to have a sample project and a .vrs file for testing each feature that can be tested via the XR Simulator (which includes passthrough along with the Scene API and Spatial Anchors, using synthetic environments, which is pretty cool).

However, the goal of this PR is just to figure out the structure, including one example .vrs file.

This is marked as a draft, because there's still a few things to figure out:

BastiaanOlij commented 4 months ago

Neat, I think the only thing we need to be careful off is that we don't put ourselves in a jam when upstream changes in Godot cause false positives in testing and we find ourselves unable to merge features because someone changed something in the rendering engine and missed the impact on XR (happens sadly all to often).

dsnopek commented 4 months ago

Neat, I think the only thing we need to be careful off is that we don't put ourselves in a jam when upstream changes in Godot cause false positives in testing and we find ourselves unable to merge features because someone changed something in the rendering engine and missed the impact on XR (happens sadly all to often).

Since we can play them back and automatically record the new outcome, I think we could fairly easily make a script that just re-records all the .vrs files when there are rendering changes. We should probably pin a specific Godot version, and then when we update that, we can run that script if necessary.

dsnopek commented 4 months ago

I spent a little more time trying to make SwiftShader work, but still haven't managed it. So I don't lose track of it, this is the error that appears in the console when Godot tries to start an OpenXR session when using SwiftShader:

[XrSim][00000.146735][E][arvr\projects\openxr_simulator\src\sim_xrapilayer_rendering_vulkan.cpp:419] Failed to find a suitable GPU

We then get these messages from Godot on the following lines:

OpenXR: Failed to obtain vulkan physical device [ XR_ERROR_RUNTIME_FAILURE ]
ERROR: Condition "!device_retrieved" is true. Returning: ERR_CANT_CREATE
   at: _initialize_devices (drivers/vulkan/rendering_context_driver_vulkan.cpp:475)
ERROR: Condition "err != OK" is true. Returning: err
   at: initialize (drivers/vulkan/rendering_context_driver_vulkan.cpp:591)
ERROR: Unable to create DisplayServer, all display drivers failed.

However, Godot itself (even the editor) seems to run fine with SwiftShader, albeit very slowly. :-) It's the XR Simulator that seems to have a problem with SwiftShader.

dsnopek commented 3 months ago

I've updated the PR to point at Godot 4.2-beta1, and removed the dependency on PR https://github.com/GodotVR/godot_openxr_vendors/pull/149.

So, as soon as this passes tests, it'll finally be ready for review and merging!

dsnopek commented 3 months ago

This one is passing CI and ready for review now!

fire commented 3 months ago

We discussed how the vulkan code in xr simulator didn't like swiftshader and how we could check it and maybe create a minimal reproduction project (pull request) for someone to look at.