[Feature] Add `XR_FB_Passthrough` support

pierre-l commented 2 years ago

Background

Hello,

As we've discussed on Discord I've been trying to make passthrough work with Hotham, without success so far.

I suspected the headset needed to be configured in a specific way for passthrough to work, but ruled that out by running App Lab HitMotion: Reloaded - MR Passthrough Demo which can be downloaded from the app lab. Passthrough works as expected.

So if I understand things properly, the call to xrEnumerateInstanceExtensionProperties shows that XR_FB_passthrough is not supported with Hotham. An outdated libopenxr_loader.so was suspected, but the result appear to be the same with the v38.0 downloaded from the Oculus mobile SDk page. I tried to voluntarily misnamed the dynamic lib to ensure that's the one that's supposed to be loaded. It would be nice to have the version printed at runtime.

My tests can be reproduced using this branch and running the crab-saber example:

pierre@host:~/hotham$ ./examples/crab-saber/scripts/run_on_device.sh

Version

latest

In which Hotham component are you seeing the problem on?

Core

What VR System are you seeing the problem on?

Quest-2

What OS are you seeing the problem on?

Linux

Relevant log output

03-25 10:38:52.452 29524 29550 I RustStdoutStderr: Available extensions: ExtensionSet { epic_view_configuration_fov: false, ext_performance_settings: true, ext_thermal_query: false, ext_debug_utils: true, ext_eye_gaze_interaction: false, ext_view_configuration_depth_range: false, ext_conformance_automation: false, ext_hand_tracking: true, ext_hand_joints_motion_range: false, ext_samsung_odyssey_controller: false, ext_hp_mixed_reality_controller: false, extx_overlay: false, fb_composition_layer_image_layout: true, fb_composition_layer_alpha_blend: true, fb_android_surface_swapchain_create: true, fb_swapchain_update_state: true, fb_composition_layer_secure_content: true, fb_display_refresh_rate: true, fb_color_space: true, fb_hand_tracking_mesh: true, fb_hand_tracking_aim: true, fb_hand_tracking_capsules: true, fb_foveation: true, fb_foveation_configuration: true, fb_triangle_mesh: false, fb_passthrough: false, fb_foveation_vulkan: true, fb_swapchain_update_state_android_surface: true, fb_swapchain_update_state_opengl_es: true, fb_swapchain_update_state_vulkan: true, fb_space_warp: true, htc_vive_cosmos_controller_interaction: false, htcx_vive_tracker_interaction: false, huawei_controller_interaction: false, khr_android_thread_settings: true, khr_android_surface_swapchain: true, khr_composition_layer_cube: true, khr_android_create_instance: true, khr_composition_layer_depth: false, khr_vulkan_swapchain_format_list: false, khr_composition_layer_cylinder: true, khr_composition_layer_equirect: false, khr_opengl_enable: false, khr_opengl_es_enable: true, khr_vulkan_enable: true, khr_visibility_mask: false, khr_composition_layer_color_scale_bias: true, khr_convert_timespec_time: true, khr_loader_init: false, khr_loader_init_android: false, khr_vulkan_enable2: true, khr_composition_layer_equirect2: true, khr_binding_modification: false, khr_swapchain_usage_input_attachment_bit: false, mnd_headless: false, mnd_swapchain_usage_input_attachment_bit: false, mndx_egl_enable: false, msft_unbounded_reference_space: false, msft_spatial_anchor: false, msft_spatial_graph_bridge: false, msft_hand_interaction: false, msft_hand_tracking_mesh: false, msft_secondary_view_configuration: false, msft_first_person_observer: false, msft_controller_model: false, msft_composition_layer_reprojection: false, msft_spatial_anchor_persistence: false, oculus_android_session_state_enable: false, oculus_audio_device_guid: false, valve_analog_threshold: false, varjo_quad_views: false, varjo_foveated_rendering: false, varjo_composition_layer_depth_test: false, varjo_environment_depth_estimation: false, varjo_marker_tracking: false, other: ["XR_OCULUS_common_reference_spaces\u{0}", "XR_FB_common_events\u{0}", "XR_FB_passthrough_keyboard_hands\u{0}"] }
03-25 10:38:52.452 29524 29551 V OpenXR-Loader: Verbose [GENERAL | xrCreateInstance | OpenXR-Loader] : Entering loader trampoline
03-25 10:38:52.452 29524 29551 V OpenXR-Loader: Verbose [GENERAL | FindLayerLibraries | OpenXR-Loader] : searching for layers in /data/app/rust.vroom-3n3YT4PqUP-A5ulTcTiYGw==/lib/arm64
03-25 10:38:52.452 29524 29551 V OpenXR-Loader: Verbose [GENERAL | xrCreateInstance | OpenXR-Loader] : Entering LoaderInstance::CreateInstance
03-25 10:38:52.452 29524 29551 E OpenXR-Loader: Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_FB_passthrough
03-25 10:38:52.452 29524 29550 I RustStdoutStderr: Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_FB_passthrough
03-25 10:38:52.452 29524 29551 I OpenXR-Loader: Info [GENERAL | xrCreateInstance | OpenXR-Loader] : RuntimeInterface::UnloadRuntime - Unloading RuntimeInterface
03-25 10:38:52.452 29524 29551 I OpenXR-Loader: Info [GENERAL |  | OpenXR-Loader] : RuntimeInterface being destroyed.
03-25 10:38:52.452 29524 29550 I RustStdoutStderr: Error [GENERAL | xrCreateInstance | OpenXR-Loader] : xrCreateInstance failed
03-25 10:38:52.452 29524 29551 E OpenXR-Loader: Error [GENERAL | xrCreateInstance | OpenXR-Loader] : xrCreateInstance failed
03-25 10:38:52.453 29524 29550 I RustStdoutStderr: thread '<unnamed>' panicked at '!!FATAL ERROR - Unable to initialise OpenXR!!: a requested extension is not supported', /home/pierre/workspace/hotham/hotham/src/engine.rs:73:30

Resources

Code of Conduct

[X] I agree to follow this project's Code of Conduct

pierre-l commented 2 years ago

As @kanerogers stated on Discord, passthrough has to be mentioned in the android required features. https://developer.oculus.com/documentation/native/android/mobile-passthrough/#project-setup

Initialization succeeds now. I still see a black-ish background, but am unblocked. TY!

kanerogers commented 2 years ago

No worries! Hope we can get this going @pierre-l !

pierre-l commented 2 years ago

I just published two drafts:

Both have numerous rough edges that I still need to fix but at least this PoC is persisted now.

pierre-l commented 2 years ago

@kanerogers Playing with this raised some questions and I would like you input on these.

The first problem I have is that passthrough can be used in various ways:

simple AR would be implemented with passthrough as a global underlay,
less common applications might just add passthrough to limited virtual surfaces,
it's possible some use cases might require a mix of both, intertwining virtual and passthrough layers. All of this seems to be permitted by the OpenXR API. My current #178 only implements the first one. I think it's a good compromise for a first version but such a simple implementation will have to go through significant changes, including API, in order to provide all features.

Would you be OK with initially implementing just this first step?

Also, I expect many Hotham users not to need passthrough at all. What would be the best way to make this optional? Would it be fine to just add a passthrough_underlay boolean to the Engine and XrContext constructors?

leetvr / hotham