VR driver rewrite / Linux support

[zmerp]

Add support for Vulkan and Linux. This should be the last step of the Rust rewrite. By the time this issue is reached, Valve could have added an OpenXR driver interface. In that case that should be preferred in place of OpenVR. Another option is implementing an OpenHMD driver.

Can someone provide a simple VR driver example for Linux? It doesn't need to be integrated in ALVR just yet.

EDIT: You can follow the development progress here: https://github.com/alvr-org/ALVR/wiki/Linux-Support-development-progress

[revan1611]

Please please please 🙏🙏🙏

[zmerp]

Update: it seems that the IVRDriverDirectModeComponent interface that ALVR uses is not supported on Linux (https://github.com/ValveSoftware/openvr/issues/749) and we still don't know anything about the IVRVirtualDisplay interface (https://github.com/ValveSoftware/virtual_display/issues/5)

[jarrard]

I was hopeful that someday Oculus would release a Linux version of their app, if that happened would it in some way help projects like this? (assuming Link support would work under Linux)

I wish I had the skills required to help out with this. Using Windows10 for VR all the time is annoying. In saying that SteamVR under Linux still has some random issues for some games. (mainly when running VR via proton)

[fezboy]

An oculus app on linux wouldn't really help ALVR afaik, because ALVR doesn't use the oculus app. On the PC it only interacts with steamvr, and the client on the headset is installed and managed via sidequest, which is crossplatform

[jarrard]

Oh, :( I guess ALVR is our only hope for WiFi VR under Linux then. I remember talking to the Virtual Desktop dev about support for Linux and he pretty much gave me the middle finger! :( obviously not a fan of Linux .

Its quite hard finding developers for VR software whom are ALSO fans of Linux! it seems everyone is firmly kissing Microsoft's butt when it comes to VR development, except for Valve.

[zmerp]

Update 2: by looking through https://github.com/ValveSoftware/SteamVR-for-Linux issues I see logs of built in SteamVR drivers (like for the Lighthouses) that implement IVRVirtualDisplay, so writing a driver using that interface could be theoretically possible.

[jarrard]

sideQuest wires into the Quest under Linux with its linux build, perhaps how it does it is somehow useful in this case.

[zmerp]

@jarrard Sidequest has nothing to do with this. Sideqest uses adb, that is compatible with every android device. We need support for a feature specific to SteamVR. What we need to do is not transferring images from the quest to the PC, but from the PC to the Quest. SteamVR, through OpenVR, can provide the frames rendered by PC VR games.

[jarrard]

@jarrard Sidequest has nothing to do with this.

Oh ok, I just thought maybe it had something going on because it can stream video and such to the Quest from what I know so its not just basic ADB commands?

[zmerp]

As I said, ADB streaming feature works in the wrong direction, and there are not many parameters we can control. But this is not even the matter of this issue. The question here is how do we obtain the images in the first place, before streaming them. I already found a possible solution with IVRVirtualDisplay, now someone could try implementing it and seeing if it works.

[sandersr]

Is it worth looking at:

https://monado.freedesktop.org/

As an alternative to SteamVR?

[zmerp]

@sandersr AFAIK it doesn't have a device driver API.

[lunarcloud]

@sandersr AFAIK it doesn't have a device driver API.

https://monado.pages.freedesktop.org/monado/md_writing-a-new-driver.html

[zmerp]

@lunarcloud Thanks! apparently I didn't look hard enough. Let's hope it supports something like OpenVR virtual mode.

[jarrard]

Apparently AMD has a VR streaming app you can use under Linux. However I haven't tried it out yet.

[fezboy]

@jarrard do you have a link? DuckDuckGo gave me ReLive VR, which as far as I can tell is Windows only.

[jarrard]

Yeah sorry I was mistaken, the Quest installs a package which connects to the windows app atm. Github link below with requests for Linux support, which should be possible as it does not rely on Oculus APP.

Its open source so might help with this project on some level?

https://github.com/GPUOpen-LibrariesAndSDKs/Radeon-ReLive-VR

[AmmarkoV]

I would very much like a Linux version of ALVR! For the Oculus Quest2 the Oculus Mobile SDK comes with several VRSamples like the VrCubeWorld_Framework.

So given that one can actually write a native Oculus 2 application that can act as a "driver" and you can use Linux to compile the native application to render to the left/right eye and send back the 6-dof pose of the head and controllers what is the best format to transmit the image? Should it be on an image to image basis ? as a mpeg stream or what ?

Btw for anyone trying to compiling a native .apk for the Quest from linux.

You need to set-up android studio

Add a rule for the Oculus device in /etc/udev/rules.d/51-android.rules ( This will allow adb to work with the quest device with normal permissions )

SUBSYSTEM=="usb", ATTR{idVendor}=="2833", ATTR{idProduct}=="0186", MODE="0666", GROUP="plugdev"

Set up the correct .bashrc exports

export JAVA_HOME=/path/Programs/android-studio/jre
export ANDROID_HOME=/path/Android/Sdk/
export ANDROID_NDK_HOME=/path/Android/Sdk/ndk-bundle
export ANDROID_NDK_ROOT=/path/Android/Sdk/ndk-bundle
export ANDROID_NDK=/path/Android/Sdk/ndk-bundle
export OCULUS_SDK_PATH=/path/Documents/3dParty/OculusVRSDK/ovr_sdk_mobile_1.37.0_my/
export OVR_PROJ_ROOT=/path/Documents/3dParty/OculusVRSDK/ovr_sdk_mobile_1.37.0_my/VrSamples

Add some extra flags on cflags.mk of the Oculus Mobile SDK

LOCAL_CFLAGS    += -Wno-strict-aliasing         # TODO: need to rewrite some code
LOCAL_CFLAGS    += -Wno-unused-parameter
LOCAL_CFLAGS    += -Wno-missing-field-initializers  # warns on this: SwipeAction    ret = {}

You can then run the build.py python scripts for example VrSamples/VrCubeWorld_Framework/Projects/Android/build.py which will compile and upload the sample application.

Also having the Oculus connected and running you can cast the image of the headset using adb

adb exec-out "while true; do screenrecord --bit-rate=2m --output-format=h264 --time-limit 180 -; done" | cvlc --demux h264 --h264-fps=60 --clock-jitter=0 -

This example can stream from the oculus to the linux PC but it works pretty badly, especially in a non USBC connection

[zmerp]

Oculus Mobile SDK is a game facing API, not a driver API. The only VR driver API is OpenVR, but doesn't seem to work with non wired headsets on Linux. The best solution so far seems to emulate a virtual hmd display and then screen capture it, but no test had been made yet with this.

[AmmarkoV]

Yes but what is the reason why a driver can't be written as a "game". It's ok if you need to manually start the "game" .apk to start serving the textures from the Linux PC.

For example this is how Steam Link works. If you adb install Steam Link.apk as a "game" you can connect to Steam as a 2D streaming solution.

A "screenshot" from my phone ( since actual screenshots are disabled for this application ) of Steam Link playing Monkey Island 2 on the Quest2

[zmerp]

The problem here is how to get game frames in the first place. You could search in the GPU memory or inject games but that is very difficult and error prone. OpenVR was supposed to return the game frames but it does not work on Linux.

[AmmarkoV]

Well I have code that grabs an X-Server window and serves it as JPEG through HTTP, however this works @ 5-6fps, for a native OpenGL application I also have code to set up a GLX3 context then render to p-buffer textures then download the image then i.e. write a jpeg image in memory and sending it through a socket is trivial.

Of course a solution using NVENC would be far superior performance wise..

[scalway]

I hope there will be some way to use it on linux... but I know how hard such stuff can be.

[jarrard]

Yeah transferring to a OS agnostics coding echo-system is very hard for allot of Windows only developers. But its for the better really. Opens up many doors.

Allot of people also really want to use something like Linux for VR, just such a shame its so incredibly limited atm, even the tethered Valve Index etc options have quirks about them under Linux. (stuff like steamplays proton doesn't always play nice with VR)

[Geo25rey]

@zarik5 This driver at the very least compiles on Linux. Also, this Vulkan example might be useful.

The main issue with the changing to a crossplatform architecture is the dependencies of Windows only libraries such as wil.h (Windows Implementation Library) and d3drender.h (DirectX).

Do you have a branch for the crossplatform rewrite, or should PR's just be directed to the master branch?

[zmerp]

@Geo25rey The sample in the first link doesn't use any interface to obtain game frames, and the second link is for the game facing API, not the driver facing API. We already moved on from OpenVR. We thought we could use something like a Vulkan validation layer to extract frame data directly from the SteamVR runtime internals. No tests have been done on this method yet.

https://github.com/alvr-org/ALVR/wiki/Linux-Support-development-progress

[Geo25rey]

@zarik5 What if I get the virtual display driver example working on Linux with Vulkan?

[zmerp]

We tried. It crashed on Nvidia and does nothing on AMD. You are welcome to try again though, maybe we missed something

[hedmo]

I like to help to with porting ALVR to Linux. I am not a coder but i have been checking vr in Linux for quite som Time now . What is the problem with drives? How about openHMD?

[zmerp]

OpenHMD does not have an API to query game frames, so basically it's restricted to use only with wired headset.

[xytovl]

I don't understand why you need to grab frames.

Are they not provided by steamVR in IVRDriverDirectModeComponent::SubmitLayer ? From what I gathered, the buffer format depends on platform, D3D11 on Windows and Vulkan on Linux.

Is there any branch that at least compiles?

[zmerp]

IVRDriverDirectModeComponent does not work on Linux. We searched a lot but there are no working samples online.

[jarrard]

I only know of this, but its windows to linux, don't know of it working the other way.

https://forum.level1techs.com/t/looking-glass-guides-help-and-support/122387

[xytovl]

And in IVRVirtualDisplay::Present waht does the pPresentInfo->backbufferTextureHandle contain ?

[zmerp]

We didn't get that far. Present gets never called, the driver crashes before or does nothing.

[FeckingPotato]

Valve could have added an OpenXR driver interface They did add an OpenXR driver interface: https://store.steampowered.com/news/app/250820/view/3044967019267211914

[xytovl]

I understand it as the opposite: on Windows only, an application can use openXR to talk to SteamVR. But SteamVR still uses OpenVR driver interface.

For a Linux port, it could be interesting to be a driver of monado, which would require to implement a mechanism in monado to replace the output inside a driver. Unfortunately this would only cover Linux, and Windows would still need the OpenVR driver.

[zmerp]

@FeckingPotato That is still only for the game facing API. We never got any news of VR runtimes starting to support a driver facing API, not even with OpenXR. Monado has a driver API but only support wired headsets. OpenVR is the only one out there. For Linux the only possibility seems to inject games to replace their openvr.so with our custom one. It will be very time consuming and error prone (we need to make a different injected .so for every published OpenVR version x graphics library), but I don't see other alternatives.

[omigeot]

I'm pretty sure you guys have already looked into the Relativty project, but since I don't see it mentionned yet.

They're said to have SteamVR support for their opensource (wired) headset : https://github.com/relativty/Relativty

[frostworx]

Just stumbled over this: https://github.com/alvr-org/ALVR/wiki/Linux-Support-development-progress

[nfl0]

ALVR discord

[Geo25rey]

@xytovl @zarik5 Looks like progress is being made on a Linux driver. Are you looking for testers?

[xytovl]

@xytovl @zarik5 Looks like progress is being made on a Linux driver. Are you looking for testers?

I am not actively working on alvr at the moment, I will resume in 2 weeks I hope. The current version has severe known limitations (instability, poor GPU support, hardware requirements) so testing alone is not needed at the moment.

You are free to test, but currently we are more looking for patches for the 2 solutions under discussion (acquire images through https://github.com/ValveSoftware/openvr/blob/master/headers/openvr.h#L3431 GetMirrorTextureGL, or through a vulkan layer to be developed)

[ckiee]

Resolved by #604.

[jarrard]

Resolved? so Quest2 now works on Linux? it plugs into SteamVR correctly under Linux???

[m00nwtchr]

Resolved? so Quest2 now works on Linux? it plugs into SteamVR correctly under Linux???

Yes, and no, there's an experimental version that mostly works but it still has a few issues (mostly NVIDIA exclusive ones afaik), you can test it out for yourself though https://github.com/alvr-org/ALVR/wiki/Linux-Support-development-progress, also join the discord for updates (and a version of the client which you'll need to test if you have an nvidia card/official drivers) @jarrard

[jarrard]

Cool, I have a 6800XT.

UNFORTUNATELY I sold my Q1 and Q2 earlier this year. Waiting for next next gen VR to arrive.

[ckiee]

@jarrard There are still many problems but, as @lmarianski said, it does work.

A few problems off the top of my head:

• audio streaming is not working
• foveated encoding is not implemented
• position is stuttering on nVidia (WIP fix exists in #linux-dev by CoffeeKat)
• it is not packaged for mainstream distros

[m00nwtchr]

Well, microphone streaming, I was able to stream game audio to the headset.

[ckiee]

@lmarianski That isn't supposed to work! We haven't worked on that at all yet and no one has gotten it running before you so I have some questions:

• What audio server are you running? (PulseAudio, PipeWire, pure ALSA)
• Did you perform any extra steps in your audio server to get it to work?
• If you're on PulseAudio, can you run pactl info and pactl list and send the output?
  • pw-cli info all for PipeWire.
• Did you patch any code?