New GUI framework and the future of ALVR development

[zmerp]

Context: I'm (very very slowly) building an alternative to ALVR from the ground up designed to be cross-platform (OS/hardware/headset agnostic) and to support the latest tech (sliced encoding). I've chosen the language Rust for its features (namely expressiveness, type system and zero cost abstraction memory management) and library ecosystem.

Since ALVR would benefit from a new GUI for the server (expecially since WinForms is awful for collaborative coding) I thought I could avoid "spreading" my time resources and develop one GUI base for both ALVR and my project.

So I found a few options for the new user interface:

• Use Rust + Iced: Pros:

  • I'm comfortable with Rust
  • Iced is lightweight (very thin runtime and no external dependencies)

  Cons:

  • Other ALVR contributors may not be familiar with Rust
  • Iced is still young and lack some UI controls but it is the most promising GUI library in Rust
  • Hard to integrate with existing C# code

• Use Dart + Flutter Pros:

  • There are more potential contributors familiar with Flutter (statistically)
  • Flutter provides an end to end environment for building and publishing apps

  Cons:

  • Flutter for desktop is still in technical preview (not even alpha), but there is a lot of work behind
  • Need to restructure the whole project around Flutter
  • Hard to integrate with existing C# code

• Use Javascript + ReactNative: Pros:

  • Even more potential contributors

  Cons:

  • The final product needs to embed an entire browser rendering and Javascript engine, so it can be slow.
  • Hard to integrate with existing C# code

Other alternatives:

• C# + WPF: Pro:

  • Can keep C# code
  • We're already familiar with the language

  Cons:

  • I cannot reuse code for my project

• Continue with C# + WinForms

The GUIs for Rust, Dart, Javascript are written in their native language, WPF uses XAML, WinForms does not have user editable code.

For my project I thought I could begin with a Rust GUI and switch to Flutter or React if requested by contributors.

@JackD83 Even ignoring my personal preference, what do you think is the best approach for ALVR?

Note that I don't have an ETA both for my project or the new GUI. I'm just asking for feedback on where to invest my time.

As an aside, for the rest of ALVR the codebase, It would be easy to integrate Rust with C++ because they are both native.

[zmerp]

For the context, I already have some pieces of Rust code (that I cannot translate to other languages) for automatic settings schema generation, so I would like to add Rust to ALVR codebase anyway.

[JackD83]

I personally have no experience in either c++ and c# other than this project. Rust would be a welcome challenge and would help to make a cleaner codebase than the current mix of c style c++ code and various c++11 styles.

There is another alternative UI solution: Integrate light weight http server into the alvr_server and provide simple api for a html/js UI. Like SteamVR is doing it for their dev tools and the controller mapping

Pros:

• Self contained software
• Easy to create and extend UI
• UI could be accessed from a phone or inside the Quest

Cons:

• UI is only available if SteamVR is running

[zmerp]

I really like the HTTP server idea! Controlling ALVR from Quest browser would be a lot better than putting the headset on and off to tweak settings.

Having the UI be accessible only when SteamVR is running could be confusing for existing users, maybe it's better to leave the components independent from each other.

Adding another language would not make a cleaner codebase but Rust code is a lot easier to maintain (and is almost immune to memory related bugs)

So here's the master plan:

• The UI is driven by an HTTP server. On pc it is launched in a window ad hoc, similarly how it is now, but is can also be accessed in any browser that has network access to the pc.
• The client handshake code is rewritten in Rust and it is integrated in a module "alvr_common. This module will also contain the code for settings interop and will be included in both the UI and driver. The communication happens through a JSON file. (I don't know what the current ALVR-common does but the code can be moved inside alvr_server)
• Launching SteamVR will also launch the HTTP server without its window.
• Removing C# removes the need for .NET framework dependencies. The VC++ redistributable cannot be removed but I've read online that it can be embedded in the main exe/dll. So ALVR would become truly portable.
• On the UI an Advanced tab is added with procedurally generated settings directly from the settings JSON file + a custom JSON schema. Modifying advanced settings will be discouraged.
• The client remains untouched.
• The overall performance/stability of ALVR remains unchanged because C++ code is not touched (apart from the settings bit).

I may need some help on the aesthetic side. I know a bit of javascript but I have no experience on web frameworks, I'll see what I find.

[zmerp]

It could be possible to have a single binary for driver and web server but to avoid SteamVR complaining that there is no headset connected (and then shut down the driver, if the Quest is on sleep) there would be the need for an emulation layer (a thing that I planned since the beginning for my project). This would require a lot of extra work.

[JackD83]

I would propose an architecture like this:

The websocket is used for all communication between the browser and the config-service. Http is only to deliver the files. Generating an ui from a given JSON would be very easy on the web

[zmerp]

Yeah, this is right about what I had in mind, thank you for the diagram. I thought I could make the control socket communication via file for simplicity (a unified file for logging and statistics) but I would have to test the performance. Would it even be a good idea?

[poVoq]

Just a quick comment: please keep portability to Linux in mind this time. Thanks a lot :)

[zmerp]

Yes, that is a priority for this rewrite and for my <unnamed-alvr-next> project. After this there are still the graphics engine, the network sockets and a few other sys calls that need to be rewritten to allow Linux support. For now I'm focusing on increasing ease of use on Windows.

[JackD83]

Just a quick comment: please keep portability to Linux in mind this time. Thanks a lot :)

None of us worked on the original ALVR. There are a lot of things we would like to change but are very difficult to change with the current architecture.

I thought I could make the control socket communication via file for simplicity

I have never used this method. I like the sockets because of their explicit connection status and on connected/disconnect events.

I for got to add the headset discovery in my diagram. This has also be handled by the config-service

[3unjee]

IMHO you should definitely stick to C / C++ for the lower level streaming part. Performance and portability are never the true focus of anything else. Convenience is always a tradeof or a short term (often market driven) advantage.

Do whatever you want for the GUI and high level stuff, but the core should be C++.

[zmerp]

@3unjee Both Rust and C/C++ can be compiled to LLVM IR. Currently Rust is at worst 1.5x slower than C and at best 1.2x faster (benchmarks) so they are comparable performance wise. On the other side there is Virtual Desktop that is written in C#, that is significantly slower and that is affected by GC lag spikes, that manages to perform better than ALVR, so raw language performance doesn't seem to matter much.

[zmerp]

I like the sockets because of their explicit connection status and on connected/disconnect events.

I thought about using a file because the flow of data is almost exclusively one-directional. Having the config-service be able to ask the driver status is definitely useful. I will have dig into how ALVR is currently working to have a clearer understanding.

[Ekristoffe]

hello all, isn't the ALVR client directly communicating with the ALVR server ? What is the control socket for ? If only for logging and reading config files, i think a file system is a better idea. If a file system is used we can also only use the ALVR headless (when all the config are done).

[3unjee]

@3unjee so raw language performance doesn't seem to matter much.

Well i've discovered that performance always matters when it comes to software. Sure it doesn't matter everywhere but when you take the convenience road you lose the freedom of applying it where it matters. Also benchmarks are not always relevant, maybe you should look at real life projects instead.

I understand the urge of trying new languages, because it's shiny and that last project was painful... But in reality finding better design(s), frameworks or libraries is a often a better solution. Back in the days I've switched to the Qt framework instead of embracing the Java road and I'm glad I did it. Anyway just my two cents folks :).

[JackD83]

@Ekristoffe the control socket is used for sending commands while alvr_server is already running (config changes, debug flags etc)

The current architecture looks like this:

Its a mixture of different languages, patterns and coding styles. Its grown over time and it shows. In this current fork, only a handful of people have contributed code and choosing rust over C++ could help to make the project more attractive. The C++ code currently used is very hard to read.

@3unjee You can choose any language and slow it down to a crawl

[zmerp]

@3unjee I know that performance must be a priority for a project like ALVR, but you should always make some sort of compromise. If you want the highest possible performance you can use SIMD and ASM everywhere but that makes a poorly maintainable project. I have 7 years of experience in C++ and 3 in Rust. It's not much but I think that is enough to make an informed choice of what is the best tool to use in ALVR, given his history and support status. I have a project that comes before ALVR that is written in C++, and it didn't pan out well.

Having said that, ALVR is just a "pile of glue", it connects components like video codecs and libraries that are where most of CPU/GPU time goes. If we find that performance is not good on some crucial areas that are written in Rust, there is the unsafe keyword to basically return to C realm.

I plan to maintain the code written in Rust. If you want to contribute on ALVR, having you aboard would be more valuable than the choice of language, in that case we can continue with only C++ if you wish.

[Ekristoffe]

Wouldn't it be easier to have : The config service which do the config file write/read, inside the alvr server, The alvr client talk with the server directly, The http server talk with the config service via websocket ? Just in case so if we need to change the frontend we just reuse the websocket to the backend ? Here is something I thought about:

[zmerp]

@Ekristoffe The problem is that the lifetime of alvr_server (and its components) is managed by SteamVR. If SteamVR is not running there would be no way to save the settings or to see if the client is available for connection.

[3unjee]

I plan to maintain the code written in Rust. If you want to contribute on ALVR, having you aboard would be more valuable than the choice of language, in that case we can continue with only C++ if you wish.

I have to admit I've been using VirtualDestkop a bit lately and find it amazing, in particular for VR streaming as expected. But I don't like the fact that it's closed source, it seems to me streaming technologies are extremely valuable to the community and should be as open as possible.

That being said I also need to make a living, so I've been thinking about an hybrid licencing models like GPL / Private. Having a low level ALVR "brick" would enable me to import it and then build on top (since it's MIT). Kinda like I'm currently doing with libVLC or libtorrent.

But to be honest I'm also tempted to rewrite part of it, using Qt, since it covers Android really well and is a significant boost over standard libraries.

Would you consider building something like a low level library like libALVR ?

[poVoq]

@3unjee Fully open source and selling it on Steam are not contradicting. I (and for sure many others) would gladly pay 10€ for a nice, Linux supporting & open source streaming client conveniently available on Steam. But I consider open-source and Linux support to be vital as otherwise there are already plenty of alternative options.

[zmerp]

@3unjee In this rewrite I plan to expose a REST API for interacting with settings and polling the server and client status. You can build your software on this.

Me and JackD83 do not own 90% of the code and do not profit for it, if you want something like libALVR you should contact the original author @polygraphene for licensing. I'm not a lawyer, but since ALVR is MIT licensed, I think you can just take the code and do whatever you want, you just have to point to the original licence and source code.

Personally I wouldn't like someone make a profit on someone else work that was made available for free, but that is just my opinion.

Final note: I'm still a student, so my time is really valuable. Because of this I would really prefer spending it doing what I want. The work I'm doing on ALVR is not guaranteed to ever see the light of day.

I would like myself sell some software (maybe by making partnerships) but it would require being responsive for technical support and patches, both things that I'm not in a position to guarantee.

[3unjee]

@zarik5 A REST API definitely wouldn't be enough if I want to build something else on top of ALVR, I think seeing this as a low level VR Streaming brick is far more interesting and future proof because of the convergence between fiber / 5G and standalone VR headsets like Oculus Quest.

@poVoq The point is a clear separation between a technology and its implementation. Take libVLC and VLC for instance, I wouldn't mind as much if VLC became closed source than libVLC because most of the technical value is here.

We could have something like libALVR_server and libALVR_client licensed under MIT (or LGPL if the original author is okay) and then everyone build from these. Maybe we could even convince the coder of Virtual Desktop to contribute to this since the community would probably do a better job at covering all the odd cases in the long run and that wouldn't defeat the value of what he builds.

[zmerp]

The idea of a libALVR is not bad, but I am not totally familiar with ALVR codebase, I don't think I can do such a thing, especially since the original author is not responsive. Instead I would like to rewrite ALVR from scratch with cross-platform and future proof in mind, but currently that is more a dream than anything else.

[3unjee]

@zarik5 Yeah having something like this with a very open licence is an interesting perspective if we want to build interesting projects on top of VR Streaming, while keeping the technology open for new comers across the board.

I feel I would be up to the challenge doing that (even though pure C++ is a bit tedious) but I would like to have the developper of Virtual Desktop to give us a hand because he could help us a lot to improve the ALVR solution

[zmerp]

You can definitely ask @ guygodin for help. I'd say I first try to build the new GUI backend, so I familiarize more with ALVR, then I'll see what can be done. For an ETA, do not expect anything before July.

[3unjee]

@zarik5 @JackD83 Maybe I could contribute to the lower level ALVR library that would process inputs, extract and encode frames so you could use that with the higher stuff you are trying to build ?

[zmerp]

@3unjee That would be awesome! Thank you!

For sure you know these things better than I do, but to make them clear, I want to list some things to keep in mind when dealing with ALVR unmaintained codebase:

• Try to make small incremental changes and test everything after each one.
• Even if you make large changes, try to keep them as confined as possible.
• Jack an I can help you with code reviews, but we don't have insight in every corner of the codebase, so for every change there will be a chance of introducing bugs.

If you instead want to pursuit the quest of a libALVR the whole project would need a major rewrite:

• stateful globals should be removed (apart from the OpenVR server instance required by HmdDriverFactory).
• Every component (that will be available in libALVR API) must be made independent from each other.
• Every component should not own externally allocated memory; it needs to be stateless or its state must be simple and clear. Its better to have static functions (even when private) that take more arguments than having to deal with complex mutable state.
• Write tests (I can help you in this). Such a rewrite would benefit from them and at the same time would make them easier to write.
• Other rules for maintainable code apply, if you have experience you're good to go.

You are free to give this a shot but we would definitely need your help to maintain the code.

[3unjee]

@zarik5 As far as I can tell the libALVR would be mostly the code inside the alvr_server folder.

As far as I can tell, the idea would be having this standalone bricks that takes inputs and returns frames while respecting the user configuration. I've been investigating a bit about Virtual Desktop and it seems aside from the OpenVR driver they also rely on "injection" to trick a binary into thinking an Oculus Rift is actually plugged in. I'm not entirely sure how it's built, feels like black magic to me.

I could come up with something, but I would like to license libALVR under LGPL, it seems to be a better license in my opinion than MIT. Also since it's a bit tricky to maintain pure C++ I probably need to add a dependency to Boost or QtCore (not the GUI stuff). There could be CLI binding for the C# client and maybe one of you could write test(s) for the library ?

I would like to open a Discord to discuss about this, would you care to join it ?

[poVoq]

Spoofing a Rift will be problematic with crossplatform stuff. Maybe better to implement it as HTC Vive clone? http://www.openhmd.net/ seems to do something like that?

Or maybe just go OpenXR directly? This might be a good open-source resource for that: https://monado.dev/

[3unjee]

@poVoq So openhmd basically provides an open source driver for headsets like the Oculus Rift ? That's interesting I wonder how well it actually works ?

(Monado seems to be a VR engine so I'm not sure how it would help us with video streaming)

[poVoq]

@3unjee Not so well, as much of the functionality isn't implemented yet and there are difficulties especially with inside-out tracking (which wouldn't be the case for the Occulus Quest in streaming mode I guess).

Monado isn't a VR engine, but also an open-source driver framework, but one targeting the new standard OpenXR instead of OpenVR (Valve's inhouse one). But it looks like Steam will support OpenXR sooner or later, as they take part in it and it is the OpenGL/Vulkan of VR headsets. Learn more about it here: https://www.youtube.com/watch?v=LdPq3Bke8j8

[3unjee]

Monado isn't a VR engine, but also an open-source driver framework, but one targeting the new standard OpenXR instead of OpenVR (Valve's inhouse one).

Yeah, I guess it might take a bit of time to be adopted depending how friendly Oculus folks and game developpers are willing to embrace it. And that wouldn't help with "injection" and supporting current Oculus Rift Store games, but that's definitely a good news for the future of VR Streaming.

Coming back to libALVR I think it's clear we shouldn't entirely restrict the architecture to OpenVR and be open to other methods.

[zmerp]

@3unjee Khronos planned to make a OpenXR device plugin interface (similar to OpenVR driver API) but AFAIK development hasn't started yet. So the options now are only OpenVR and injection. There was the driver Vireio Perception (or the successor VorpX) that also uses some kind of injection, in this case to convert a non VR game to VR. I think the injection can be done by "simply" making the game discover first a custom made dll with the same public symbols as the Oculus runtime dll, instead of the original Oculus dll. I would never attempt do such a thing, maintaining a backend for an evolving API with its many legacy interfaces would be a nightmare for a single person.

Responding to your earlier comment: I don't think you can re-licence the existing code yourself, you need to contact the original author. Adding other libraries I think it's fine, as long as it would not make the installation process more complicated for the end user.

I would like to open a Discord to discuss about this, would you care to join it ?

Sure, send the invite to my email if you want.

I would like to hear @JackD83 about this direction of development (libALVR).

[3unjee]

@zarik5 @JackD83 Definitely would want to keep the legacy of the original author and the work of Jack intact if I start working on libALVR. We should make sure that we have no regression with quite a bit of testing. As for the rest I guess it's meritocracy if I end up writing something that's any good we'll likely use it. I have a little bit of experience in video streaming with previous activities, we'll see how it fares out...

I think the VirtualDesktop developper (@guygodin) actually maintains a list of compatible games with his injecton method. I'm pretty sure the dll stuff you've mentioned is actually what he does, because we can't seem to be able to run the games from the official Rift store.

Also, does anyone know where @polygraphene went and his official reason for quitting the project ?

[JackD83]

I'm really not sure about a libALVR. Maybe a clean re-implementation with a cleaner architecture would be enough to be able to reuse most parts in other projects or if a new VR standard is actually used. I also know that if we already want and include too much in this phase, complexity will rise and nothing will be done.

I personally would prefer a clean OpenVR implementation over any injected dlls and hacky stuff.

As why @polygraphene stopped working on ALVR I can only guess and speak from my experience here: The project is aimed at end-users but does not have the community to support it. Most of the bug reports are from people creating an github account just for this. Very few could compile it or start a debug session and the issues are piling up. Without help, one alone can not support this and it gets frustrating. (That is why issues get now automatically closed after a time :P ) ev7 had 38,350 downloads!

[3unjee]

@JackD83 Excellent diagram, makes a lot of sense. Even if we don't go the libALVR route I think it would be a good idea to make the alserver a standalone multiplateform application (as your diagram shows). I'm interested in revisiting the @polygraphene source code here, it feels like a coherent move having 10+ years in C++ and experience in streaming with libVLC (developed by a few french "compatriotes" :)).

I could contribute mostly on that and you guys could focus on the HTTP server and ALVR client stuff, if that's okay for you? Are you actually dropping the C# interface and replacing it with HTML ?

All in all, Even though I'd need help for testing and reviewing changes to make sure we don't have regressions, I think I could come up with something maintainable and easy to read.

@JackD83 @zarik5 @poVoq Made a discord to discuss about this: https://discord.gg/ypagkhV

[JackD83]

@3unjee yes the C# has to go. The ui is unmaintainable anyway.

Here are some diagrams I made while digging into ALVR. They are a good start: https://raw.githubusercontent.com/JackD83/ALVR/master/ALVR_classes.png https://raw.githubusercontent.com/JackD83/ALVR/master/ALVR_Sequence.png

[3unjee]

Okay, anyone who wants to discuss some more about the new design is welcome on Discord: https://discord.gg/ypagkhV

[Ekristoffe]

About the next alvr, will you plan to also support the other VR headset ? It would be quite cool to have the Oculus GO. For example I only have a GO and I am stuck with the original ALVR without any improvement made by this repo.

[zmerp]

I thought about supporting the Go in a second wave of rewrite. I will need help from others because I don't own an Oculus Go.

[poVoq]

This might sound a bit strange, but Cardboard and maybe Daydream are also still viable platforms for VR streaming :)

[zmerp]

Cardboard and Daydream are deprecated. I think it's worth looking only at supported and future platforms.

[Ekristoffe]

I have a GO, maybe I can help (if it is a programing language I understand)

[zmerp]

@Ekristoffe Thank you. I can start working on that in a month (after ALVR v12 is released), then it will take another couple of months until v13 is ready for testing.

[poVoq]

Cardboard is fully open-source now and being adopted by the FPS quad-copter racing scene, so I don't think it is really fair to say that it is being depreciated. Daydream... yeah sadly.

[zmerp]

@poVoq Cardboard has a completely different API than Oculus mobile SDK for the Quest and Go. Supporting both requires a lot of glue code that makes things unmantainable. @polygraphene started going this route, but then abandoned it. It's easier to make a different fork of ALVR for only cardboard.

[MPfaffe]

Just for my understanding: the current ALVR experimental v11 is built up with Oculus mobile SDK and therefore does not work with other Android devices? But applications built up with Android SDK would work with oculus device too? Are there sdk Cardboard SDK specific functions needed for ALVR, or android SDK only would be appropriate for cardboard-like HMD (smartphone)? (video-/audio stream decoding/playback, sending device orientation)

Maybe a generic part (limited functionality e.g. 3DOF HMD only) of ALVR should be built-up with Android SDK and only Quest-specifics (handtracking, 6DOF, Controller Tracking) should be built-up with oculus mobile SDK and both parts connected by an interface to get full functionality for e.g. the quest. Some kind of modular software design for specific devices. This could consider compatibility to all related android devices (e.g. smartphones, stand-alone HMD) and maybe make it easier to adapt/optimize ALVR for other upcoming devices...

[zmerp]

ALVR uses the Oculus SDK for frame submission, for obtaining controller and head tracking data. The head orientation must be obtained from the Oculus SDK because it gives also the exact timestamp of the data, that is used to make the asynchronous timewarp work correctly. Any other library used is part of the Android SDK. The long term goal would be to use only platform independent libraries (no android, no oculus), but if I will still be working on this, I will get to that no sooner than in a year.

[3unjee]

@MPfaffe As far as I can tell there is room for coming up with a generic VR streaming implementation that implements generic functions, ideally both for the PC server and the android client. Then the idea is to build the OpenVR driver and the Android client on top of these classes. That's what I'm going to try with HEVR -> http://omega.gg/HEVR But it's a long journey, so in the meantime ALVR is going to deliver the best experience for the Quest headset + Windows PC.

[craig4096]

I believe the original ALVR server implementation was based off of valve's 'virtual display' example which can be found here: https://github.com/ValveSoftware/virtual_display

There's a pdf in this repository that explains a lot of what goes on in the alvr_server code.