Closed Kaktus224 closed 11 months ago
kisak-valve
commented
11 months ago Hello @Kaktus224, the previous evaluation of this question at https://github.com/ValveSoftware/Proton/issues/1493#issuecomment-422022116 is still relevant.
Closing as a duplicate.
mdrejhon
commented
10 months ago Thought exercise, @kisak-valve. I understand that Valve is trying to work on a standalone headset, which I will call "Deckard" as per known publicity. I am also making the assumption it will be ARM-based. I believe I have an engineering solution, that might unlock the viability of this feature. Some Quest 2 streaming apps implemented a rescue reprojector for fixing WiFi PCVR stutters, and that might fix Dynarec emulated PCVR stutters too.
(Running a PC VR application, Steam Deck style, completely inside Deckard VR headset, on its ARM GPU).
What if the upcoming Deckard VR headset could run some older PCVR games completely in-headset? There's a new solution to make this performant enough; see below:
Basically, it would have to be "Steam Deck approved" processes, to whitelist PCVR games for installation into a next-generation ARM headset (M1-and-better league, including the newer chips by Qualcomm). An FSR/XESS/etc system built into the graphics driver would help add framerate, with a 3dof reprojection scroller for headturns (prevents old PCVR games stutter).
Maybe not yet, but perhaps with additional workarounds.
We already have 3dof headturn stutter-prevention reprojection in PCVR streaming even when the PCVR game stutters (and ASW / ASW2 is disabled!). This is simply moving 3dof headturn reprojection tech to headset-side, as Quest 2 already supports client-side 3dof reprojection of VR streamed gaming.
IMPORTANT! CRITICALLY IMPORTANT SCIENTIFIC-TEST ADJUSTMENT: Make sure your PC's video codec priority is MUCH higher than the GPU render priority, because you don't want the video codec to stutter during this stutter-experiment. Use appropriate power-user utilities or developer software, to ensure that this scientific test prevents assuming all stutters are cuz streaming (Hint: it's not). The PC-side 3dof reprojector even stutters where the Quest 2 side headturn does not. That's headset-side 3dof stutter recovery.
(even with PC-side ASW turned off, using a utility such as Oculus Tray Tool)
One of the streamer tools had to do this due to tiny WiFi-created dropouts, to smooth inferior WiFi. The Quest 2 side 3dof reprojector "smooths over" those dropouts. What this means, this accidentally provides the breakthrough solution making PCVR usable through ARM dynarec stutters. (dynarec = dynamic recompilation, which means software is slow/stutters at first, before running smooth)
Now, instead, you port this into Proton instead, so you've got a headturn-reprojecting graphics driver between the Deckard display and the PCVR x86 game running in Proton ARM emulation layer. Headache performance problem solved!
While this is not 6dof, this is CLIENT-side 3dof reprojection -- to solve PCVR headturn stutters (even when the PC-side FAILS the 3dof headturn reprojection!!. All this client-side 3dof reprojection technique does is client-side spherically scroll the last successfully-streamed frame, in sync with headturn, during delayed frames (WiFi reception, etc).
Voila.
Headturn stutter problem solved independently of PCVR performance problem!
So we already have already-on-the-market headset-side reprojection that destutters a remote PCVR game, at least for headturn moments (not 6dof reprojection of lateral movements, but it permanently eliminates the stutters of the framerate-critical headturns regardless of the PCVR framerate!)
* I still recommend a powerful GPU (RTX 3000 series or better) for the fantastic video codec for PCVR streaming, and then artificially kill your GPU framerate without killing video encoder framerate, to successfully conduct this experiment properly. Obviously, this suggestion (for the experiment) is ONLY because of video streaming (which you won't need to worry about when running standalone x86 PCVR in a hypothetical ARM Proton layer on Deckard).
Obviously the translation stutters still exist (e.g. hand movement stutter) but that's way less nauseating than a driver-side headturn reprojector that is already being used by Quest 2 for streaming PCVR.
As a software developer myself, the architecture would be:
[PVCR game running Steam Deck style on Deckard] -> [Wine+ARM] -> [Deckard Proton Driver] -> [Outside Wine] -> [Native Driver + FSR2 + 3dof Reprojector] -> [Deckard Screen]
The fallback rescue 3dof reprojector would be close-to-metal! Much like how close-to-screen Quest 2 reprojector for perfect streaming from remote GPU stutters. This is the breakthrough solution for ARM-DynaRec-stuttery PCVR games. By architecturing this way, you guarantee stutter-free headturns, no matter how much your Proton ARM emulated PCVR stutters! Magic.
Since 3dof reprojection is simply a spherical scroll and is constant-GPU time (roughly 1-2ms on Quest 2), you can simply use your low level kernel GPU scheduler, and kernel-level priority (it's outside Wine so you got control). To prioritize that final important rescue reprojector stage just before VSYNC. For example, in a 11ms budget of 90Hz, you can dedicate 9ms to the stuttery PCVR execution, and 2ms for the de-stuttering in-headset 3dof reprojection. Whatever you do; you can performance-profile and use maximum priority for whatever you do outside the Wine layer, to solve all those ARM-x86 dynarec stutters for headturns.
You might not apply this to Alyx-level (you're probably working on a native port of HL:Alyx for Deckard, I bet)... ...But a giant amount of PCVR content could still work fine with the (1)+(2)+(3)+(4)+(5) technique. As long as you use a Quest-3-era CPU/GPU (or better) for Deckard.
This is existing tech: Quest 2 GPU rescuing PCVR shortcoming. We're just repurposing the client-side headturn stutter-rescue algorithm from WiFi/remote-GPU stutters, towards ARM/Proton emulator/dynarec stutters.
The PCVR side is completely unaware of the display-side co-reprojector yet it works perfectly for recovering VR streaming stutters, which I feel is proof that this concept should make ARM Proton practical. Stutters are bad, but ARM dynarec stutters are temporary anyway, and you've now hidden that completely from headturns, with the close-to-display-metal fallback 3dof reprojector.
Wouldn't this solve a lot of problems and allow untethered PCVR, for retro-VR perservation. There's at least a hundred pre-2020-era VR games that should be able to run on Steam Deckard in their original x86 executables, if using the Steam Deck approval workflow, COMBINED with the (1)+(2)+(3)+(4)+(5) trick concurrently.
The breakthrough that may occur from simultaneously combining (1)+(2)+(3)+(4)+(5) to make ARM support for Proton performant enough to allow Deckard to double as a Steam Deck for x86 VR games? Maybe not a big market, but I'd like to have PCVR preservation incubated now, in a potential future where people may end up playing on standalones in the post-Deckard era.
Will you please reconsider this, in the light of this new information?
Wouldn't it increase Deckard sales, even if only under 10% install x86 PCVR? The razor and blades strategy; You say "come for standalone PCVR, stay for our amazing in-headset native ARM SteamVR store?" What about the developer buy-in and positive publicity? Nudge, nudge? Wink, wink?
This feature addition MAY be is worth million $ to Valve in this new marketing light?
Remember, Steam users are fiercely loyal. And Steam users are reluctant to use a new platform that can't utilize their existing games. If you can "bridge over" a few PCVR apps to run standalone, they're more likely to buy Deckard if they can run at least two or three of their existing PCVR without the hassle of reactivating their old VR rig or configuring their WiFi link to be strong. A lot of % moved from PCVR to Quest, and have let their PCVR go fallow, so it's a headache to reactivate all that config, and won't 100% all bother to buy Deckard. But with this additional incentive -- more could. And you now have an ARM ecosystem in their hands automatically. BOOM. You get to sell lots more ARM games as the 800 pound Steam gorilla. Win-win-win? No?
Have I convinced you to get the Valve Team together for a meeting to reevaluate the ROI of this feature, in the new-info light?
I am the founder & CEO of Blur Busters and TestUFO, and have an article at www.blurbusters.com/frame-gen and I'm the one who accelerated low-persistence introduction to VR (see "How Blur Busters Convinced Oculus Rift To Go Low Persistence"), with me published in a book.
There are spinoffs of benefit for all the pre-requisite spinoff features that would tie into this item. Doing this exercise creating a close-to-metal driver-based 3dof reprojector (spherical scroll compensation for headturns), could actually improve the performance of your native Deckard games (standalone ARM). See?
The rearchitecturing you end up doing to have a guaranteed-realtime-GPU-scheduled 3dof headturn reprojector (something that is already in Quest 2/3), would be universal, and benefit any native games that sometimes underperform now-and-then too. The realtime optimization needed, means it would act only on the latest previous frame (whether native ARM frame, or from Proton PCVR frame) if the new rendering is too late for its scheduled delivery to VR screen refresh cycle. So you guarantee 90fps headturns regardless of the underlying framerate of whatever, including native ARM apps standalone PCVR in Proton+ARM layer.
RESULT: Native Deckard ARM games improves!
More win-win.
mdrejhon
commented
10 months ago This is still without the rescue 3dof solution but:
"I've tried this on Windows with Virtual Desktop and I was able to get about 60fps, and you can turn on spacewarp to raise that to 90 or 120, or use Snapdragon Game Super Resolution to upscale, but unfortunately Valve doesn't support Windows properly and the GPU encoders don't work so it's running on the CPU and that takes up a lot of usage."
https://www.reddit.com/r/SteamDeck/comments/17gh0e9/steam_deck_can_run_pcvr_using_a_quest_2/
With some further native optimizations (not done by public), it could possibly get the rest of the way (GPU-based instead), like using Snapdragon Game Super Resolution on the GPU instead of CPU, and do it like as the "FSR2" equivalent on the realtime priority GPU schedule just outside the Wine layer.
[PVCR game running Steam Deck style on Deckard] -> [Wine+ARM] -> [Deckard Proton Driver] -> [Outside Wine] -> [Native Driver + FSR2 + 3dof Reprojector] -> [Deckard Screen]
Then you add the rescue 3dof reprojector algorithm that I suggsted in my previous comment;
I think that an M1 with a performance margin 3x+ of a Steam Deck x86, can probably more than overcome, especially if the tricks I suggest are utilized. Especially with a Steam Deck style wizard to babystep the end user to optimal PCVR settings for standalone PCVR on Deckard, perhaps
mdrejhon
commented
8 months ago More about the new Snapdragon X Elite coming June 2024.
The early info I keep reading on its performance, suggests It may now have the grunt power for standalone PCVR even with a ~3:1 emulation overhead, as long as at least two properly surgical optimizations:
...an external "rescue 3dof reprojection" algorithm (already in Quest 2/3 to destutter WiFi streaming. But to destutter headturns during dynarec-overheads in PCVR) ...launcherautooptimized detail level settings, much like Steam Deck's autooptimizing of apps.
Hello, I just want to ask whether making ARM support for Proton would be possible, and if so, could you please do it? I'm asking because I bought the orange pi 5b recently, and it's really powerful, but I can't run most windows games (Geometry dash for example) even with Proton turned on. I tried using Wine, but I just couldn't get it to work. Thank you for your time, Kaktus.