search

polygraphene / ALVR

ALVR is an open source remote VR display for Gear VR and Oculus Go. With it, you can play SteamVR games in your standalone headset.
MIT License
1.82k stars 247 forks source link

Improving ALVR: Help Wanted! #482

Closed JackD83 closed 3 years ago

JackD83 commented 5 years ago

Hi fellow developers,

after trying ALVR for the first time, I knew I needed my own quest and wanted to get involved. The past view weeks, I dug trough the code and try to understand it. My thanks go to the original author, the encoding parts are just crazy complex and still beyond me.

My first results where some diagrams of the code: https://raw.githubusercontent.com/JackD83/ALVR/master/ALVR_classes.png https://raw.githubusercontent.com/JackD83/ALVR/master/ALVR_Sequence.png

I started working on the parts I understand and where I can be of some use. I started with the current master branch, but I noticed a regression in visual quality using h265 on NvEnc after commit f7e91d6ce6140dfb3756e1f848dd84a25b229e31 I could not find or fix :(

The image gets blocky on some parts and recovers a few seconds later. f7e91d6ce6140dfb3756e1f848dd84a25b229e31 does not have that problem.

That said, I started to work using that commit and the alpha 5 client.

You can find the result here: https://github.com/JackD83/ALVR/releases/tag/eV1

-fixed tracking on latest oculus quest update -fixed controller rotation -fixed controller position offset

It seems that the quest uses another origin for tracking the controller than the touch controller on the rift s. I tried to fix that offset by hand, which looks OK. Problem is the rotation speed and rotation acceleration. If these are reported for a different tracked point, they should be wrong?! This could be the cause for some of the controller shaking. Any thoughts?

On the rendering part, maybe its possible to implement this: https://www.cl.cam.ac.uk/research/rainbow/projects/trm/

No easy task :P

SSSTREDDD commented 5 years ago

works great, thank you for sharing your knowledge and build.

canalaiz commented 5 years ago

Confirming this works really good!

beyhes commented 5 years ago

Thanks man !.. I just registered github to be able to thank you !.

JackD83 commented 5 years ago

Found the error for bad image quality on the master branch. It was caused by using intraRefresh for the nvidia encoder.

New build based on master can be found here: https://github.com/JackD83/ALVR/releases/tag/eV2

canalaiz commented 5 years ago

@JackD83 Tested with Pavlov, everything works smooth as butter!

Thank you for all your efforts, unfortunately I'm not able to contribute properly on this other than providing feedback :P

artriedes commented 5 years ago

This is much better than the official version. Particularly the images getting blocky when screen contents are changing fast in the official version. That is totally fixed in this version. Now gameplay is smooooth :). However, I dont get why the controller rotation was changed. To me the official version seems to have the controller rotations accurate while this version seem to have broken it. Perhaps there is a way to make the controller rotations optional?

JackD83 commented 5 years ago

@artriedes do you have the latest SteamVR? The rotation and offset are nearly exactly how the real controller are oriented. In e.g. Space pirate trainer, the guns are correctly aligned

artriedes commented 5 years ago

Hi Jack, thanks for the quick response. I have the latest SteamVR Beta update version 1.6.10. Could it be a problem with the beta version? I can try opting out from the beta and check if that fixes the issue.

beyhes commented 5 years ago

Hi Jack this is perfect.. Is there any way to improve resulotion ? When I change from ALVR it does not seem any difference

artriedes commented 5 years ago

I just tried running Space Pirate Trainer with your version. As I expected, the guns are pointing down when I try to aim forward. And anyway I can clearly see as soon as I enter Steam VR that the controller angle is off by 30-45 degrees in your version. Then I tried the official version again, and the controllers are perfectly angled, but the blockiness problem is back, which you fixed in your version.

pvtpaula commented 5 years ago

Hey Man just registered, to post that.

  1. The contoller angles are finally fixed, Thank you!!! Every game had such a strange controller Offset i could not change it, even with the advanced settings.
  2. The picture quality is noticably better, and stable. Even while occuring packet loss, witch also seems to be reduced.
  3. The BAD parts. It feels indeed buttery smooth, but more like wading through butter, I have inconsitent latency 70-90ms where as i had 60-70 before, Every movment exept head movment feels really off.
  4. Seems like th 'predicting' Part of the Soft is not working at all.
  5. Every now and then i run into heavy latency problems where coding and encoding latencys goes up to around 100.200ms. With strong stutters. Almoust make you barf.

Sorry for the bad englisch, and that i can't help in programming, just thoght i'd share my experience.

artriedes commented 5 years ago

I dont understand why im the only person for who the controller angle is broken. Could it be because im using steamvr beta?

JackD83 commented 5 years ago

First of all: I did only change a few lines of code. All credits go to @polygraphene !

I have to working setups here (with different quests) without any unexpected problems. Here are some notes that may help you:

Someone did a few blogpost on some of the points: https://imaginevr.home.blog/author/imaginevrresearch/

pvtpaula commented 5 years ago

@artriedes I can't say, cause on the main version, everybodys just sayin 'well my orientation is fine' and i was reallly, really happy to see someone having the ability to help me with that problem. I'm using ALVR to stream SteamVR 1.5.16 (on win7) to a Quest. Mabe that helps.

@JackD83 Yes of course 'all hail to @polygraphene ' !

Thanks for the tipps, although i am aware of a few of those, some are very welcome. I am not running the most beeffy PC, but before i tryed your settings, it had run well enought. My biggest problem was that my controllers were missalingend. A bit of packet loss here and there. But less of a framedrop problem (after 45min 0-1 dropped frames)

I dont know how you did it but, man, the picture i get is so crisp now, even with lower bitrates, were Skyrim was pretty blotschi before that. My problem is that the delay of the controllers is very nauseating for me. So i tryed around a bit in the options, but to no avail. I had to go back to the main version of ALVR to convince myself i had not fried my PC by any chance.

After i tryed yours and the original, I would say for some reason your version does, in my instance, not compensate for controller movement and is thus, very smooth. But the controllers are always dragging behind your real hands. Which can be dizzying and is not good for games where you need somewhat fast reflexes. So i be staying with the main version, even with the tilted, jittery Controllers.

And now for the big BUT! Those few tests opend my eyes to what could be possible with the lag-compensation, and i really would like to see somewhat of a scale implemented, were you could set the intensity of the lag compensation. Just to be able to dial it down if you wanna play something that requires a bit more precision. Although i have no idea if thats even possible or how much work that would entail. I think that could be something worth while.

DannyDan commented 5 years ago

just tried V2 with the Quest and I can confirm, that the controller movement is not so smooth as in V1. but the rotation looks good.

rubgonama commented 5 years ago

@JackD83 Hey. I've been testing your build, and it works almos flawlessly, it's a good improvement indeed.

Also, dunno if you know how could you "fix" one thing that happens in Transference and dunno if it's something which happens in other games or not. On Tales of Escape, there was a problem which made you not being able to "pick up" things, but that's fixed right now. But the Transference problem is still there. Basically, what happens is that, in order to move, you gotta press the stick and while pressing it, move it to the direction you want to move, making it almost unplayable as it is really uncomfortable. My guess is that it has something to do about the game detecting it as a "trackpad", and it detects you as "touching" the trackpad when you press the stick.

Do you think that's something fixable on ALVR's end?

Thanks for all the work!

FingrMastr commented 5 years ago

I have an Oculus Quest incoming, so in the meanwhile, I installed this version on my Oculus Go just to see if it works correctly. I must say that smoothness is improved, and also the controller is very smooth in movement. But I have a similar problem to @artriedes : my Go controller was working fine with the main version of ALVR. With this version there is a misalignment from my real hand. The virtual controller is heading much higher, 20 or even 30 degrees higher.

JackD83 commented 5 years ago

I'm sorry to say that I target exclusively the Oculus quest. I have no other hmd and even removed the support for other headsets to get a more cleaner code base.

I only changed and reimplemented things concerning the tracking and controllers. There are many parameters that influence the movement and can be changed:

I fixed the tracking point problem btw. Angular acceleration and linear acceleration of the controller where set with the wrong coordinate reference (controller instead of world) and needed to be transformed back. This fixed some of the wobble shaking the controller. Unfortunately, controller is still lagging :(

All development is done in my experimental branch: https://github.com/JackD83/ALVR/tree/experimental

FingrMastr commented 5 years ago

My Quest finally arrived!! What a great device! I tried immediately your V2 version. Apart from the lag of controllers (something I can live with), there is a strange problem when I use “125%” resolution. The image shows strange horizontal lines, at regular intervals... It’s something about wrong downscaling algorithm apparently? This problem is not appearing al 100% or 150%... apart from this, today I roamed a lot on steamVR ambients, and all seems completely ok. Tomorrow I will test a little more with “The lab” and some other real VR app and let you know...

Sp4iK commented 5 years ago

Hi, I'm also messing around with this release and it is working really well. But I've discovered that Quests mic is not being streamed to pc for voice chat. I see that it was an old request, so maybe this could be the opportunity to implement it?

jakefever191 commented 5 years ago

Hi, I'm also messing around with this release and it is working really well. But I've discovered that Quests mic is not being streamed to pc for voice chat. I see that it was an old request, so maybe this could be the opportunity to implement it?

i would love to see the mic working its the only thing i wish they had

DannyDan commented 5 years ago

@jakefever191 as a workaround I use a bluetooth headset connected to the PC. works great.

FingrMastr commented 5 years ago

@JackD83 I tested more deeply and the problem of the horizontal lines is always present with the setting video 125%. Every fixed number of lines, the image looks “breaked” horizontally... any other user experimented this problem?

FingrMastr commented 5 years ago

Ok some more investigation on this "horizontal lines" problem. I installed original master branch 1.4.0A5 version of alvr and on 125% it shows exactly the same problem. So indeed there is something in original code that's causing this problem. I noted, however, that Oculus Quest resolution reported in ALVR is 2432x1344, but real display resolution of Quest is 1440x1600 per eye, so 2880x1600. If I set ALVR on 100% the SteamVR image is very very blurry, much worse than native Quest apps. On 125% setting ALVR is outputting 3040x1680, so a little higher than native Quest res, and in this case SteamVR image is crisp, completely comparable with native apps. However, on 125% there are these "lines", like "cracks" in the image. It seems that in correspondence of these lines a line or two of pixels is missing. If I move slowly the head up and down I can clearly see that. It's very annoying indeed, because 125% is targetting my GPU performace sweet spot (obvious since it is nearly native res). If I set 150% the image is even crisper but I have some stuttering in some games.

I hope this info is useful...

JackD83 commented 5 years ago

@FingrMastr 2432x1344 is what the quest reports as resolution trough the api. I can manually set it to 2880x1600 and it works with that resolution. I notices the lines and they seem to be missing pixels. They only occur on the quest, the send video is fine. I took a quick look at the rendering, but its still beyond me.

I have no idea why the high res video is send to the quest anyway. Should't it be enough to render in high resolution, do the down scaling on the PC and send only a video in the native resolution?? So many questions, no answers :( (yet)

FingrMastr commented 5 years ago

Discovered a minute ago that the lines are showing only in H265. In H264 the image is perfect.

andy-d1969 commented 5 years ago

1216x1344 per eye is the default resolution used by the Quest. You can change it by entering adb shell setprop debug.oculus.textureWidth 1440 adb shell setprop debug.oculus.textureHeight 1600 while your headset is connected to your pc. It will be detected by ALVR as the 100% resolution and used by Steam VR after you restart it.

Mario-119 commented 5 years ago

I'm using this build and for some odd reason it's dropping 1000s of packages every few seconds. I'm right next to my router (which is a Nighthawk so fairly high end), and I'm using 5GHz 802.1ac WiFi. My PC is also wired directly to the router, and I have pretty fast internet, around 200mbps.

I've tried fiddling around with settings to no avail. What settings are you guys using?

FingrMastr commented 5 years ago

Internet speed is not going to make any difference... it is a local WiFi affair.

Some things to check:

  1. ALVR suffers a lot of the WiFi is crowded with other devices. Check it.
  2. Check the WiFi channel of the visor connection, and if router has this ability, use a dedicated channel.
  3. Avoid mixed WiFi band and protocols, and use only 5ghz ac protocol.
  4. Eventually, use a separated 5ghz WiFi access point for visor-pc connection. No need for a router, an access point is sufficient and there are very cheap ones on the market.
  5. Start with 100% res video and 30mbit-200kb settings. In this config it should be a butter, with no almost no lag and no packet loss. If you have packet loss, it’s not a matter of ALVR settings but almost certainly network problems.

Hope this helps.

JackD83 commented 5 years ago

New pre release with some changes:

https://github.com/JackD83/ALVR/releases/tag/ev3

I guess that the improvements from setting video to 125% or higher came from the rendering and not the video itself. The resolution that was set in the UI was reported to steam as the default resolution so it would add its own factor on top of it. 2432 125% (ALVR) 1.5 (SteamVR e.g. SteamVR Home) = ‭4.560‬ px and then down scaled back to the set ALVR resolution.

I separated those two. I suggest to keep the video resolution to 100% and use steamVR to use SS I updated the default internal resolution on the Quest to 2880x1600

As mentioned earlier, the missing lines at 125% are caused by the video decoder on the quest.

Again, I'just messing with the code. I have never done this before and I'm no expert. I'm open for every suggestion.

ghost commented 5 years ago

I'll get a cheap 5ghz access point. 100mbps ethernet speed and 450mbps 5ghz speed. From what I've read online, I assume the real speed of the access point will be around 170mbps. I tested the speed of my pc's 5ghz wifi hotspot, it was roughly 90mbps/12mb/s. Only began getting issues from high bitrate at 50mbps in alvr.

@JackD83 Microphone support would be awesome. I looked around the source code of latest nonforked version and android development pages a while ago. Sending the mic data is probably the easiest part, though, but the reality is I don't have anywhere close enough experience to really help with development and cuda refused to work when compiling. I did have the correct version, can't remember what the error exactly was anymore. And the latest release from you seems to have fixed some quality issues I was having, before (on ev1) everything looked way more like a low quality youtube video despite having a good bitrate.

FingrMastr commented 5 years ago

@JackD83 it’s not completely clear to me how you separated ALVR upscaling from steamVR upscaling...

Are you saying that I cannot no more choose upscaling from ALVR ui and it will always be 2880x1600? Is the selector in the ui gone away?

JackD83 commented 5 years ago

Its hard to describe the process. I made a diagram to explain it a little better. here

You are now able to set a resolution for rendering and one for the transmitted video. For example, you are now able to render at full resolution of 2880x1600, but send only a video at 50% percent of that. This gives you a better image quality at lower encoder/decoder delay and less bandwidth. Or you can set only the rendering in steamvr to 150% and transmit video in the native resolution. The choice is yours. There "should" be no advantage to use higher video resolutions anyway.

LAP87 commented 5 years ago

Very nice diagram, thanks for the explanation, can't wait to get home and try it out!

Mario-119 commented 5 years ago

@JackD83 I don't see multiple settings for resolution in your latest build, only a single option. I looked at the diagram, but I'm still not sure to what you're referring too.

JackD83 commented 5 years ago

The second option is in SteamVR -> Settings -> Video

Digeeboy commented 5 years ago

Now that Quest supports 60Hz could this be an option you could incorporate in your client?

Mario-119 commented 5 years ago

There's already an option for this in the Other tab, it's called Force 60Hz.

Digeeboy commented 5 years ago

That doesn't seem to work. .. with the option selected it says connected at 72Hz. It's definitely doable as VD has recently added it.

JackD83 commented 5 years ago

The checkbox did nothing. I used it to set the refresh rate to 60Hz and its working as expected. I advice against using it, makes me a little sick :P

Digeeboy commented 5 years ago

How did you set it to 60Hz?? Steamvr reports 72Hz whether or not the 60Hz option is selected in your app.

JackD83 commented 5 years ago

Sorry I was not clear enough: I fixed it in the code. Everything was already implemented and I added 3 lines of code to make it work. There is no release yet!

pvtpaula commented 5 years ago

I wanted to try your newest endeavor. But it immediatly killed my Visual C++ runtime. Could it be because of my old build with Win7 or did i do something wrong?

JackD83 commented 5 years ago

I never tested it on Win7, sorry. Support for Win7 ends in a few month anyway, so time to upgrade! I think there are still a few methods working to upgrade for free

JayTee33 commented 5 years ago

I've noticed that sound seems to be way out on this build? I didn't time it but it must have been no less than 1 second.. Everything else seemed okay though.

Although I can help but feel 100% (ALVR) + 150% SS (Steam) is looking as good as 150% (ALVR) + Auto SS (Steam) in these experimental builds. Just seems to be a lot a aliasing issues.

But none the less, frames haven't missed a beat yet!

JayTee33 commented 5 years ago

I've noticed that sound seems to be way out on this build? I didn't time it but it must have been no less than 1 second.. Everything else seemed okay though.

I take it back, must have just had a sketchy instance.. played again today and sound was 100% on point.

JackD83 commented 5 years ago

Released ev4 with microphone support:

https://github.com/JackD83/ALVR/releases/tag/ev4

Sp4iK commented 5 years ago

Released ev4 with microphone support:

https://github.com/JackD83/ALVR/releases/tag/ev4

Hi JackD83, I've installed your latest experimental release with CABLE driver. Thing is, I'm not sure if the whole input/output configuration is done correctly as on a first try on Steam chat it coupled and I only could hear a sound getting louder. Then I saw I had CABLE Input as output stream and CABLE Output as mic. I changed it to be my default output and left CABLE Input as mic but can't hear myself on chat voice input test. How should it be configured?

JackD83 commented 5 years ago

Did you get the request to access the mic on your Quest? You need to set CABLE Output as your default Microphone and something other as CABLE Input as your default audio device!

Sp4iK commented 5 years ago

Did you get the request to access the mic on your Quest?

Yes and accepted.

You need to set CABLE Output as your default Microphone and something other as CABLE Input as your default audio device!

Is this on system audio configuration or in ALVR? I've actually set both (system & alvr) audio outputs to my default audio output device and mic to CABLE Output but can't hear myself on voice input testing. I'll check it further when at home.