search

GPUOpen-LibrariesAndSDKs / Radeon-ReLive-VR

156 stars 19 forks source link

Quest 2 stream freezing #83

Closed zico-io closed 3 years ago

zico-io commented 3 years ago

So far the only game I've tested this in is HL:A, occasionally while playing the game the stream in the headset would completely freeze, while it looks perfectly normal on desktop. The only way to be able to resume playing the game is to restart the ReLive app on my quest, which will allow me to play for another ~20 seconds before freezing again.

I had this issue last night when I first tested ReLive, so I put it down and decided to come back at it today. This morning I was able to play for awhile without any issue, but when coming back to it a couple of hours later the issue came back. I hadn't made any changes in the settings between the sessions.

AMDWirelessVR.log

GennadiyAMD commented 3 years ago

@zico-io - the log shows that you're streaming over TCP. Try switching to UDP in Radeon Settings.

wessberg commented 3 years ago

I too have this problem. Stock settings, UDP, 100Mbps bit rate, Oculus Rift S headset emulation, with otherwise great network conditions.

It runs for around 10-20 seconds, and then freezes in the headset (but continues running on the PC).

One way to make this happen even sooner is by increasing the render resolution/encoding resolution. Conversely, lowering them will increase stability. But even with stock settings and 1440 x 1440 render resolution, I'm lucky to get 20 seconds of playtime in Half Life Alyx and even SteamVR home. I don't experience freezing with other streaming clients, including Virtual Desktop (which is TCP only by the way)

GennadiyAMD commented 3 years ago

@wessberg - could you please capture and attach the log located in c:\Program Files (x86)\Steam\logs\AMDWirelessVR.log? Please make sure that you make the problem happen before capturing the log.

Which GPU are you running on? Increasing resolution would not change the amount of data that goes through the network, but it would increase encode and decode time, which would cause a drop in the frame rate, but should not cause a freeze you're describing. But changing the bitrate would certainly affect the network. Does the problem go away when you reduce the bitrate to, say, 50Mbps or 35Mbps? Also do you have any kind of QoS enabled in the router?

wessberg commented 3 years ago

Let me fetch that and update this issue with the log. Until then, I can add a few details: I'm running an RX 6900 XT GPU. I do have QoS enabled in the router, but with default settings. Can this have an impact on local streaming performance?

I'm thinking that even if there's some bottleneck somewhere where do frames are dropped, it should never lead to freezing but rather stuttering or lowering of the bit rate, but something else might be going on of course.

I'll update with the logs later.

wessberg commented 3 years ago

Alright, I did some testing today under excellent network conditions, which may factor into my test results. I tried disabling QoS, but that lead to +300ms network latency, as well as constant freezes. Reverted to enabling QoS.

System specs:

Network:

All tests conducted inside Half Life: Alyx.

1440x1440_hevc.log

It would seem that HEVC encoding is the cause of most of these issues. There are very visible compression artifacts in the H.264 stream, so it would be great to be able to use the HEVC encoder, as I do with Virtual Desktop.

GennadiyAMD commented 3 years ago

@wessberg - it seems to me that the issue is not encoding, but the network. RX6900XT is a very capable card. What I see in the log is normal streaming with a rather good latency for a while and then frames stop arriving, the client is requesting an IDR frame repeatedly, which again does not get through and then streaming stops. This points to the network, not encoding. Your findings about QoS also point in this direction. HEVC vs AVC should not make a difference in this case, I think it's just a coincindence.

One setting that looks suspicious to me is the 80MHz channel. It sounds counter-intuitive, but wider channels make it more difficult for the router to find a good channel with minimal interference. This is because it bundles several adjacent channels together and needs a contiguous frequency range for them. This is great when you live on a farm with no other WiFi anywhere close, but when you have a lot of traffic from other APs, it makes the router hop channels more. I'd suggest to change this to Auto, or reduce to 40 or 20MHz if your router does not have the Auto setting. I've covered this in more detail in this Wiki article.

Also might worth trying streaming over TCP rather than UDP. This really depends on your router and is not a universal recommendation. UDP works better for most routers, but I've seen some where TCP performed better.

Hope this helps. Please let me know if it does not. Also if you have access to another router, please try that as well.

wessberg commented 3 years ago

@GennadiyAMD,

Thanks for the follow-up. Hehe, yeah, the GPU is definitely not at fault here :-)

About the 80MHz channel width: I've been experimenting with switching from 40 to 80 mhz channel widths for a while and generally found most stability (shortest ping times and fewest spikes) on the wider channels in my specific environment.

I don't have access to another router, but given that it is an ISP-provided one (Zyxel VMG8924-B10A), I might see better results by switching to a higher end solution. But I actually think my network conditions is better than most, given that I have the entire 80Mhz range free from crosstalk from other routers according to Wifi Analyzer and live in a house.

As per your suggestion, I did try switching to 40Mhz, but unfortunately experienced the same freezing with HEVC.

About the network, a few things comes to mind:

I'll try to perform a few more tests using TCP instead of UDP to see if this increases stability. But again, it feels strange that it is never able to "recover" from lost frames, especially given I don't experience this behavior with Virtual Desktop).

GennadiyAMD commented 3 years ago

@wessberg - let me double-confirm - you do not see this problem with h.264, but only hevc? A few other important notes:

wessberg commented 3 years ago

you do not see this problem with h.264, but only hevc?

Correct!

Not to put the blame on the ISP, but most complaints that we've seen so far that were resolved by changing the router have been about ISP-provided routers and setups with WiFi repeaters or mesh networks.

I don't doubt that this is correct. It looks fine spec-wise, yes, but it is probably very CPU-limited. But this is probably representative for the vast, vast majority of the potential user base of Relive VR. But I am strongly considering investing in a new router.

HEVC is designed to produce better visual quality at low bitrates, but at 100Mbps you would hardly see any difference between H.264 and HEVC. If AVC is more stable with your router, stick with it, you're not missing out on anything compared to HEVC.

I've generally experienced two things with HEVC encoding as compared to H.264, including in VD, on my old 5700 XT and the new 6900 XT:

When you try TCP, try prioritizing traffic to port 1235 in QoS -

Thanks for the tip, I'll do that!

Try going all the way down to a 20MHz-wide channel, just for the sake of experiment.

I'll do that too. That will be interesting.

The render resolution in your last experiement (3114x3264) is too high for any practical purpose. It would increase rendering time and hence the overall latency, but not the image quality as it would get scaled down anyway. Keep in mind that all resolutions in ReLive VR are per eye.

That argument is built on the assumption that supersampling/super resolution doesn't have any positive impact on visuals. Which is something I've always believed. But then I've changed my mind after playing around with super resolution on the quest with Oculus Link and SteamVR Resolution scaling and found that super resolution benefits at least the Quest 2 a lot in terms of sharpening up background visuals.

I'd also suggest to try the default settings in ReLive VR

I actually did exactly that before reporting this issue. I deleted the config file, and had a new one auto-generated. The only thing I played around with was headset emulation and video codec.

I will report back with test results :-)

GennadiyAMD commented 3 years ago

Not to put the blame on the ISP, but most complaints that we've seen so far that were resolved by changing the router have been about ISP-provided routers and setups with WiFi repeaters or mesh networks.

I don't doubt that this is correct. It looks fine spec-wise, yes, but it is probably very CPU-limited. But this is probably representative for the vast, vast majority of the potential user base of Relive VR. But I am strongly considering investing in a new router.

Actually, this particular complaint is not very common, we've only seen a handfull of these, with some being caught during internal testing. But anyway, if you end up buying a new router, one thing to look for is Gigabit wired Ethernet. There are many models on the cheaper side of the spectrum that support AC Wifi, but only 100Mbps Ethernet, like this one - stay away from these, make sure it explicitly states Gigabit Ethernet support, don't assume it has Gigabit Ethernet because it has WiFi5.

We also have not seen much benefit from the high-end AX/WiFi6 routers unless you're running multiple streaming sessions at the same time - they performed just the same for a single session.

HEVC is designed to produce better visual quality at low bitrates, but at 100Mbps you would hardly see any difference between H.264 and HEVC. If AVC is more stable with your router, stick with it, you're not missing out on anything compared to HEVC.

I've generally experienced two things with HEVC encoding as compared to H.264, including in VD, on my old 5700 XT and the new 6900 XT:

  • I get substantially less latency when I use HEVC.

I cannot comment on VD as I don't know what it is doing, but with ReLive there shouldn't be any difference in latency. I get the same numbers on my 5700XT regardless of the codec used and this is really the result expected.

GennadiyAMD commented 3 years ago

@wessberg - just one more idea to try in addition to the previously mentioned suggestions: in settings.json, there is a parameter in the "Headset" section called "DatagramSize". By default it is set to 65507, which is the maximum value allowed. Try reducing it to, say, 1024 - if this makes it work, keep increasing it until it stops working and leave it at the maximum value that works. This parameter is only applicable to UDP and is ignored when streaming over TCP.

What this parameter does is controlling the maximum size of UDP datagrams that ReLive VR sends. Messages from the headset do get through to the server as you mentioned that the PC keeps responding to the headset, if I understood you correctly. These messages are small in size. Video that goes back from the PC to the headset produces much larger messages. If QoS in the router prioritizes smaller datagrams over the larger ones, it might start dropping larger messages resulting in frame loss. Limiting the max size might help the video to get through.

No idea if this would help, this is all guesswork, but this is quite easy to try. Please let me know if this helps.

Another recommendation unrelated to this - do not force the WiFi channel to some number, let the router pick one automatically. If you force yours to use some channel with the least traffic, but your neighbors are selecting theirs automatically, your neighbor might start broadcasting on your channel creating interference, but your router would not be able to switch to a different channel because you forced it to stay on that one.

wessberg commented 3 years ago

Actually, this particular complaint is not very common, we've only seen a handfull of these, with some being caught during internal testing.

What I meant was that I would assume the vast majority of users are sitting on ISP-provided, low quality routers, so ideally we can approximate better stability even on low-end router hardware.

But I'm back with pretty incredible testing results! I actually started off with your very last tip about lowering the UDP datagram size as the first thing I did at all. And it worked shockingly well, but only when the DatagramSize is 1024.

The only thing is that the refresh rate was apparently stuck at 75 hz. Well, every time I checked, it was 75hz. Note that this has been a problem with large values for DatagramSize as well, and happened too with H.264. Any clues as to why that is?

Here's the tests I ran, in the order that I ran them. To be 100% sure, I rebooted both the PC and the Quest 2 between every test.

I see way less compression artifacts with HEVC with Relive, and the perceived latency was lower (though I need data to support that claim, but it definitely felt that way). The picture quality is acceptable. The only problem is the refresh rate as mentioned.

And, one nice improvement over VD is that there's little to no stuttering going on here.

I didn't try any of your other tips yet, but at least this seems to work.

You should consider either exposing a control for this in the Adrenaline application or adjust the size of the datagrams dynamically based on some heuristics. But so far, this is great.

I would like to do more testing in worse network conditions to see if I can run into unrecoverable errors under poor conditions, but hey, this is something!

Oh, and it would be nice if proper Quest 2 controllers was supported instead of just the Rift S and CV1, but that's a different conversation for a different time.

GennadiyAMD commented 3 years ago

Actually, this particular complaint is not very common, we've only seen a handfull of these, with some being caught during internal testing.

What I meant was that I would assume the vast majority of users are sitting on ISP-provided, low quality routers, so ideally we can approximate better stability even on low-end router hardware.

I understand what you meant and completely agree with you. I'm actually surprised myself that we haven't seen more complaints about this, but in all honesty you're maybe the fourth person who has run into this issue and complained about it. But regardless of how many people encountered this, if there's a solution, we'll solve this.

But I'm back with pretty incredible testing results! I actually started off with your very last tip about lowering the UDP datagram size as the first thing I did at all. And it worked shockingly well, but only when the DatagramSize is 1024.

Ok, this is something. This actually proves that your router is the culprit.

The only thing is that the refresh rate was apparently stuck at 75 hz. Well, every time I checked, it was 75hz. Note that this has been a problem with large values for DatagramSize as well, and happened too with H.264. Any clues as to why that is?

You're probably looking at the wrong place for the FPS - just recently another user ran into the same. From the STeamVR pop-up menu (three horizontal lines in the upper-left corner of the little SteamVR window) click on the Display Performance Graph - it will show the graph and the current FPS there. This is the value you want to look at, not the one in SteamVR Settings.

You should consider either exposing a control for this in the Adrenaline application or adjust the size of the datagrams dynamically based on some heuristics. But so far, this is great.

We absolutely will. We just needed to know what the problem was - once you know that, fixing it is easy.

I would like to do more testing in worse network conditions to see if I can run into unrecoverable errors under poor conditions, but hey, this is something!

Oh, and it would be nice if proper Quest 2 controllers was supported instead of just the Rift S and CV1, but that's a different conversation for a different time.

Quest controllers look and work exactly the same as Rift S as far as I can tell. CV1 has the ring turned the other way, but Rift S is a very close match. What other differences do you see? SteamVR Home app is a good testing tool for this (or even nothing running in SteamVR at all - I mean the transient scene they show when a game is loading, the one with the radial grid and the horizon - that shows the default controller model that comes with SteamVR). SteamVR knows nothing about Quest as it is not a tethered headset. We need to emulate something that SteamVR and the games know about, otherwise they won't work correctly. And many games have their own controller models that can look different - Serious Sam is a good example. I don't think anything can be done about this without breaking compatibility with games.

At any rate, thank you very much for your time - this is a very good discovery.

Happy holidays!

wessberg commented 3 years ago

Yeah, its great that we found a solution! 🎉

I know I keep bringing up Virtual Desktop, but that's simply whatever else I and many others are currently using for this use case, but I'd like to switch over to ReLive since I experience less stuttering with it, so that's great. But you'll have to excuse me for bringing up just a few additional references here in the next paragraphs ☺️:

SteamVR knows nothing about Quest as it is not a tethered headset

As for the Quest 2 controllers, SteamVR recently integrated support for them, and with VD it is already using the Quest 2 controllers inside the SteamVR home environment, so this should be technically possible. It works great, too. Here's the release notes.

You're probably looking at the wrong place for the FPS - just recently another user ran into the same. From the SteamVR pop-up menu (three horizontal lines in the upper-left corner of the little SteamVR window) click on the Display Performance Graph - it will show the graph and the current FPS there. This is the value you want to look at, not the one in SteamVR Settings.

Yeah, in the Display Performance Graph, it does show 90 hz and it looks great there.

Is there any way for you to "seed" the one in the SteamVR settings with the correct refresh rate? When I use VD, it shows 90hz. I can imagine this might not be the last time you get this question! I checked because it didn't feel like 90hz, but the frame times look excellent in the performance graph, so I guess whatever lag I was experiencing came from something else. But I'll take that over constant stuttering any day.

Happy holidays!

Happy holidays to you too! ☺️

GennadiyAMD commented 3 years ago

Yeah, its great that we found a solution! 🎉

I know I keep bringing up Virtual Desktop, but that's simply whatever else I and many others are currently using for this use case, but I'd like to switch over to ReLive since I experience less stuttering with it, so that's great. But you'll have to excuse me for bringing up just a few additional references here in the next paragraphs ☺️:

Oh, no worries about bringing it up - we do not consider VD a competitor since ReLive VR is a free value add. We're happy when people use any software on our GPUs :-)

SteamVR knows nothing about Quest as it is not a tethered headset

As for the Quest 2 controllers, SteamVR recently integrated support for them, and with VD it is already using the Quest 2 controllers inside the SteamVR home environment, so this should be technically possible. It works great, too. Here's the release notes.

Yes, you are correct, my bad, thanks for pointing this out. This is very recent (Dec 17). We'll look into this, although Quest and Quest2 controllers are very similar, almost identical to Rift S.

You're probably looking at the wrong place for the FPS - just recently another user ran into the same. From the SteamVR pop-up menu (three horizontal lines in the upper-left corner of the little SteamVR window) click on the Display Performance Graph - it will show the graph and the current FPS there. This is the value you want to look at, not the one in SteamVR Settings.

Yeah, in the Display Performance Graph, it does show 90 hz and it looks great there.

Is there any way for you to "seed" the one in the SteamVR settings with the correct refresh rate? When I use VD, it shows 90hz. I can imagine this might not be the last time you get this question! I checked because it didn't feel like 90hz, but the frame times look excellent in the performance graph, so I guess whatever lag I was experiencing came from something else. But I'll take that over constant stuttering any day.

The one in SteamVR settings doesn't show the live value, it shows what the headset is capable of, as far as I can tell. It would still show 90Hz there even when the real frame rate drops to 60Hz. Problem is that ReLive VR supports multiple headset types and we don't know which headset will connect when SteamVR starts. VD has a separate server that handles connections from the headset, running in a separate process and it starts SteamVR after you have connected the headset, so the headset is known at the time of SteamVR startup. We chose this design for its simplicity - you just start SteamVR and run the app on the headset, a number of people have commented that this was easier to use than VD or ALVR.

At any rate, thank you very much for the feedback!

Happy holidays!

Happy holidays to you too! ☺️

Thanks!

ftp-vr commented 3 years ago

Lol only 4 people. There is a ton of people with the seam freezing problemes. and me too!

wessberg commented 3 years ago

Lol only 4 people. There is a ton of people with the seam freezing problemes. and me too!

Its great that you're letting your voice be heard, but as a maintainer of several open source projects myself, I know just how frustrating it can be to receive an issue or a comment on one with no details to act upon. It will be of so much more help to the maintainers of this project if you can provide a log file and as many technical details as possible.

@GennadiyAMD, by the way, I'd like to share some further insights with you: I went and bought an expensive and generally well-recommended router for this kind of scenario: the ASUS RT-AX86U, which is a premium WiFi 6 router with DFS-channel support. I've experimented with it on 20mhz, 40mhz, 80mhz, and even 80+80mhz channel widths with ReLive. From that, I've learned the following:

It is definitely usable to me, however, and it's great to have this built-in to Adrenaline and free of charge. Nothing here needs to be acted upon, just wanted to share my experiences with you.

GennadiyAMD commented 3 years ago

@wessberg - thank you! :) @ftp-vr - we're always happy to help, but would need more technical data, such as logs (c:\Program Files (x86)\Steam\logs\AMDWirelessVR.log) and a more detailed description of the problem you're facing. Sarcastic comments are fun to read, but they don't provide enough data for a helpful advice. Have you tried any of the suggestions made to @wessberg or anything described in the Troubleshooting section of the Wiki? As to how many people actually have faced this problem - we only know about those who brought it up here. This is exactly why the Troubleshooting section of the Wiki exists. Perhaps some have been able to resolve those issues themselves without reporting it here?

ftp-vr commented 3 years ago

only hevc codec seem to work and i have to put all the specs to the max. i sill have laging time to time but at least i can play. my router is ax3000 my gpu is gtx 1060ti. cpu i59300h i have ssd. my computer is wired with ethernet and my headset is on the 5g at 1200 mbs. i have to open the game using steam vr. if i open it with the app itself its freeze and crash. if i select auto or h264 it freeze and crash. if i dont put the specs to the max it freeze and crash

GennadiyAMD commented 3 years ago

@wessberg - datagram size makes some difference, but not night and day, of course. Each UDP datagram is a separate message routed separately, there's no guarantee that they would arrive in the same order they were sent in, so to send a continuous stream over UDP requires additional headers in each datagram to reassemble the original stream. Increasing the datagram size minimizes the number of datagrams needed to send the same payload, thus reducing the number of those headers, which are overhead. But the difference is not huge. The optimal channel width depends on many factors, including how polluted your environment is in terms of RF. If you don't have a lot of traffic from the neighboring APs, 80MHz might work better, of course. As a general rule, all routers that we have tested worked best with their default settings (which is usually Auto for channel width). There's not much value in trying to match the native resolution of the panel in the headset. The reason is that you can't avoid scaling because there's a lens distortion correction while the game renders to a rectangular surface - most pixels will be altered by that regardless. The actual image on the LCD panel does look like the one in the infographics in the top-right corner on our SideQuest page - with the rounded corners with the barrel distortion introduced to compensate for the pincussion distortion of the lenses. Increasing the encoder resolution will increase latency as encode time is proportional to the resolution. However, it has no effect on the network performance since the stream would still be encoded at the same bitrate. But higher bitrates would certainly contribute to network hiccups. I guess you'd have to find the right balance between image quality defined by the bitrate and the number of hiccups that looks best to you - this is very individual and depends on the content too - hiccups would be more noticeable on content where the image moves on its own rather than in response to head movements (like car racing/rollercoaster style games as opposed to shooters). Btw, when you change the bitrate in the UI (both Radeon Software or Web-based) - it is applied immediately, no need to restart SteamVR.

At any rate, thank you very much for all of the info - I will look into whether anything in the Wiki can be updated based on your findings in the short term and whether anything can be auto-configured in the longer run.

GennadiyAMD commented 3 years ago

@ftp-vr - your GPU is a GTX 1060Ti? But that's an NVidia product, ReLive VR is a product by AMD and runs on AMD GPUs only. Are you sure you're running ReLive VR and not ALVR or Virtual Desktop or something else?

ftp-vr commented 3 years ago

yes i made a mistake. i was trying to find a solution for virtual desktop without a cable link for occulus 2. sorry. i tought that forum was about that too.

ftp-vr commented 3 years ago

and there is many people like me experiecing the same troubles. so this is why i was hurt by "only 4 people" sorry again

GennadiyAMD commented 3 years ago

@ftp-vr: Virtual Desktop is a 3rd party product AMD has nothing to do with. Please refer to their site for support.