Video filtering options/togle

[Kuuko-fokkusugaru]

First of all, I am not programmer so if what I ask or suggest sounds extremely stupid please be nice with me lol.

So as we all know, pretty much every single video it's "filtered". I am not sure which filter it is, bilinear, trilinear, I guess that may depend on each player. My concern it's that I feel vr would benefit from no filtering at all. I am talking about the filter applied to played videos so if you stream at less than 100 % everything it's extremely blurry. This would come handy for FFR too. If you have tried some games on Quest, the FFR makes everything pixelated around edges but I feel that this is much pleasant than a blurred view since that produces eye strain and fatigue.

The ideal case would be having different filtering options, from smoother ones to sharpen ones and even no filter at all. I am not sure whether this would have a negative hit on performance at time of the client decoding. One would guess that not having to filter would alleviate things but, again, I am not programmer. Either way I think that a not filtered video would deliver a better experience. This is specially noticiable when you try to see the steam desktop view from vr, you can't read anything even using 150% of resolution and sending the video at 150MB/s.

I am also aware that no filter could introduce aliasing but I am fine with that, it would be optional after all.

Aditionally, as suggested above, FFR (which I couldn't try yet due buggy release) could have it's own filtering options too on the ALVR side. I have read that there is a couple of blurring filters for later on give a pass with a sharpen one. While it looks good, some of us may prefer to have it unfiltered if we gain some sharpness on the sweet spot.

I guess the main issue here it's that, while it looks good on a flat screen, in vr any bit of blurriness just starts to cause fatigue and eye strain. There is a similar effect on games on CV1 that used an extremely aggressive FXAA and turning it off was more pleasant for several people.

OK now it's when you guys comment and explain why my suggestion it's extremely silly 😋. I am really sorry if my suggestion only worked to clutter the page in such case I could delete it. Thank you for your hard work Btw 😉.

[zarik5]

Don't worry, no suggestion is stupid if it is sincere 😉 ALVR does not do any unnecessary filtering, at least not intentionally. If you show an image that is smaller or bigger than the display, the GPU tries to fit it on the screen an that can look a bit blurry. GPUs have an option for making the image look sharp anyways, and that is used for example in games that have pixel art, but I can tell you on ALVR this would make the image look horrible. On a tethered VR headset multisampling could help make the image less blurry if it has not the exact resolution of the screen, but for ALVR, when you decided the video stream resolution, there is little we can do to make the image look sharper. The current FFR implementation does not have any deliberate blurring or sharpening, but despite that the image looks a bit blurry (I missed something in the math). Not too long from now I should write another FFR implementation that have no blurring whatsoever.

[Kuuko-fokkusugaru]

Oh I may have missed something. I didn't mean that ALVR applies filtering I meant the client on Quest itself, the "video player". I am sure the video it's fine, it's just at time of playing it. For example, you can do similar stuff when you play a video with media player classic and tinker with codecs and filters. You can define how the image will be filtered (specially useful for content that it's of lower res than your screen). This is why some videos looks a bit more blurry on players that uses internal codecs than media player classic even when they are 1080p.

So my suggestion was more regarding the apk itself, to have different filtering systems and the option of turning it off as well (from ALVR). I am 95% sure that the image would be more pleasant.

As it is right now, I can see sharper image on CV1 than Quest even tho Quest have a noticiable higher resolution and I stream at extremely high bit rates so I have no compression artifact. And may be the same reason why going beyond 100% resolution doesn't seems to make any difference.

Btw my comment regarding FFR being blurred and sharpened was just from reading your shader explanation example but I guess you do it differently here ☺.

Well I guess if it can't be done it can't be done but if you ever get curious about it and there is a way for the apk to disable/change the video filtering then give it a try. I feel like virtual desktop does this and that's why it's image it's so sharp but I may be wrong and maybe it's something else. Thank you anyway and can't wait for the next release fix to try FFR 😊 ♥.

[zarik5]

With ALVR I meant the combination of ALVR server (the driver on PC) and ALVR client (the apk). The apk also does not do any unnecessary filtering. I don't know, I don't have a wired headset, so I can not see the difference myself.

[Kuuko-fokkusugaru]

Well there is clearly some filtering just like when playing any video otherwise if we set it to play at 25 or 50% resolution we would get an extremely pixelated image but instead we get an extreme low res but blurred image. What I am talking it's similar to the difference of an image being zoomed in on an image viewer showing blurriness vs doing it in Photoshop that shows the actual pixels. So I am not saying that the app applies a filter because it may be how all videos works internally, and more like forcing it to display the image as if it were dot/pixel perfect or kinda like Minecraft textures if that makes it easier to understand. I am sure this would increase image quality sharpness. Or at least it would be something worth trying. This would open doors for even different settings like custom sharpen filter, brightness, contrast, etc since we are seeing a video after all so we could have some little extras that we can't get directly in game.

[zarik5]

Ok, yes, ALVR does some filtering, but for me it didn't count because it is the default behaviour of GPU rendering (and yes, ALVR client also need to render a rectangle containing the image rendered on server). I did look up what the client does and I saw that for resolutions above 100% it uses bilinear interpolation and for res below 100% it uses nearest neighbor (i.e. no filtering). I rebuilt ALVR ev4 modded so that the behavior is inverted (res > 100% -> nearest; res < 100% -> bilinear). I let you judge which version is better 🙂 app-release.zip. An alternative could be also trilinear filtering (would work only for res > 100%) but it requires the computation of mipmaps for every frame. Or for res < 100% there is also bicubic filtering which preserve the contrast a bit better and does look less blurry. If you want to explore how texture rendering works you can google "min filter" and "mag filter".

[Kuuko-fokkusugaru]

If below 100% isn't filtered and uses nearest, shouldn't look pixelated? I will give it a try but I do know that for smaller res, bilinear always deliver sharper results (I always use bilinear for scaling down textures for the same reason. This modified apk works just fine with the latest v6, right? Thanks a lot for dedicating some of your time to this test. I often play at 75% so it may be worth trying.

Btw what kind of filtering uses when it's just 100%? Bilinear or nearest?

[zarik5]

For me on the original ev4+ looks like there are artifacts for res < 100% like a grid pattern, while my modded version does not have artifacts (but looks obviously a bit blurry). For res > 100% I didn't notice differences.

Btw what kind of filtering uses when it's just 100%? Bilinear or nearest?

Bilinear or nearest (or even trilinear/anisotropic) setting make sense if the image is not already at 100 scale, they tells the GPU what to do if the pixel of the display does not exactly line up with the pixel of the image

This modified apk works just fine with the latest v6, right?

I think no, because ev4 is pre FFR and ev5 changed the communication protocol version. You can still use ev4 on the PC too. If you say it is better my modded version for res <100 % i can submit a PR (maybe along with the new FFR).

[Kuuko-fokkusugaru]

Hey, so I have tested both, made screenshots, compared at different res, etc... I didn't notice an extreme difference. When playing at res like 25 or 50, it's true that the modded version looks slightly better. You get a bit more of blur but you can "predict" some text that otherwise would look like a block of pixels. But this may not be of much help because I highly doubt that anyone would play at such res. At 75% the difference it's minimal. This may be because the source image it's also reduced. Not like we are playing at 100% then reducing to 50% using different filters so if the details aren't there to start due lower resolution, it wont be either on the final image.

But leaving aside all that, this is not exactly what I meant because I can still see the video being "filtered. I will show you a simulated image scaling up a low res image so you get what I meant and what was my desired result. Take into account that I exagerated the effect a bit to make it more noticiable.

Here is the edited image. At the left it's similar to how it looks on Quest and at the right is how it could look if there were no filtering on the video when it's being played and stretched on Quest:

https://i.gyazo.com/e3a7abaeca435b876fe6bb6d8ebbc45e.png

And here it is a real capture from Quest at 25% res. As you can see it doesn't just looks pixelated but also blurred. Sort of like when you resize an image with bilinear or trinilar filter while my desired effect would be to be not filtered with nearest neighbour:

https://i.gyazo.com/be714ed271c899177f632153d872bbcd.jpg

And I know you may think that this looks crazily horribly lol and it's true. But for resolutions like 75 and 100% I feel like we would have a false sense of sharpness and make it more pleasant to the eye. I really don't want to consume any more of your valuable time but if someday on some spare time you feel like testing (if it's even possible at all) give it a try and let me know. In theory it should work sort of like when you see an unfiltered texture, like when you watch the desktop directly from Oculus dash on CV1 and Oculus S. When you are at Oculus home, every texture it's obviously filtered, you don't see them like on a PS1 or Minecraft. But when you open the monitor tool, this one doesn't seems to have the same kind of filter, it looks sharper and if you look at it from the sides, it looks pretty bad (not blurry, just like a mess of pixels one on top of each other precisely because the image it's too sharp).

Well, thanks for everything, now it's time to test the latest release with the fixed FFR. Regards.

[zarik5]

On server side probably there is another bilinear filter somewhere, and if you notice it then it means we should manage the resolutions better. It could be that the problem is in the CUDA code. If you are on Windows 7, the nvidia encoder has missing accepted texture formats and that force us to use CUDA to convert the output of SteamVR to an available format for the encoder. I cannot test the CUDA code because I'm on Win 10, and same for JackD83 I think, so you should upgrade if you want to use the newest versions of ALVR.

[zarik5]

These are various filtering techniques: https://matplotlib.org/mpl_examples/images_contours_and_fields/interpolation_methods.hires.png

[Kuuko-fokkusugaru]

These are various filtering techniques: https://matplotlib.org/mpl_examples/images_contours_and_fields/interpolation_methods.hires.png

That's quite interesting but my suggestion is not much regarding the filtering at server side but on the client at time of playing the video. There is indeed different filtering techniques on the server side since you managed to change them. These haven't a huge impact because the source image it will be dependent on the resoluton set on the ALVR. This would be different if SteamVR always renders at 100% and then it's ALVR the one that resizes the video to lower resolution in case that you have choosen one. Then the filtering would have a noticiable impact. For example, bilinear it's one of the ones that gives better results when resizing down, keeping the sharpness pretty well and it's cheap.

But if we set the res to 75% and SteamVR itself plays the games at 75% then there is not resizing here (unless you use FFR in which case, since it's reducing size, it would be better to keep it bilinear) so the filtering doesn't matters extremely.

I am more concerned about the filtering on the client, on Quest. I still feel we could get some sharpness increment in exchange of some aliasing but I feel that this shouldn't be extremely annoying and we could always have the option to turn it on, off and even select which filter to choose for the Quest resizing method at time of stretching the image to fit Quest's res.

So I don't think it matters too much what version of Windows I do use if the final filtering, the one I am concerned about, happens on Quest. We could even go beyond and apply a couple of filters on Quest at time of playing the "video" like trilinear followed of some sharpen for those that plays with less than 100% res (even at 100% res it would give some benefit, matter of taste probably).

The main thing would be to know wether it's possible at all to disable and change the filters used on Quest at time of playing the streamed video. If it's possible to have control over it, I would really apreciate a couple of options to disable, change, etc... but if not then we can't do much about it.

Thanks for the link again, very informative. There were several filtering options that I didn't even know :D

[JackD83]

for me on the original ev4+ looks like there are artifacts for res < 100% like a grid pattern

Thats a long standing issue with ALVR and the quest that could be prevented by using h246 as codec

[zarik5]

@KitsuneShan I will make some more experiments in a week.

@JackD83 What I did is replacing GL_NEAREST with GL_LINEAR here: https://github.com/JackD83/ALVRClient/blob/master/app/src/main/cpp/ovr_context.cpp#L81. BTW thank you for the invite!

[JackD83]

@zarik5 I changed the filtering in the client to linear. I could not see any difference in the visual quality and it does indeed fix the artifacts on lower (and higher!) resolutions

[Kuuko-fokkusugaru]

@JackD83 have you tried with no filter/nearest? Could it be possible to have these settings to play with different filtering options on the server side please? Those artifacts were on 264 or 265? (or both). Thank you for improving this! And yeah, as I suggested, linear it's better for downsizing so if you play with higher res you get some extra details. Glad that it worked for lower resolution too. I really want to see how would look without filtering. Probably not good for higher res but probably for 100% and lower.

[JackD83]

Those artifacts where only visible using h265. There is no ways to disable these filter. Its always NEAREST or LINEAR or some combination with mipmaps.

I know what you mean by no filtering, but this is not it. The image is basically the same and there are still filters somewhere.

Right now we have no configuration on the client to let you change things.

[Kuuko-fokkusugaru]

@JackD83 yeah I know that this is not it. What I meant it's exactly that, nearest. Having the options between nearest and bilinear despite of the encoding method on the client side. If what we see it's basically a projected texture, we should gain sharpness specially on 100% and lower. Isn't exactly like we will get more details out of nothing but don't you think some people may prefer a sharper image over a slightly blurred one? I may be totally wrong because I'm 3D artist so I may be focusing this on a wrong way while you guys are completely pro programmers but if you ever get the chances or curiosity on trying it would be cool.

Also, I am not sure why the server side itself apply a filter? @zarik5 gave me an apk that would change the filter behavior but I guessed it was on server side and not client or was it client? If it was client, why it looked "blurry" instead of nearest as expected? I wonder if the image itself was nearest but then it was filtered again or if it's just the screen of Quest which gives this "blurry" effect. In such case I may have been requesting something impossible to fix. But since native games can look sharper I guessed it would be possible. For example VRChat on certain parts like loading screen looks like there is no AA so it looks extremely sharp and quite good. With ALVR and using desktop low settings, AA on PC VRChat it's disabled but it's still slightly blurry. Granted this is and always will be a video but if it's 1:1 on render resolution and video being sent besides even trying at 250MBs, shouldn't we get like a crystal clear image? What it's preventing us from having so? Could it be some res mismatch that provoques always a bit of stretch?

[zarik5]

The apk I built does the exact opposite of what you wanted, that is changing GL_NEAREST to GL_LINEAR. The change was client side, I couldn't affect the server changing only the client. A native VR game is just rendered and then displayed, but on ALVR the image must go through multiple rendering passes so the sharpness can be lost. I'm finishing the new FFR, then I will try a few ideas to lower the blurriness even with FFR disabled.

[Kuuko-fokkusugaru]

@zarik5 that sounds awesome. After you tinker with stuff, could you please come back and report? I am very interested :D

[Kuuko-fokkusugaru]

Hey guys, I was just wondering something and didn't want to open a new issue just for that. It's sort of related tho. I was using a 780Ti so I couldn't use H265, it would crash the Steam server directly after hitting connect (I guess it would be better to just disable that option on system without such capabilities but that's not the point) so I has been using 264 just fine. I know 265 requires less bitrate to maintain quality and reduce latency but I was wondering if this is always the case or if for some reason 265 it's "harder" to encode and may take more time than 264. I have now a 2080Ti so it may not be an issue but I still wonder if there is any advantage on 264 over 265 like could be that, encoding/decoding times or if I should totally forget about 264 all together now.

[zarik5]

H265 can use more complex compression algorithms that usually are enabled only for non-real-time streaming applications because they introduce a lot of latency. For real-time applications such as ALVR these algorithms are not enabled, so the behavior of the two codecs becomes almost identical.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.