AMD should reuse/adapt the academic State Of The Arts

[LifeIsStrange]

Hi @rys It is often deeply institutionalized in companies and even in the human nature, to reinvent the wheel or to work in isolation. I have much admiration for AMD and its software developers and trying new approaches is a way to drive innovation. AMD FSR while not using neural networks is a very interesting way to look at the problem.

But please AMD you are not alone, there is a widespread community of very smart people out there: the scholars and they regularly release innovations in machine learning, especially regarding the topic of super-resolution. It would be a shame if you didn't take strong inspiration from their amazing ideas that can be combined in order to create a synergetic algorithm, on par or superior to DLSS.

The #1 reference website for tracking the state of the art is paperswithcode.com See: https://paperswithcode.com/task/multi-frame-super-resolution https://paperswithcode.com/task/image-super-resolution https://paperswithcode.com/task/video-super-resolution https://paperswithcode.com/task/super-resolution https://paperswithcode.com/task/3d-object-super-resolution Note that i also invite for a collaboration with Intel XeSS. You might also take inspiration from this new Nvidia open source library: https://www.phoronix.com/scan.php?page=news_item&px=NVIDIA-Image-Scaling-SDK

The graal would be if for your next neural based DLSS competitor, to publish your results (PSNR, SSIM) on the same benchmarcks as paperswithcode.com, additionally it would be a great scientific contribution if you could submit to paperswithcode.com game/textures oriented datasets.

[Crytoma]

Agreed. Although Nvidia I think is following a similar approach with DLDSR and will probably move to generic up-scaling following their Image Scaling. Although I could be wrong.

[brunogm0]

Here goes a suggestion that improves on the Lanczos kernel. johncostella.com/magic/

2021: Beyond Magic Kernel: Analytically derived the Fourier transform of the Magic Kernel in closed form, and found, incredulously, that it is simply the cube of the sinc function. This implies that the Magic Kernel is just the rectangular window function convolved with itself twice—which, in retrospect, is completely obvious. (Mathematically: it is a cardinal B-spline.) This observation, together with a precise definition of the requirement of the Sharp kernel, allowed me to obtain an analytical expression for the exact Sharp kernel, and hence also for the exact Magic Kernel Sharp kernel, which I recognized is just the third in a sequence of fundamental resizing kernels. These findings allowed me to explicitly show why Magic Kernel Sharp is superior to any of the Lanczos kernels. It also allowed me to derive further members of this fundamental sequence of kernels, in particular the sixth member, which has the same computational efficiency as Lanczos-3, but has far superior properties.

2021 paper: Solving the mystery of Magic Kernel Sharp

[brunogm0]

"This strategy of “generalized sampling” has appeared in a few graphics papers, but is largely unexplored in our community. This survey broadly summarizes the key aspects of the framework, and delves into specific applications in graphics. Using new notation, we concisely present and extend several key techniques. In addition, we demonstrate benefits for prefiltering in image downscaling and supersample-based rendering, and analyze the effect that generalized sampling has on the noise due to Monte Carlo estimation. We conclude with a qualitative and quantitative comparison of traditional and generalized filters."

"The interpolating cardinal B-splines β*int consistently outperform the more traditional filters for the same N and W. "

https://w3.impa.br/~diego/publications/NehHop14.pdf

[LifeIsStrange]

In other news: https://www.phoronix.com/scan.php?page=news_item&px=AMD-FidelityFX-Super-Res-2.0

[brunogm0]

I hope the FSR2.0 use something like this:

"Fast temporal reprojection without motion vectors" https://jcgt.org/published/0010/03/02/

[LifeIsStrange]

Great find!

[brunogm0]

Another great find that helps with why people have so much variance on "sharpness" preferences ( view distance and the display effects).

Presentation: https://www.youtube.com/watch?v=D0EcX0NtpW0 Paper: https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13942

[rys]

I'm going to close this issue since it's not directly related to FSR1, but I also wanted to comment on a couple of the general themes to hopefully reassure you around how we go about our developer-focused applied research.

We definitely don't do that research in a vacuum, and honestly I'm not sure how we could. Like every company trying to invent the future in some way, we definitely absorb and take inspiration from the state of the art as we see it, in order to inform what we try, how we try it, and how we shape what ends up in the hands of developers.

Our outlet at the end for this kind of thing is always laser focused on getting the code and related information, tooling, documentation or whatever it is into the hands of developers, and working with those developers to put what we've made into what matters: their games.

So we start by thinking about developers, and we end by delivering what we've made directly to them, often by working with them every step of the way. If we can publish a paper or talk at a conference then we do that when we can (we're presenting about FSR2 internals at GDC 2022 as I type this!). Hopefully the balance is the right one.

<3

[LifeIsStrange]

@rys It is reassuring to hear that you try to a maximum to survey the advances in the state of the art. However it is very easy to miss interesting and unknown papers, for example a key major advantage of FSR 1.0 is that contrary to DLSS and XeSS it does not require motion vectors information. Which means that it can be applied on any software/game, games do not need to be designed for it and hence you can provide a global radeon setting. As an example, all game emulators are integrating FSR 1.0 and it has dramatically improved the rendering quality of old video games. I understand that everything's a tradeoff and algorithms with motion vectors are currently generally better than FSR 1.0. However researchers have innovatively found that it is possible to have temporal based antialiasing like FSR 2.0 without requiring motion vectors. This is disruptive as it would allow graphisms superior to FSR 1.0 (unclear wether < or >= to FSR 2.0) for every game out there, out of the boxes (including emulators). I am refering to the paper cited by @brunogm0 https://jcgt.org/published/0010/03/02/ While FSR 2.0 align with XeSS/DLSS, the cited paper would enable AMD to keep having the best option for games not specifically designed to provide motion vectors (AKA 99.9% of video games) Hence I sincerely hope AMD will consider reusing this paper for an innovative FSR variant BTW the cited paper algorithm has the ability to input motion vectors if available, which then allow the best of both worlds

Likewise, if true motion vectors are available, these can be used atany scale and for any masked region in the image to guide or overwrite our matchingprocedure for more stable results.