Request for Feature: Screen Space Reflections

[bhouston]

One thing that I find missing from ThreeJS that would be amazing to have would be screen space reflections.

Here are some references for the ambitious coder:

https://docs.unrealengine.com/latest/INT/Engine/Rendering/PostProcessEffects/ScreenSpaceReflection/index.html

http://www.gamasutra.com/blogs/BartlomiejWronski/20140129/209609/The_future_of_screenspace_reflections.php

http://www.frostbite.com/2015/08/stochastic-screen-space-reflections/

Example code from the talented Morgan McGuire:

http://casual-effects.blogspot.ca/2014/08/screen-space-ray-tracing.html

Example open source code for Unity 5:

http://www.kode80.com/blog/2015/03/11/screen-space-reflections-in-unity-5/ https://github.com/kode80/kode80SSR

[jgwinner]

My only concern here is that it doesn't work really well with stereo Rendering, so it would be tough in WebVR, but please don't think I'm against the feature :) :+1:

[bhouston]

Just turn it off in VR. A lot of the costly effects do not work well in VR. UE4 even suggests going with forward rendering rather than deferred in the VR context for speed purposes.

[jgwinner]

Exactly, which is why I'm not against it.

I stumbled on here while working on a React Native implementation of reflection mapping, Screen space reflections are great for non-VR games.

[mrdoob]

THREE.Reflector works in VR ✌️ https://threejs.org/examples/webvr_sandbox.html

[jgwinner]

Nice!

Although even on Nightly I get "VR Not found" but I'll check it out.

My particular problem (which isn't the OP's) wasn't to solve the reflection problem, it was how to create three.js native objects, and render them within ReactVR's existing scene and renderer. I got OK results, the only thing was the background image was a little strange, so reflections look weird - but that's as expected, React VR creates an inside-out sphere for the background pano. I haven't tried a cube map, but that might work better.

I'm WAY over on pages on the book so I'll probably leave the existing demo the way it is, but this looks like a great technique.

[gkjohnson]

I've been wanting to learn more about this type of effect so I tried my hand at making a composer effect for it. There's still some work that could be done but here's what I put together based on Kode80's implementation and Morgan McGuire's blogpost (the sponza scene takes a bit to load in):

https://gkjohnson.github.io/threejs-sandbox/screenSpaceReflectionsPass/

Some things to do next would be cleaning up the jitter artifacts, glossy reflections (with a blur pass or mip map sampling + depth map pyramid) and multiple bounces via temporal reprojection.

If anyone is actually interested in using this effect I'd be happy to work with someone to polish this out a bit more!

[WestLangley]

@gkjohnson Sweet. Your scene may be too complex, however. In any event, your demo seems to be largely unresponsive. Can you create a simpler example? Also, an on-off toggle would be nice.

[gkjohnson]

@WestLangley

your demo seems to be largely unresponsive

Are you able to see the demo? The screen may be black for a few seconds or so because the Sponza scene will take a little bit to download depending on your connection.

If it's not working at all I can take a look at creating something simpler -- it works on my Pixel 2, though

an on-off toggle would be nice.

Turning the "intensity" slider down to 0 looks the same as turning it off, though ray tracing is still happening so there won't be any performance difference in that case if that's what you're looking for.

[WestLangley]

@gkjohnson Got it. On my mac, your demo is black for awhile, and then the frame rate is 4.

[gkjohnson]

@WestLangley It's a very resolution-dependent effect so if you make your window smaller the framerate will probably go up. Doing the ray tracing at a lower resolution and then compositing with the full resolution buffer would help make the effect more scalable, but I haven't taken the time to optimize it.

[bicubic]

@gkjohnson how would you mask it to reflective surfaces only? If I crank up the intensity everything starts reflecting and there are some pretty jarring artefacts.

[gkjohnson]

@bicubic At the moment no surface attributes are being taken into account but it shouldn't be too hard to add a basic version of that. You could fade or blur the effect based on the specular / roughness attribute of the material.

Unfortunately I have no use for the effect at the moment so I haven't gone out of my way to add these other features. Like I mentioned before, though, if there's interest in using it for something I'd be happy to collaborate and round out the effect!

[looeee]

You could fade or blur the effect based on the specular / roughness attribute of the material.

It would be more useful to be able to specify objects as reflective / non-reflective. At the moment everything is reflective, including things like curtains and plants. Ideally, only the floor should be reflective in this demo.

[gkjohnson]

@looeee I agree that's what I'm saying -- sorry if I was unclear. The material attributes would have to be written to a target so they can be sampled in screen space and used to fade or blur the reflections at the given pixel. This would allow roughness maps etc to affect the reflections, as well.

[bicubic]

I would love to see this become part of three.js default capabilities.

There's a lot of stuff like this that makes you wonder whether a deferred renderer architecture should also become part of the default offering. Especially for webgl2.0

[looeee]

@gkjohnson got it, that makes sense. However, it still seems like a mask of some kind might be neccessary to prevent artefacts. In your example, you would want zero reflections on the drapes and plants. Would fading to zero based on those material's properties work correctly?

[gkjohnson]

@bicubic I agree I'd really like to see a dedicated deferred rendering path, as well. With WebGL2 and multiple render textures it's much more viable.

@looee It depends on the effect you want but with the approach I described the rendered roughness value would effectively behave as the mask for the scene with a roughness value of 1 meaning "render no reflection at this pixel" and a value of 0 meaning "render full reflection at this pixel". Values in between would render reflections at a partial intensity. The intensity slider in the demo shows the effect (but obviously for the whole scene instead of per pixel).

Another approach is to blur the reflected pixels based on the roughness value to emulate a diffuse surface reflecting light. So the drapes would still have some reflections, just not sharp ones. This would allow for the drape being illuminated by a brightly lit blue floor, for example.

[looeee]

a roughness value of 1 meaning "render no reflection at this pixel" and a value of 0 meaning "render full reflection at this pixel".

Seems like this could work well. Unreal uses a "max roughness" parameter and reccommends it be set to 0.8.

[looeee]

Oops

[Fyrestar]

@gkjohnson This is very interesting and i love your result, i'm integrating it in a WebGL2/MRT renderer right now and will experiment with it. Did you noticed you create a material in each render call? this.scene.overrideMaterial = this.createPackedMaterial();

[gkjohnson]

@Fyrestar Thanks! And I hadn't noticed that! But like I said I think there's still a bit of cleanup to do and performance probably isn't quite at a point where it's super usable, yet.

These types of effects are really more well suited for a deferred renderer than forward, anyway (presumably like you're working on). I've been wanting to put together this and some other high quality effects, but I'm inclined to wait until a proper deferred renderer is available before putting to much more time into that. Is the deferred / MRT renderer you're working on available on Github?

[Fyrestar]

I've integrated it now, while the SSR pass outputs to a lower resolution RT, basically only the reflections and then combine it using a depth blur with the main image to remove/reduce the errors. I render the roughness values to an attributes RT and use it again to determine the strength of blur with 1 being invisible to discard the fragment.

The blur also removes the error which gets stronger in distance. One issue yet seems to be transparency again and objects partially covering others causing to clip the others reflection. I really need to read more about the technique.

I'm using it with a custom engine on top on THREE, not using the compositor of THREE. THREE is actually ready to use a WebGL2 context in the current WebGLRenderer, i've added the MRT there. I'll put it on github soon i guess. It still requires to render the scene a second time for the backface depth buffer though, but that isn't too expensive.

[MEBoo]

@Fyrestar Hey any news about?

[gonnavis]

I wrote my own and make a pr to three.js, but now just support OrthographicCamera, I'm trying to support PerspectiveCamera. Demo, Demo2

[gonnavis]

PerspectiveCamera support is OK now. https://github.com/mrdoob/three.js/pull/20156

Demo

======EDIT======

======Origin======

[bhouston]

This is beautiful and quite the accomplishment. It is very slow on my machine -- I get 5fps at 1920x1080. What is the difference between your implementation and Sketchfab's? I suspect if we knew the different we would know how to optimize yours....

[gonnavis]

@bhouston Thanks!

Now everything in the scene is reflective, but it is not needed in most practical projects. I'll add some like IntensityMap property or use exsiting Roughness Metalness information, combined with the distance attenuation, at the pixel with a value of 0, terminate the calculation directly, I think it'll greatly improve the fps. May also randomly select part of reflective pixels.

I have tried to view the code of Sketchfab, but failed. I'll try again, but I don’t feel much hope. If anyone has the code of Sketchfab, please share, thanks!

------EDIT------ In addition, Sketchfab's presentation window is not fullscreened by default, and stop render when no operation. If fullscreened and keep rotating, even simple scene will also has high gpu usage.

[WestLangley]

@gonnavis The shader code for that example should be viewable via the Spector.js Chrome extension.

[bhouston]

Just playing around with Sketchfab I Can see that they are shifting their SSR resolution and refining it. I wouldn't be surprised if they are 3 to 5 quality levels that they are jumping through progressively. This ensures there is an interactive framerate but that it refines to perfect when static.

[looeee]

This ensures there is an interactive framerate but that it refines to perfect when static.

The downside is that there's a noticeable jump when the camera stops moving and quality levels are switched. It's more obvious in some scenes than others. Here's one where it's very obvious.

EDIT: they do it with more than just reflections - it seems like lights, shadows, reflections, and maybe texture resolutions are progressively enhanced. Watch the light/shadow on the back wall of this scene.

[gkjohnson]

When I was experimenting with SSRR above I referenced Morgan Mcguire's article on it here quite a bit, which explains a couple techniques that it looks like Sketchfab might be using.

Specifically the pixel stride can be increased so fewer pixel pixels are sampled which can be coupled with a regular per pixel jitter so sibling pixels "skip" different depth samples while stepping. If the ray stride is high enough you can add a binary search at the end once something is intersected in the depth buffer to find a pixel that a ray would have hit first. Returning early on fragments that aren't reflective would probably help like you already mentioned @gonnavis.

I also experimented with a depth and normal-aware upscale of a lower resolution raymarch but I never got to a point where I was happy with the way it looked. I think this technique in games can benefit pretty heavily from the fact that available resolutions are often divisible by two, which makes an upscale simpler.

I'm curious as to how Sketchfab is handling the the rough reflections that blur based on reflection distance, though. I've seen that Godot Engine does a blur after with a radius based on the ray distance and surface roughness but that results in artifacts I don't feel like I'm seeing here. I'll have to poke around with Spector.js when I have a chance (thanks for the tip @WestLangley!). The Frostbite paper @bhouston posted initially suggests using a depth pyramid when ray marching but generating that can be a pain in webgl1 (and maybe 2?).

It looks good @gonnavis! Glad to see it might make it into the library.

[gonnavis]

@WestLangley Thanks! but when I use Spector.js on Sketchfab, I got "No frames with gl commands detected. Try moving the camera." error.

I doubt that after SSRPass is optimized as I said earlier, the performance of Sketchfab will still lead significantly. Especially in the case of continuous rendering and no downgrade strategy as @bhouston @looeee mentioned. I even think that if don’t talk about the implementation details, the core concept of this SSRPass may be already the optimal solution.

@gkjohnson Thank you for your information and support! I'll keep improving.

[mrdoob]

@gonnavis

I recommend Safari's Graphics tab:

If you use Linux you can use Epiphany (aka Gnome Web): https://webkit.org/downloads/

I don't think there is any WebKit build for Windows though...

[gonnavis]

@mrdoob Thanks! Though I mainly use windows, I'll try on Mac if needed.

But I'm wondering whether it is allowed to upload Sketchfab's code to open source library, if find any useful stuff such as glsl function?

This SSRPass's structure is base on three.js SSAOPass, and demo scene uses three.js webgl_shading_physical example, key pointToLineDistance function comes from wolfram.com, and others mainly wrote by my own (of course is based on a lot of other previous experience, especially other three.js examples ). I think these all no any problem, but is it the same case of Sketchfab's code?

[WestLangley]

Thanks! but when I use Spector.js on Sketchfab, I got "No frames with gl commands detected. Try moving the camera." error.

Scroll the mouse to zoom the camera. That should be sufficient to trigger Spector.

Also, use a static scene in your example that does not require a render loop. Currently, the frame rate is too low to be acceptable.

[gonnavis]

@WestLangley Succeeded in this scene, thanks!

[bhouston]

Do not post Sketchfab's code please. I was merely suggesting we understand their techniques. Which we can do without sharing their code. You also can not copy anything from their code, that would be against copyright.

On Tue, Aug 25, 2020 at 6:21 AM Vis notifications@github.com wrote:

@WestLangley https://github.com/WestLangley Succeeded in this scene https://sketchfab.com/3d-models/iron-man-helmet-captain-america-shield-endgame-02556e341dd84fa5b9ef92c5eeeb3287, thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/8248#issuecomment-679939119, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPV7OEK245U5EF35YTZQDSCOGBPANCNFSM4B4V62SQ .

-- Ben Houston, CTO M: +1-613-762-4113 bhouston@threekit.com Ottawa, Canada ThreeKit Visualization Platform: 3D, 2D, AR, VR http://https//threekit.com/

[gonnavis]

@bhouston OK, for reference only if necessary.

I even often feel that, after receiving some inspiration, keep looking at other people’s paper and code is more difficult than writing it myself, especially when in different environment.

For example, this time, I was mainly inspired by this tut, but it is difficult to clearly understand the meaning of the text at start, and because of the different coordinate system he uses, and I don't know how to run the .cxx file, so I only read a little text and hardly read the code. So in the end, the two pictures imgA imgB that helped me the most.

[gonnavis]

Add some demos.

Demo, Demo2, Demo3, Demo4, SelectiveDemo, OrthographicDemo.

[gkjohnson]

Inspired by @gonnavis I finally got around to trying out a few new things in the SSR implementation I posted a couple years ago so I thought I'd share here. I have no plans to use this in a real project so it's not optimized and at this point and it's become a heap of different SSR features but I've learned most of what I want to from it so I'm going to call it "done" for now. Perhaps there are a few things others can take, though. There are still things that I know could be added or improved but maybe if I ever pick it back up I'll leverage some features in WebGL2 to simplify things.

Some of the new features:

• Support for blue noise ray jitter so the intersection pattern is less regular.
• A depth and normal-weighted upscale / blur from a lower resolution march texture to preserve detail.
• A couple of eye-tweaked approaches to rendering glossy reflections with jittered samples and mipmaps.
• Support for normal maps and alpha maps.

It is a pretty intensive effect but I feel like with a mix of reflection upscaling, blur, jitter, and low step count you could something that works well especially with smaller embedded canvases rather than full window apps. By resolving the final image over multiple frames like has been discussed in https://github.com/mrdoob/three.js/issues/14048 you could probably get decent glossy reflections, as well. Of course as has been mentioned elsewhere the reflections still won't be really right without separate passes for diffuse and whatnot but there's always something else to improve.

Here are a few screenshots with higher step counts:

No Glossiness	Multisample Glossiness	Depth Hierarchy Glossiness

And here's the demo:

https://gkjohnson.github.io/threejs-sandbox/screenSpaceReflectionsPass/

[Ben-Mack]

There's a slide in Siggraph 2015 about "Stochastic Screen-Space Reflections" that you might be interested: http://advances.realtimerendering.com/s2015/index.html

[01000001khan]

Was support for disabling reflections on rough surfaces ever added? This is something I'm struggling to figure out in my project. I don't need diffuse reflections on rough surfaces, just the ability to turn them off per object or material.

[gonnavis]

I've tried https://github.com/mrdoob/three.js/pull/21487, but full PBR integration was too challenging for me and I couldn't complete it yet. However, it might still be helpful as a reference.

[01000001khan]

Oh wow, good luck! I had thought roughnessCutoff might be what I was looking for, but I guess not?