Selective lighting

[goodsign]

I'm not sure whether this is already a planned feature or maybe even a finished one, so I'll try to explain the task first.

I need to do selective lighting for different rooms. For example, I have two rooms. One light must only affect objects and inner walls of one room. The second room must not be affected by this light.

Currently, if I add a light to the scene, it affects all objects in its distance. And I get strange effect when light comes "through the wall" of the second room.

So I think I need some kind of groups or channels for lighting, so that I could set the objects that are affected by one light source and objects that are affected by another.

I haven't found anything like that in either lights or objects, so I thought that maybe it could be a good feature to have.

And, btw, if it is not implemented yet, what is the recommended approach to solve such tasks using current state of three.js?

[curiouspers]

+99999 again

[brianchirls]

This would definitely be very helpful. And we now have layers working, which it sounds like is what's been holding this up. So, here's the friendly bump. ;-)

[rohan-deshpande]

@manthrax check this!

[fritx]

+99999 again

[bnolan]

I'd love for this. I have a situation where each avatar is lit by only the nearest light, and I can have thousands of lights into the scene, it'd be nice to specify a single light for each avatar.

[tiesselune]

Hi! I'm going to need this feature, and I thought of implementing it, since I think I've located where this could be done (initMaterial in WebGLRenderer) and it should be pretty straightforward using existing layers.

I'm thinking of creating a light-specific bit mask (as a Layers object) to select which layers are affected by a specific light object (independant of the light's own object layer), then filtering lights/shadowmaps that are set as uniforms for each object in the above-cited function.

Would that be relevant?

I'll try this and submit a pull request as soon as I've acheived this unless you redirect me to some other method before that.

[tiesselune]

UPDATE:

https://github.com/tiesselune/three.js/tree/selective-lighting

I'm getting somewhere. Not sure how this might affect performance or how to optimize my code for perf, but it's working on the few tests I've made. Per-layer lighting! Yay!

I still have to test some more though, and make a relevant example before submitting a pull request.

[tiesselune]

UPDATE:

I've made a few more tests, and having a same material on two objects that are on different layers (and have different lighting settings) causes the renderer to update the material constantly (via needsUpdate) since the lighting settings on the material changes between two objects, which was impossible before. This means a huge perf drop and visual oddities (since the last material upload wins at render time)

Of course, this can be avoided by creating a new material instance instead of using the exact same one for two objects with different lighting settings. But I doubt cloning it for the user is a good idea, since it would mean watching the original material for changes to mirror them onto the clone.

Here are two solutions I could use:

1. I could just display a warning when two objects on different layers with different lighting settings share a same material, so the user can create and manage a new clone material by himself.

2. I could add layers at a material level instead of the object's level (and create a parrallel layers system for lights) so that it's mandatory to use different materials to achieve different lighting settings. That would mean there would be the current layer system for visibility (on objects and cameras) and a different one for lighting (on materials and lights).

What do you guys think?

[brianchirls]

I was just thinking about this as I saw your comment come in. I think displaying a warning is not helpful, as many people probably won't see it. Of the two, I definitely prefer setting light layers on the material rather than on the object. It's a little counter-intuitive, but it still makes sense.

Just to clarify, when needsUpdate is set, in this case, it just recalculates the uniform values, right? It shouldn't have to re-compile a shader because the different shaders should all be cached. What kind of performance hit are you seeing? Did you use a profiler to see exactly where the extra computation is happening?

By the way, filterAmbiantLights should be filterAmbientLights.

[tiesselune]

You are right about the misspelling. I actually had written everything with "ambiant" before noticing it was wrong and I forgot an occurence. 😇

I think I'm gonna move the layer info on the material then. It seems more relevant and consistent with the way it works.

The performance drop came (and still comes) from initMaterial (and then acquireProgram and getShaderParameter which is tremendously slow for some reason) being called every frame several times because the lightHash would be different everytime, setting material.needsUpdate to true.

BTW I'm using Chrome's profiling tools. (Timelines and profiler).

[brianchirls]

That's fine by me, but I'm not clear why acquireProgram should be running all the time. If the number of each type of light gets added to the program parameters, then shouldn't recompiling the shader each time be unnecessary?

[tiesselune]

I don't get it either;

Here's the call stack (at every frame) and the terrible memory leak that follows, if you're familiar with these, that could help me understand what goes wrong. It looks like it's creating new WebGLPrograms every frame...

Maybe it could be replaceLightNums that updates the vertex/fragment shader code during the WebGLProgram's creation, causing it to recompile, or considering it's a different shader... ?

EDIT : Anyway, using layers at material level solves the problem as expected. I just have a memory leak somewhere that I should investigate but otherwise it looks quite good.

[brianchirls]

It's probably a good idea to understand why this is happening just to make sure you fully understand the code and are not introducing some new bug. Can you step through it to see why the shader code changes every frame?

I don't know the WebGLPrograms well enough to say what's going on, but it seems like it should be caching those shaders for you.

[tiesselune]

It's quite simple, actually:

• to know wether or not to call acquireProgram, initMaterial gets the shader "code" from the cache (it's more of a hash, actually, constructed in in getProgramCode) and compares it to the"code" of the current material's program.
• Since this function creates the "code" from parameters including number of each type of light, which is filtered and though different between objects, the code can be different for a same material ( so to say, the same program) from an object to the other.
• Since this process is iterated through for each object, the "code" changes several times per frame, causing it to recompile as many times in a single frame: which causes the perf drop.
• That said, since several objects have the same material, and so, the same program, the last object iterated through imposed its parameters to the program, which caused both objects to be lit with the same light setup but not necessarily the same shadowMaps (that can be calculated for objects in a different order, but suffers the same "last object wins" problem), causing my visual oddities.

Now everything works as expected (since each object with different light setups have a different material and a different program), except for a memory optimization I could do with my own light hashes, which I'm going to tackle soon.

There could be some improvement if light numbers were variable, so that each material are just instances of the same program, but since those numbers are inserted in the shader code (in replaceLightNums) it seems incompatible with the way things currently are (and their optimization).

Hope the explanation was understandable enough

[mrdoob]

I wonder if we can pass the layer mask to the shader (one uniform per object and one more uniform per light). Then we could add something like this in here.

[mrdoob]

Something like...

for ( int i = 0; i < NUM_POINT_LIGHTS; i ++ ) {

    pointLight = pointLights[ i ];
    pointLight.distance *= float( object.mask & pointlight.mask > 0.0 );

    ...

}

[mrdoob]

Meh, I don't think we can do bitwise operations in glsl...

[tiesselune]

Yep, it looks like bitwise operations begin in GLSL 1.30 and that standard WebGL uses 1.00. 😕

[erichlof]

Hi @mrdoob and @tiesselune ,

Chrome 56 and Firefox 51 have just landed, which means WebGL 2.0 is enabled by default (WebGL2 follows the OpenGL ES 3.0 spec). That means that Bit operations (and other cool stuff like 3d textures) are now available. I don't know if Three has been slowly preparing for the shift to WebGL 2.0, but I made a few mandatory changes to my copy of three84.js and enabled WebGL 2.0, and did a bit operation in a shader just to make sure it worked, and it did! https://developers.google.com/web/updates/2017/01/nic56 https://developer.mozilla.org/en-US/Firefox/Releases/51

Just wanted to let you know. +1 for the masking idea :-)

[tiesselune]

Hi @erichlof ! That's great news for WebGL2; it means that WebGL2 renderer will probably be able to approach the masking problems in a more efficient way.

Anyway, given the very limited use of browsers supporting WebGL2 this far, I don't think we can discard efforts to make it work in WebGL 1 : it's taken a long time until nearly all browsers can run WebGL apps, so it will probably a bit longer until WebGL 2 is really usable at a large scale... 😕 But thanks for the hint!

[tiesselune]

Hi again!

New update: I've done what I thought necessary for memory management, but JS heap garbage collection seems to have a lot to do with my software configuration (running programs, open tabs, extensions) at the time I run profiles. Can someone with a bit more experience on that confirm that my code is not that memory hungry compared to the current version of threejs?

Anyway, I've made an example in examples/webgl_lights_selective_lighting.html. The thing is, I got shadowmap-related visual artifacts when using two spotlights (on the same layers) simultaneously on the skinned objects of the scene. The static objects behave as expected. I've run tests and it happens on the dev branch also, without considering selective light layers at all (actually, I just added a spotlight in the shadowmap example, on the dev branch). Does anyone know where that comes from?

Here's a screenshot:

[looeee]

It should be possible to view the example live here:

https://rawgit.com/tiesselune/three.js/selective-lighting/examples/webgl_lights_selective_lights.html

I'm getting an error though:

Uncaught TypeError: Cannot read property 'set' of undefined
    at init (webgl_lights_selective_lights.html:117)
    at webgl_lights_selective_lights.html:67

[tiesselune]

@looeee : you'll have to run npm run build-uglify on a local host. I voluntarily did not include builds in my commits for mergeability purposes...

Or should I?

EDIT : here's a working link on a different branch for test purposes : https://rawgit.com/tiesselune/three.js/selective-lights-test/examples/webgl_lights_selective_lights.html

[sasha240100]

Hey, +999999 from me. This should be implemented (at least with THREE.Layers)

[fritx]

$$('.comment-body').reduce((acc, el) => {
  let mat = el.textContent.match(/\+(\d+)/)
  let num = +(mat && mat[1] || 0)
  return acc + num
}, 0)
>> 1219997

[mrdoob]

🤔

[makc]

@mrdoob he calculated +N-s in this thread

[makc]

btw, +14570 for selective lights

[trusktr]

This will be incredibly great for fine tuning how lighting works in a higher abstraction.

For example in the following pen, I'd like to have lighting work one way for the DOM elements vs another way for the Sphere in order for lighting to be more realistic:

https://codepen.io/trusktr/pen/RjzKJx

[trusktr]

Certain effects that look like a single-point-light can be achieved by combining two or more lights that are selectively affecting only certain elements.

For example, in my previous example, I can increase the light intensity to get a good looking shadow on the "DOM element", but then the Sphere will look a bit too shiny and bright. If I were able to have a dimmer light for the Sphere, and a brighter light for the "DOM element", then I could achieve something more realistic this way, where it would seem to the viewer that there is still only one light. Then this sort of thing can be abstracted away in a higher-level API that makes it seem like there is only one light being manipulated when behind the scene two Three.js lights are actually in play.

[trusktr]

@WestLangley

As a work-around, can you achieve most of your requirements by having a separate scene for each room (and its lights)?

Do you have a link to an example of something like this? Can it have other unexpected rendering effects on the outcome?

[makc]

other unexpected rendering effects on the outcome?

rendering order of transparent objects, for example - transparent objects in scene 1 will be rendered before opaque objects in scene 2.

[makc]

Do you have a link to an example of something like this?

here, I made one just for you: https://jsfiddle.net/f2Lommf5/524/

[trusktr]

rendering order of transparent objects, for example - transparent objects in scene 1 will be rendered before opaque objects in scene 2.

That's what I was thinking; it makes the workaround only a workaround for very limited cases. Looking forward to the real solution!

[makc]

well, you could argue that the problem would only happen if you could see opaque objects of scene2 from scene1 trough said transparent objects. but if that's the case, the lights should pass through as well, and there would be no reason to separate the scenes to begin with. but I agree this kind of argument is not really convincing.

[trusktr]

@makc Here's an example of the problem I'm trying to solve:

https://discourse.threejs.org/t/how-to-make-shadows-darker-on-transparent-objects/1389

I think selective lighting would really help with this.

[makc]

@trusktr I think you can solve that one by altering mrdoob's shadow material. it just shows black-white shadow texture, and you can modify it to be transparent as necessary

[trusktr]

Masks in PlayCanvas seem really easy to use for selective lighting: https://forum.playcanvas.com/t/set-certain-object-to-not-receive-light/785.

[ErikBehar]

can we get this merged in ? I think the shadow problems were due to Lambert shader nothing to do with the change to the code. @tiesselune

[tiesselune]

@ErikBehar I was actually looking for something to do this afternoon, I'm going to try to update the code to latest threejs and submit a pull request maybe? I'd actually enjoy knowing I made a real contribution to such a big project. (And good to have confirmation that this bug had nothing to do with my changes)

EDIT: just discovered that this part of the code changed a lot. Gonna need a little time I guess since the states that I was previously using moved from WebGLRenderer to WebGLLights.

[ErikBehar]

@tiesselune I kinda needed to get this into a project so I went ahead and ported your code over to v93 see: https://github.com/ErikBehar/three.js/tree/selectiveLights

I can do a PR if you guys wish ? @mrdoob

[tiesselune]

@ErikBehar Well it does look like what I was aiming for but I did not have the time to do it entirely. I think I would have moved the filtering/hash functions out of WebGLRenderer and added them to the WebGLLights state object though. I feel like they belong there now, the layer system being a part of the state and not a part of the renderer.

[mrdoob]

@ErikBehar a PR would be great!

[ErikBehar]

@tiesselune I'll look into moving it as you suggest, or feel free to push a PR on my fork lol = ] and I'll post the PR soon @mrdoob

[VaelynPhi]

What's the status on this? I've merged the r94dev code from @ErikBehar into r94, then merged r97 into it, only a couple very straightforward conflicts (version change, a few variables for selective lighting, and the hash creation in renderers/WebGLRenderer.js; I'd be happy to put in a PR. @tiesselune if you can give me some idea where you intended the selective lighting state to go, I'd be happy to move it, test, and put in a PR.

EDIT: A little later: I see that this needs some work to function, now, with the new lighthash.

[makc]

@VaelynPhi I'd think that, regardless of whether they are willing to accept @ErikBehar 's code or not, it is good idea to submit the PR, just to replace the outdated one in case other people will want it, like you yourself did

[ErikBehar]

Sorry to drop the ball on this guys = / ... @VaelynPhi can you post your fork / branch ?

[VaelynPhi]

No worries; I understand being busy. Alas, I couldn't get even the branch you had working, @ErikBehar; I settled on reading through the code to try and break it down so I could move the state to the appropriate location and hopefully fix any bugs. I have yet to get to a working port state even on v94. Perhaps I can clean it up and put in the PR just to update it as @makc suggests. Give me a bit; I'm super busy. :) At the very least maybe that'll help highlight the changes that need to be made to pull selective lighting into the recent version.

[byondrnd]

added a PR based on: https://github.com/ErikBehar/three.js/commit/ac0499b70b82bc7bb780100a8372fcdf318d1424#diff-5e43a0b5002eb2c419def3baf67d4e67 by @ErikBehar can anyone give a hand with some review and the example ?

https://github.com/mrdoob/three.js/pull/15223

[Flyrell]

Hey guys, what's the status on this one? @tiesselune @ErikBehar Might I help you with something? Would be nice to implement it finally after 4 years 😄💯