Sunlight shadow never propagates into parts of the world that aren't loaded when placing blocks that cause the shadow

[Barteks2x]

I was trying to figure out how Terasology handles sunlight propagation in some hard cases. I first read the code but it wasn't obvious at all how something like that would be handled. So just to verify that the game really handles these possibilities I tested it.

I built a platform 500 blocks above the ground. The shadow never reached the surface. This isn't visible in most of normal gameplay, but it will be visible when building larger structures with low render distance.

(tested using version stable-68 2016-10-08)

[skaldarnar]

Hi @Barteks2x and thank you for your investigation!

@emanuele3d may know the details of the code best, let's see what he has to say about it. I guess we have a hard limit for calculating shadows over a specific range, so I don't know whether we should consider this a bug or a known limitation of sorts.

[Barteks2x]

I will do some more tests, but it seemed to me like it depends on render distance.

[Barteks2x]

I tested it again with different view distance, and yes - there is a hard limit. But a hard limit has it's own issues. For example it's possible to make completely closed structure that still has full sunlight inside:

I wanted to see how it's done because I'm working on minecraft mod that uses 3d chunks, and I'm implementing sunlight propagation. I was curious if there is already some solution. The system I have can work, but it's not very fast and it's very easy to get it wrong.

[OvermindDL1]

I wanted to see how it's done because I'm working on minecraft mod that uses 3d chunks, and I'm implementing sunlight propagation. I was curious if there is already some solution. The system I have can work, but it's not very fast and it's very easy to get it wrong.

Why not a 2d matrix of the surface that details when the sun 'stops' (first opaque block)? Easy to test for, easy to pass to the GPU shaders if encoded well. You could potentially even have different 'stops' for different colors so a red skylight block blue/green but lets red continue until it hits lower, if you do not mind using 3x the memory for it (it will not be big regardless).

[emanuele3d]

Dear @Barteks2x, unfortunately I can only be partially of help.

Lights and shadows in Terasology (probably) have two components. On one hand some lighting is done in a traditional way using a deferred renderer taking advantage of a single shadow map rendered from the point of view of the main light (sun/moon). On the other hand I think there is some light propagation happening with a cellular-automata-like algorithm that feeds data to the renderer.

To make things more complicated some lighting code has been commented out more than a year ago because it wasn't working, although I suspect the behaviour you are observing isn't related to that.

Of the two lighting aspects of the rendering engine I am more familiar with the first one: the renderer using a shadow map. In this context there are a couple of guesses I can make:

1. the obstacle that should be causing a shadow is in a chunk (32x32x64 blocks I think) that isn't in memory. If it isn't in memory it can't be rendered in the shadow map and it can't cast a shadow. This should be easy to verify though: if it isn't in memory you can't see it. The view distance setting is what affects how much of the world is loaded at any given time, centered on the position of the player.
2. the shadowmap is rendered using an orthogonal camera with a cubic viewing volume (frustrum) 1000 units a side. If, for some strange reason, the shadow-casting object OR the shadow-receiving surface are not within this volume, the shadow isn't cast, isn't received or, in extreme cases, both. Perhaps the shadowmap camera offset can account for this problem. The shadowmap camera is moved 320 units away from the player in the direction of the mainlight. 1000 - 320 = 680. Following this reasoning even if a shadowcasting block was within the viewing volume of the camera, a block more than 680 units away from the camera would not receive it. This would be strange though as loaded Terasology worlds tend to be smaller than that, especially vertically.

The second lighting aspect, light propagation to fake so called "global illumination", I have not looked into it. And the bad news is that at this stage nobody knows everything about the rendering engine as the developer who wrote it has moved onto other projects, well before I jumped aboard.

The good news is that the rendering engine is undergoing a massive refactoring with the expressed goal of making it more accessible and understandable, not to mention more extensible. Progress is slow because my free time is limited but it is progress nevertheless. In this process I am getting to learn all the inner details of the renderer but there still are many holes in my knowledge.

If you want to investigate the effects and limits of point 2 above, head to the ShadowMapNode class in org.terasology.rendering.dag.nodes.

Feel free to use this issue to discuss your findings and suggestions. Or directly submit a PR if you find specific ways to improve the code.

[emanuele3d]

@OvermindDL1 : Your suggestion (a static depth map effectively, if I'm not mistaken), would only really work for straight up light sources, i.e. taking in account only overall sky luminosity. Shadows cast by the sun move (quite visibly in Terasology), and the bottom of a deep, narrow shaft should be lit only if the sun is straight above it. Conversely, the light at sunset should get deep into a west-facing horizontal cave. With some tricks your suggestion could (narrowly) be realistic in the first scenario, but I can't see how it would work for the second scenario.

In short, I'd rather find what's causing this problem and fixing it as the current shadow implementation is somewhat coarse but generally does a pleasant enough job.

[Barteks2x]

I'm almost sure that there is some light values calculated using what was described here as cellular automata algorithm, and this is I think that main part large scale shadows, and this is most likely the issue. I didn't know there are things that also depend on sun position. I haven't see the effect in the game, but that may be because I reduced graphics settings to minimum to make it work on integrated GPU (I don't want to go into details why). What I noticed is that shadows spread only downwards.

Light values depending on sun position complicate things a lot. I didn't look too deep into the code, but I certainly will.

The approach I use in my mod is similar to heightmap + the cellular automata, spreading from each block would have sunlight based only on heightmap. But even with heightmap there are corner cases that require some clever data structures. If you want the shadows to propagate infinitely down - there are several issues:

• Propagating shadows/sunlight into chunks that aren't loaded - values can be checked when the chunk is loaded
• Finding the next top block after the current top block is removed - currently there are no commands for mass block editing in terasology so it's hard to find a situation where it would matter. But imagine making a platform at y=1000 and at y=2000. The you break the platform at y=2000 and teleport to the surface. How would the game know that there is platform at y=1000 when it's not loaded? I do have a solution to that if only heightmap is being used, but it's not very fast and needs several changes.

But I still don't know how exactly light is supposed to work in terasology.

[OvermindDL1]

@OvermindDL1 : Your suggestion (a static depth map effectively, if I'm not mistaken), would only really work for straight up light sources, i.e. taking in account only overall sky luminosity. Shadows cast by the sun move (quite visibly in Terasology), and the bottom of a deep, narrow shaft should be lit only if the sun is straight above it. Conversely, the light at sunset should get deep into a west-facing horizontal cave. With some tricks your suggestion could (narrowly) be realistic in the first scenario, but I can't see how it would work for the second scenario.

In short, I'd rather find what's causing this problem and fixing it as the current shadow implementation is somewhat coarse but generally does a pleasant enough job.

Actually think of it as a heightmap, you can access it within the shader to do an occlusion of the light based on the heightmap data and your projecting through it, not just down. You will still need the normal shadow casting for nearby things, but combining that with such a projected 'heightmap' of distance is a quick and easy way to fake it.

EDIT: It would fail for the sun appearing 'below' a massive sky structure, but if that is within chunkloaded range then you can ignore that shadow heightmap data area.

[emanuele3d]

@Barteks2x: to see the effect of the shadows cast by the main light, press F3 to go in stats/debug mode and then use the arrow keys to go backward and forward in time. You'll see very clearly the effects of the shadowmap on the landscape. That been said, it only works if you have Shadows On in the video settings.

Regarding the scenario you mention, platforms at y=1K and y=2K, indeed the game doesn't know there is a platform 1K higher. I just checked and the ViewDistance class, even in the extreme setting, only loads 7 chunks in the vertical direction. That's 7x64=448 blocks at most. I did open issue #1710 to do some smarter chunk loading that would allow larger loaded world, but implementation is still far off.

In general, @Barteks2x and @OvermindDL1: I can assist you by sharing what I know and potentially where to look and how to fit things in the new rendering architecture. But ultimately if you want to improve a specific aspect or you want a new feature you are welcome to submit a PR and we can work on it together through the review process.

[Barteks2x]

I first need to figure out why it works now. And implementing working sunlight for one thing is hard enough. Implementing it twice for 2 entirely different platforms... that's quite a challenge.

I don't think it has much to do with rendering at all. This is the values calculated on the CPU being wrong.

As for the heightmap and not detecting the platform at y=1000 after breaking the one at y=2000, my solution is to have a variation on run-length encoded list of opaque/transparent flags. Most of the time it doesn't use too much memory but there are some bad cases. A few optimizations would be to write to it only when saving chunks, and keep only the top entry loaded.

[emanuele3d]

@Barteks2x: take your time and fair enough about implementing things on two platforms: it's though. That's why you should work exclusively on Terasology. :laughing:

Regarding the suggested change, I'd recommend you get in touch with @flo and @msteiger as they know more about chunks, their data structures and how they are handled. I "just" receive the chunks and render them. :wink:

[Barteks2x]

I saw the game has irc channel, is it the right place to ask more questions? Or is it mostly dead channel?

[emanuele3d]

@Barteks2x: the irc is fairly active and our project leader, Cervator, is often online. Flo and Martin (msteiger) are rarely there but you can ping them and sometimes they will come online. Another place where to ask questions is the forum, in particular the development area. Again ping Flo and Martin using the @ sign and their username as it's done here in github. Good luck! :smiley:

[Cervator]

I'm back from some time away and am catching up. Hi @Barteks2x! You may need some feedback from @immortius here as he's the one who added the "regenerating sunlight" feature you've encountered here :-)

I believe the reasoning was not wanting to forever need to store details / calculate light sourced from the sun when we added infinite world height. So only keep track over a certain vertical distance, then pretend the "sky platform" is too far away to cast a shadow anymore. Naturally that doesn't work super well for a massive platform as opposed to a single block, and it does result in sunlight in super-tall caves, neither of which are/were likely things to encounter for quite some time back during the original implementation.

I think there is similar reasoning for why the chunks are taller than they're wide & deep (efficiency in sunlight related calculations). I couldn't begin to guess at what a next round of improvements might involve, but wanted to at least provide what little background I know of :-)