Shadowing

[Midda-C]

So there have been a number of instances where sections of the ocean have been out of direct sight of the sun, and it appears to glow when this is contrasted against some rocks or landmass which is receiving shadows (or just the regular NdL shading). I've attached an example.

I understand this is a common issue with transparent objects, so I'm not sure much can really be done, but I thought I may as well see if you could think of any ways to approach this? I suppose screen-space reflections might reduce the issue on the fresnel, but I'm not sure about the subsurface lighting.

Not a pressing issue by any means, but I figured I'd at least get it recorded. :)

[huwb]

Ha was wondering when this would come up! Yeah I should be able to shadow the direct lighting with a shadow sample using unity stuff. Scatter light from the ocean volume might be a bit trickier, will need to look into it. Static light probes should give the ambient light, but im not sure if/when this will work. A hacky route would be to just give a slot to sample a world-aligned texture that gives AO/scales scattered light..

I'm taking the "its not a bug, its a feature" route and marking this as enhancement, for now. If this becomes higher prio let me know. I've been having some luck making a foam sim (persistent_foam branch) and think ill stay in that groove for a bit longer. Anyway I'll let you know when i get onto this.

[Midda-C]

Yep, don't want to interrupt your groove, as I said, this isn't a pressing issue. :) I'm happy enough for now that it already sounds like you have a good idea of how it could be approached.

Keen to see what you've been doing with the foam!

[huwb]

There is a mention of shadowing in this presentation as well (thanks for the link)

https://youtu.be/0KA1PszNtZw?t=2444

[huwb]

I started doing some reading about shadowing - trying to unpick various threads etc. Dumping some thoughts..

The standard unity renderer makes this difficult. There is a _ShadowMapTexture for the main directional light which is replaced with a screenspace shadow mask, and the original shadow map is gone (this is a surprising decision to me, but i guess there is a reason..). (EDIT: In hindsight this was not correct, it was a combination of misinformation and misunderstanding on my part. The shadow map is rendered to a depth target and does not disappear or get replaced... but this is shadow map per-light data and unity will not bind it to normal shaders because it doesnt make sense to do so. To get to the data a light command buffer is required, as I have done below..). This mask computes shadows for opaque surfaces and will not include the water. The mask is actually invalid for transparent surfaces and is not available for the ocean shader. This is evidenced by the numerous people asking for shadowed transparents/shadowed particles which do not seem to be supported (!), and by NReed's post: https://answers.unity.com/questions/649234/can-shaders-access-directional-light-shadow-maps.html . Additional (but old) discussion: https://forum.unity.com/threads/no-shadows-visible-on-transparency-shaders.9909/ . i also read somewhere that the option to use screenspace shadow mask is an option in the graphics settings asset (not exposed in the editor UI), and depends on platform, but I didnt test this, and it did not sound like reliable solution.

An alternative solution might be to add a new LODData that renders a plane at the sea level and writes the shadow mask value into a texture - basically assume the ocean is a flat surface, render an opaque plane there and save off the resulting shadow values on the plane for use during ocean surface rendering. This would not be 100% correct but may work ok (and could potentially be parallax corrected to improve accuracy). It could also be gaussian blurred to approximate subsurface scattering (similar to how it is done in screenspace methods for skin). I'm quite interested in this. The downside is that Unity will do the whole screenspace shadow stuff each time - could get expensive. Perhaps its ok if its only done once or a few times at some "medium" resolution(s). Caching would also be an option, although it would require static light direction/time of day, which is probably too constraining to commit to. Unless I have other ideas, i guess exploring this further is the way to proceed for now..

The lightweight render pipeline appears to be more flexible. See https://www.youtube.com/watch?v=0KA1PszNtZw&feature=youtu.be&t=2444 . I don't know yet if i should embark on supporting other render pipelines besides the standard unity stuff, as it could take a lot of work to stand everything up for each one.

[huwb]

First bit of visible progress - i have added a new lod data that measures shadows on the horizontal plane, which can then be used during rendering. i've so far plugged it into the specular shading on the water and on the foam:

next steps will be to try using it in the scatter colour to darken the water. as i mention above it might need a blur pass to get a nice, scatter-y type distribution.

[huwb]

having some fun with this :). i can use this data to mask out caustics as well:

[Midda-C]

That's looking great! So, it's sort of like another sim, then? Does that mean it'll need more layers?

[huwb]

Ahhh yes it currently does. How many layers do you have left? What sort of layers do you have of I can ask? Just curious.

We're looking at adding more loddata types, which would mean more layers in the future. Do I need to rethink how crest works with layers to avoid using them, if I can?

Do you have a ShadowProxy layer or equivalent? This one might be able to reuse that.

[Midda-C]

We've used a lot of layers, I believe largely related to our game's procedural generation stages. @dizzy2003 is the best to talk to about that, though. I think we have one or two layers we can reclaim, but we're scraping the bottom of the barrel at the moment.

[dizzy2003]

Most of our layers are actually due to physics. Very annoying that unity shares layers between camera and physics, they said they were going to fix this 2 years ago..

If I re write some of our specialist camera renders to use Command Buffers or DrawMesh(Camera) I can probably free up 4 layers, this is something I've been meaning to do anyway at some point.. And I think we have 1 unused..

On Tue, Aug 28, 2018 at 3:44 PM, Huw Bowles notifications@github.com wrote:

Ahhh yes it currently does. How many layers do you have left? What sort of layers do you have of I can ask? Just curious.

We're looking at adding more loddata types, which would mean more layers in the future. Do I need to rethink how crest works with layers to avoid using them, if I can?

Do you have a ShadowProxy later or equivalent?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/huwb/crest-oceanrender/issues/25#issuecomment-416458535, or mute the thread https://github.com/notifications/unsubscribe-auth/AOU9GEOw1eWacSrXgI34W--D8IblwI4Wks5uVNjHgaJpZM4Ufewt .

[huwb]

Alright i got something pushed that could be tested: 46926d044dda6528d5f3ff6cc22d4baeffe47047

I am having a hard time settling this new feature. The idea was simple - reuse the shadow maps already rendered by unity to shadow the ocean. Unfortunately on the whole it hasn't come together into a tight/efficient system..

1. By the time rendering happens, the cascades have been converted into a screen space shadow mask (as i wrote above)
2. This screenspace mask doesnt work with transparents (only the frontmost opaque surface), and the mask is not available to read from at all for transparent shaders.
3. Ok.. in this case i'll render an opaque horizontal plane and let unity do its thing computing the shadows, and then store the shadow value for use during ocean rendering. This works, but triggers the unity shadow system to do its thing for this camera - so renders all shadow casters X 4 shadow cascades X 7 LODs - ouch..
4. To reduce this, i realised with minimal change to the systems i could have shadow lod data exist on only 2 lods - for e.g. lods 2 and 3, and blend smoothly between them as lods change scales and avoid pops. See the screenshot below - the column with just 2 textures. These lods change scale with the rest of the chain so they adapt to the viewpoint. This is actually pretty nice, didnt realise this would work. The downside is that shadows have limited range in this case - its pretty easy get out of range of the 2 shadow lods and see the shadows missing, but its not too bad and is an artifact that is pretty common in games.
5. However this still requires computing shadows for 2 lods - better than 7, but not crazy efficient. On my laptop this may have added up to 2ms to the render time (the shadows are ~80k tris).
6. An option would be to render just ONE lod, and put up with slight pops when the viewpoint changes altitude. This was also surprisingly easy to try out and i parked it in a branch - commit: e254a6c3f4472116ea1f057bf5831101ff30c539 . Its visible if you look for it - especially if the camera is hovering around the lod scale change height - it could flick backwards and forwards and be pretty flickery. I'm generally anti-pops but maybe this is good enough in some cases.
7. Even with only one shadow data (6), its still an extra shadow computation which really doesnt feel necessary, and could be heavy if you're rendering a lot into the shadows.
8. The only other idea i have is to implement a ShadowCache, similar to the DepthCache. I guess this is my next move. EDIT: Note that if the sun direction changes, the cache is invalidated. Would/could you guys use a cache?

On top of all of the above, the system requires 1 layer as usual, but its a bit of a special one - it needs a ShadowProxy layer (name can be configured) - a layer with all the stuff that will cast shadows. Unity is really particular about when it does its thing with generating shadows - i've tried loads of things over the last days, even calling Graphics.DrawMesh() and specifying receiveShadows=true fails to trigger shadow generation if there are no Renderer components in the scene which are enabled and in the right layer.. Do you guys have such a layer/is adding one a possibility? Do you guys use shadow proxies?

Perhaps you guys can discuss and let me know if you have questions.

[Midda-C]

Okay, so I had a response typed up here about how I couldn't get any shadows to render, but I think I misunderstood how this is supposed to be set up. So, if I'm understanding correctly, all of the shadow-casters in the scene have to be on the same layer? If so, I don't think that's really going to be a workable solution for us, as we have objects on particular layers for physics purposes.

Could we specify multiple shadow-casting layers (really, I don't think our actual renderable meshes are spread across that many layers, maybe 3 or 4), with the shadow proxy object on its own layer which is hidden from the camera?

E.g. we create a layer which your big red shadow plane goes on, disable that layer in the main camera's culling mask, then on the Ocean Builder component, we can say which of our layers are shadow casters.

As a test, I set the shadow layer in the Ocean Builder script to use the Default layer, then at runtime, changed the RenderSim_Shadow object to be on the ShadowProxy layer (which the main camera doesn't render). Then—again at runtime—I moved some other objects like our terrain and boat on to the default layer just to get them included in the shadows. It worked, and the shadows looked quite nice (though the low resolution of the shadow map was apparent with moving objects). Again, all of these objects are on specific layers for a reason, this was just to test the visuals.

However, does this method mean all of the shadow-casters are being rendered a second time? Because that'd probably be a substantial performance hit for us.

Regarding a ShadowCache, I expect that'd be a good feature to have for projects with a static light, but we have a dynamic time of day, so that wouldn't work in our case.

It's a shame that this is turning out to be such a complicated feature. I guess this is the reason why Unity started the Scriptable Render Pipeline. :(

[huwb]

I guess this is the reason why Unity started the Scriptable Render Pipeline. :(

THIS!! I was going to write exactly the same thing...

Thanks a bunch for talking this through, I think you've understood perfectly.

Yes it could be multiple shadow caster layers - I think - actually i would need to check. But yes it would mean another shadow render pass and subsequent screen space resolve which is pretty dismal given that the required shadows were already rendered by unity for the main scene pass!!

There might be a cheeky way i can get access to the rendered shadow maps before they get converted into the screen space mask. I'll post back if i have any luck.

In the meantime someone just posted a thumbs up for HDRP.. i need to dive into the new scriptable render pipeline stuff and try to figure out if I'm going to need to support N different renderers! If i learn something interesting i'll share and tag you.

[huwb]

Guess I'll remove the "good first issue" tag now lol

[Midda-C]

Man, it's so frustrating that the data exists and Unity just doesn't let you use it!

On the brightside, the shader side looks great!

I don't really know anything about how it works, but this guy made an impressive screen-space shadow solution using the depth buffer and light direction, perhaps something along those lines could be useful?

https://forum.unity.com/threads/released-se-screen-space-shadows.444974/

[huwb]

Thanks for that link, good to have on record. I guess its a screen space raymarch of one nature or another.

I think it could be a lot of work to get this working stable/high quality/fast and a bit scary for me, at the moment anyway. I'd be more inclined in seeing if i can get screen space reflections going from the out of the box unity stuff which should also provide this kind of shadowing. But it looks like it needs gbuffer data, and the ocean doesnt write to the gbuffer, so thats unlikely to work.

All of the screenspace things suffer from badness when the shadow casters go off screen.

I still plan to see if i can get at the shadow maps unity renders..

[huwb]

I spent some more time, I was able to make small steps forward but have not got very close to a solution yet.

There were many interacting factors that were difficult to unpick. I'm not quite sure how, but i have a version of the ocean which is actually getting passed the shadow map depth texture. I don't know if its by luck or something stable, but even if it does turn out to be luck it should be possible to copy the depths out by adding a hook after the shadow map render (https://docs.unity3d.com/ScriptReference/Light.AddCommandBuffer.html).

One complication is there must be a shadow receiver on screen at all times otherwise unity will stop drawing shadows, but it might be enough to tag the ocean tiles as receivers which i havent tried yet.

I was also able to force the shadows to appear by manually expanding out the shadow macros (these are usually stubbed out by unity for shaders in the transparent queue), and can sample the shadow cascades manually. Again the necessary shadow matrices etc are probably only there by luck as unity wont be setting them for the transparent pass, so it may be necessary to compute them and set them myself if it stops working.

The problem now is how to sample the correct cascades and get a nice result. It turns out, I had an oversimplified mental model of how shadow cascades work! I can hardcode it to sample one of the cascades for now and i get some shadows on screen.

The shadows give either sharp or jaggy results, it won't be soft shadowing, which will be a problem, as it wont give a nice appearance of soft scattering on the water. I will think about how this can be overcome.

I've seen in the LWRP there is shader code for sampling the shadow cascades and it looks like it might help, but i dont know how much adjustment/massaging it will need before it plugs in. ( https://github.com/Unity-Technologies/ScriptableRenderPipeline/blob/42ce8fe3bab5aa12d8fcbad4d90f8bf3d7f5fad3/com.unity.render-pipelines.lightweight/LWRP/ShaderLibrary/Shadows.hlsl )

[Midda-C]

Well, I don't have much to add other than my apologies for sending you down this rabbit hole. :P That LWRP video made it look so easy!

That shadow sampling code sounds promising, hopefully it's not relying on anything exclusive to the scriptable render pipeline.

[huwb]

I think the reason why unity exclusively does screen space shadows is its probably infeasibly expensive to try to compute soft shadows in a forward pass. I think this is a problem for us..

After some more thinking, perhaps a good solution might look something like this:

• Have a shadow lod data like I did above
• Populate this data by directly sampling the shadow map cascades, instead of making unity re-render the shadows, by getting the data out from the rendered cascades as mentioned above, and computing the sample position in the correct shadow cascades by replicating the logic in unity
• This could be persistent across frames and could therefore do TAA stuff - the sample could be jittered to get soft shadows, and the sample could blend with previous frames result to anti-alias the result
• Also the cascades cover only a limited area of the world, but the shadow lod data will cover a much larger area around the viewer. If a shadow query misses the current cascades, it could potentially still take the data from the previous frame so that it's not lost. This could potentially get rid of aliasing at the edges of screen and stuff like that where information is not available off screen.

[Midda-C]

Sampling over multiple frames sounds like a good solution. If I'm understanding correctly, you're not likely to notice any artifacts from this with stationary objects, right? Would an object that's moving relatively fast have any noticeable artifacts? Though I guess the general motion of the surface would probably hide that pretty well it they did.

[Midda-C]

Whoops, didn't even realise there's a "Close and Comment" button! :P

[huwb]

After some intense struggles and giving up a few more times, i think i'm finally winning the battle with this stuff..!!

I implemented the above as described. Here's a video:

https://drive.google.com/open?id=1cLFjlM3kLZmPBWaauy8-QVEtOGMZdodP

This just works from the existing shadow maps so should be efficient. The video above does not do any jitter/blurring so the shadow looks unnatural in the scattered colour, ill try the jitter stuff next.

Sampling the shadow maps directly is very deprecated so i dont know how long this stuff will work for, but its working for now. Implementing into LWRP / HDRP would be much more straightforward than this was, so in the future i can see it evolving.

For the temporal stuff, i'll expose a slider for how much to take from previous frames and i reckon it should work ok. There are a couple of things we could do to address this if it is an issue, such as rendering two channels with different settings and picking different channels for different shading components.

Next steps - get FOV passed in properly to remove a fudge in the shader, ensure shadow data gets passed up and down lod chain properly, look at doing some refactoring to make this slightly weird lod data sit well with the other types, look at the jitter stuff, expose sliders, test with objects.

[huwb]

Nifty :)

https://drive.google.com/open?id=1qDdQ1JX5FQaiQzOpvxHF5rcOsP-6goNZ

Inching closer..

[huwb]

One more - motion test with different filter speeds/responsiveness

https://drive.google.com/open?id=1UVtyuRaB3MAQuzsTFUVq40q2bJzpACmL

[Midda-C]

Nice! had a quick play around in the Crest demo scenes, and it works pretty well! The temporal accumulation holds up nicely. I'd have been happy with just the static objects casting shadows, so it's great that the dynamic objects will be accounted for too. :)

Can't wait to get it in our game! :D

[huwb]

Cool :)

I made some final fixes/tweaks/docs and merged it into master. Theres still a pop im still hunting down which occurs in some situations, the fix has proved elusive so far so I'm gonna mull it over and get back to it. Let me know if you notice it etc, but dont bother making a video or anything.

Oh and I forgot to mention before, it doesnt require any additional layers, for once :).

To turn it on, enable Create Shadow Data on the OceanRenderer script, and provide it with the primary light in the scene (I didnt find an elegant way to just know this, unfortunately). I added some validation so it asserts light type, that shadows are enabled on the light, and some stuff like that.

Due to the way i hooked everything up, if there are no shadow receivers within range, the shadow buffers wont generate, and none of the ocean shadowing will run. I'm not sure how much of a problem this will be in practice (there are hacky workarounds that we can discuss if it is).

Some of the above would be nicer/easier with a scriptable render pipeline but hopefully it all works well enough to be usable. Let me know when you've had a chance to try it!

[huwb]

Finally fixed the pop issue i mentioned above, pushed to master.

Oh and make sure Shadows is enabled on the ocean material.

[huwb]

I squeezed in one more change - i've added a second shadow channel for hard shadows. It has much less jitter and a much faster filter, and i use this faster shadows for everything except the light scattering in the water. It seems to work well and the overhead was within measurement error on my laptop so i pushed it to master.

[huwb]

Captured a video for a presentation, thought id post it here. Kinda fun to watch the shadow data populate as the view moves around :)

https://drive.google.com/open?id=10YHSCmehpYuN3eQIoYpnoix3u9KuOVG6

Don't suppose you've had a run with it yet have you? Let me know if you need anything.

[Midda-C]

Unfortunately not yet. @dizzy2003 has made some local changes to Crest, and I'm not sure how to merge them across myself. I'm keen to see it all in there though, maybe I'll have to give him another prod. :P

[huwb]

Sure no worries. We're constantly rewriting and refactoring so i can imagine getting latest is no easy task.. but hopefully worth it :)

Let me know if theres anything that makes sense to merge up... or anything i can do in terms of structuring the code to make these easier. besides modifying the way the ocean depths work based on your last suggestion, I don't feel i've done as much as i could be doing to support and not collide with local changes.

[Midda-C]

Okay, so I'm finally getting a chance to play with this in our project. I'm having issues assigning the Primary Light parameter, though.

[Error] [0.020] Primary light must be specified on OceanRenderer script to enable shadows.

We're having to dynamically assign the light in the Ocean Renderer object, for a few reasons. One, our lights don't exist until we run the game, two, they're in a separate scene to Crest, and three, the main light source changes between day and night.

@dizzy2003 has set the light to be assigned at runtime, and the "Primary Light" field is filled in correctly, but we're still seeing this error and no shadows. He tried a few tests to get things working, and managed to get some data into the shadow sim, but still wasn't seeing shadows in game.

[dizzy2003]

I tried a quick hack of making lodDataShadow not run Start Update or LateUpdate until a light was assigned.

I did get some data in the shadow textures in the gui debug, but no actual shadows in the ocean. (as in the screen shot)

Perhaps there are other places were the init order needs to be delayed until after a light is assigned. This wont fix our issue with the main shadow casting light changing but would at least get us shadows in the day..

(if its not clear the shadow you can see is on the terrain under the water)

[huwb]

Ah I've not tested with changing the light at runtime. I'll experiment here and get back to you.

[huwb]

Oh cool. There's a Shadows check box on the ocean material, is that enabled? (I really should drive that toggle from code..).

[dizzy2003]

Just checked, and shadows is ticked.

[huwb]

Ok - i'll see what i can dig up when im at home.

I will add a debug vis mode to the ocean material to visualise the data. That would #define in some code like this at the end of the ocean shader:

col.rg = shadow.xy; col.b = 0.;

If you get a chance to hack that in it would be interesting to know if anything shows up at all or if its all white. But as i said ill investigate on my side, thanks for the info so far.

[huwb]

I didnt have much time today but i messed around a little bit. I also hacked it so that it waits for a light to be assigned before initing. And I added a Debug Visualise Shadows toggle at the end of the ocean mat.

It seems to work for me. However I might have hacked it differently - I pushed this to a temporary branch referenced above, could you compare and perhaps try my version if its different?

Also could you try the debug vis and confirm nothing is getting through to the shader?

Thanks, sorry about the turbulence!

[Midda-C]

So it turns out @dizzy2003's test hack to assign the light did actually work, it was just too subtle to see with his material settings. I've got his changes and can confirm that shadows are working (and look awesome!).

So now I guess we just need a way to achieve it in a less hacky manner, and allow for the light source to be changed when necessary.

Question about the shader side of things. From what I can see, the shadow map basically masks out the sub-surface scattering, correct? Would it be possible to also have a slider for it to influence the base diffuse colour too (so 0 would be as it currently is, and 1 would be very dark diffuse)?

I don't know if that'll look correct or not, but it'd be nice to have some way to bump up the effect a little under certain conditions, and also so shadows are still usable for people who haven't enabled sub-surface scattering.

[huwb]

Ah I'm glad to hear that!!

Yes for sure, I'll report back when done. I want to make it support changing the light at runtime. I'll also add something for the diffuse as you suggest.

[moosichu]

@Midda-C @dizzy2003 I've created PR #82 which should hopefully handle lights changing at runtime. Once @huwb's given it a look to make sure it's all ok, it should get merged into master.

On another note, is it ok if we close this issue now? That way we can create individual issues for any problems/desired features for shadows so we can better keep track of them.

[huwb]

Thanks @moosichu ! I've merged it in now.

Guys, we added a warning if there is no light assigned on startup, you can disable it in the ocean settings. Let us know if it works for you.

I also just added shadow colours in the shadow section. If you don't end up using either of these settings let us know so we can remove the code.

Let us know if this gives you what you need to switch shadows on and then we can get this issue closed as @moosichu suggests.

[Midda-C]

Hey, sorry about the delay (just had a long weekend here). So I've got latest, and I had a bit of trouble getting it to work.

For some reason, I'm not getting any shadow data when I run the game. I brought up the debug GUI to verify, and it also showed no shadow sim data. However, I noticed the "Process Shadows" toggle in there, which was already turned on. I turned it off and on again, and then shadows appeared.

Issue is, the only way I've been able to get shadows to work is by manually doing this every time I run the game. Also, it needs to be toggled off and on again when the light source changes.

I've got shadows enabled everywhere I can see—'Create Shadow Data' is ticked, I've added a shadow sim settings asset, I've checked 'Allow Null Light', and shadows are turned on in the material. I've also tried deleting and recreating the shadow sim asset.

Any ideas?

[huwb]

Thanks for all the info & testing - it turned out this was a bug that slipped through the net. I could repro it by duplicating the direction light, and then switching between them at runtime. The state in the lod data was not fully cleared, so the command buffer was not moved to the new light. Toggling the process checkbox triggered this to happen. I retested the other paths and it all seems good now.

Apologies and let us know if it works for you.

[Midda-C]

Yep, all working flawlessly now. Thanks! The shadow colour options work well too.

Really happy with how this looks. Awesome job, as usual! :D

[huwb]

Oh man.. issue 25 getting closed.. this is a milestone :)