Make Models be affected by Fog

[laurensdijkstra]

glTF Models are currently not affected by Fog. This creates a highly unrealistic image. It would be desirable to have Models be affected by Fog just like the Terrain Primitives are.

[ptrgags]

This feature has two parts: fog culling (tiles in the distance for horizon views should use lower-resolution tiles) and fog rendering (Blend the model color with the atmosphere color like terrain does.

Fog culling is already implemented in CesiumJS, but not enabled by default. It goes by the term dynamicScreenSpaceError. I find this a bit confusing, and the documentation could be written in a clearer manner.

This is an unfamiliar part of the 3D Tiles code for me, so I explored the code a bit yesterday and now can explain the parameters better:

Parameter	Effective range of values	Description
dynamicScreenSpaceError	{true, false}	The switch that turns fog culling on/off.
dynamicScreenSpaceErrorFactor	>=0.0 px	Fog culling works by reducing the computed screen space error (SSE) in pixels for tiles far away from the camera for horizon views. This factor is the maximum SSE adjustment. For example, if this is the default of 4.0, then tiles far away from the camera will be SSE = tile.SSE - 4 px. Increasing this value is the easiest way to increase the intensity of the fog culling's effects. Setting this to 0.0 has the same effect as turning off fog culling. Furthermore, this should be restricted to positive values, as negative values will cull tiles near the camera
dynamicScreenSpaceErrorDensity	>=0.0	The adjusted SSE falls off from the original value like a bell curve as distance from the camera increases, to simulate fog being more intense in the distance. Increasing this density parameter makes the bell curve have a sharper peak, which simulates thicker fog. Setting this value to 0.0 Has the same effect as turning off fog culling.
dynamicScreenSpaceErrorHeightFalloff	[0.0, 1.0] (a percentage of the height range)	Fog culling depends on the camera height such that the effect is largest when the camera is near "street level" and gradually falls off to no effect when the camera is above the data. This falloff value is a percentage of the height range of the data[^1]. When the camera is below the falloff point, fog culling will have the strongest effect. Between the falloff point and the maximum height, fog culling will gradually taper off. Above the maximum height, fog culling has no effect.

[^1]: The height range that is computed depends on the the bounding volume of the dataset. For smaller tilesets with a box/region bounding volume, this is based on the min/max height. For large world-scale bounding volumes, an approximate height range is computed relative to the ellipsoid.

Another observation: Fog culling rolls off with the camera angle. It has a maximum effect when the camera is facing the horizon, and no effect when the camera is facing straight up or straight down.

Here's a diagram of what the fog culling SSE adjustment looks like:

Here's what the density parameter does:

Here's what the SSE Factor parameter does:

And here's an animation that shows how culling is strongest when the camera is low, but rolls off as the height increases past the falloff point.

[ptrgags]

Before we enable fog culling by default, we'd like to do some performance testing to evaluate how well it works and what parameters would be the best defaults.

For the first round of tests, I'm using Google Photorealistic 3D Tiles to test how well this feature helps reduce tiles loaded for world-scale datasets. This involves

1. Measure initial tiles loaded time and tiles loaded/selected for the dataset without fog culling
2. Measure it again with the default fog culling settings
3. Try increasing just the density parameter to have the maximum effect (I'm going to try setting it rather high at 0.1)
4. Try increasing just the SSE factor parameter to simulate somewhat aggressive fog without going crazy. I'm going to try 32 px. This is twice the default maximumSSE threshold, and causes a noticeable difference in the background of landscape views.

I've already started some of these measurements, I will post the data and a summary once I have the rest of the numbers.

[ptrgags]

P3DT Fog Performance Test Results

I ran the above tests for comparison. I used 5 different areas of Google P3DT, and for each area 3 views (aerial, street level, top-down) for comparison.

Here's an example HTML file for one of the test setups. I adjusted the parameters in the tilesetOptions object and then refreshing the page as needed). performance-example-setup.html.txt

Here's a summary of the time to first load, averaged over all the different views:

Configuration	Average time to first load (s)	% difference from no fog
No Fog	7.989	N/A
Fog enabled with defaults	5.678	-28.93%
Fog with SSE Factor = 32	3.658	-54.22%
Fog with density = 0.1	5.403	-32.37%

For the full data (which includes other metrics like number of tiles loaded), see this CSV file: Dynamic SSE Perf Testing - P3DT.csv

[ptrgags]

Here's some visual comparisons of the data in CesiumJS vs Google Earth to compare how much is culled. It's hard to get the camera views exactly the same since the camera controls are so different, but here's something roughly similar:

CesiumJS P3DT	Google Earth	Comments
		CesiumJS seems to load more tiles in the distance.
		Here Google Earth has a bit more detail in the distance
		Google Earth has a bit more detail in the distance

So overall, it seems like dynamicScreenSpaceErrorFactor: 32 does create somewhat intense fog culling, but not extremely so.

[ptrgags]

Another thing we wanted to check: Does fog culling serve as a workaround for horizon culling for world-scale tilesets? Freeze frame will help here:

With fog off:

With fog on, (SSEFactor 32):

As I suspected, it doesn't completely replace horizon culling (as tiles on the other side of the globe still render), but it does reduce the LODs loaded. Not perfect, but definitely an improvement!

[ggetz]

Thanks @ptrgags for the research!

1) There does appear to be a significant performance gain from implementing fog, so we'll likely move forwards with the process 2) Based on the screenshot comparisons, we may want to do a bit of tweaking to the default parameters to get the best balance between performance and visual fidelity, but we're definitely in the right ballpark 3) It sounds like horizon culling, or even outright culling a tiling which is completely in fog, may provide and additional benefit, so we may want to take a look at that next after this is completed

[ptrgags]

I started prototyping a very barebones FogPipelineStage to add some code to the model shader to see how the czm_fog() function works. I was happily surprised that it (+ AutomaticUniforms + Fog) automatically handles applying fog and even the camera tilt.

Here's Google P3DT with czm_fog() applied (with a hard-coded false color for now) compared to a similar view of San Francisco with Cesium World Terrain. The comparison here is how much of the model is in fog with default settings:

There's still several more details to determine:

• Figuring out how to pass all the necessary variables for computing the atmosphere color to the model shader, as this needs to be passed to the fog function.
• Determining when exactly to enable/disable fog. Certainly there's scene.fog.renderable, but from the Globe shader there are quite a few defines that impact what code runs.
• It looks like the Globe shader also gives the option to do the fog calculations in the vertex color as a performance/visual quality tradeoff
• The globe shader also has several other details about when lighting is/isn't applied, tonemapping, etc. I need to learn what's important for fog in general and what parts (if any) are globe-specific.

[ptrgags]

More notes from exploring the existing globe shader and a discussion with @lilleyse:

• The sky atmosphere, ground atmosphere and fog currently are enabled and configured independently.
• Dynamic lighting (based on sun position over time) affects fog too.
• see this Sandcastle for some of the relevant settings for turning off the atmosphere but keeping fog (while enabling dynamic lighting
• To get the fog to use the atmosphere color, there are several uniforms in AtmosphereCommon.glsl that need to be added to the Model shader. These uniforms can be found in the Globe (for ground atmosphere/fog for terrain) and SkyAtmosphere, but neither are accessible from Model.
• One thought is to make these AutomaticUniforms, so the Globe settings will also apply to 3D Tiles fog.
• Longer term, we could even combine the SkyAtmosphere parameters with the Globe so they only need to be configured in one place. But this is out of scope. See #11681

[ptrgags]

Update: I added some new automatic uniforms and created some new builtin functions for computing the scattering color. I also created a new Atmosphere object (stored as scene.atmosphere) to handle updating the FrameState.

The shader for FogPipelineStage is still very bare-bones, there's quite a bit of code I have to add from GlobeFS. Also the color doesn't yet look correct, it's quite dark when screenshots above are lighter and a bit more bluish in comparison. But the fact that you see something and not a shader compilation error is progress.

There's still a lot left to do here, so let me organize my thoughts:

• [x] Debug why the atmosphere looks black. Is it due to code I haven't yet added? uniform optimized out?
• [ ] Why does everything above the horizon look darker?
  • [x] Track down what variable is causing the problem
  • [ ] How to compute ellipsoid position correctly above the horizon?
• [x] Handle dynamic lighting
• [ ] Determine how to toggle vertex shader vs fragment shader fog computation like the globe does.
• [ ] Sweep through GlobeFS and GlobeVS thoroughly and see what other details affect fog that are not globe-specific.
• [ ] Add a uniform for whether the tile is in fog or not (as fog should only be computed for distant tiles). This likely will be similar to the dynamicScreenSpaceError computation
• [x] Allow enabling/disabling fog through the settings. Sometimes this will require the Model pipeline to be rebuilt for infrequent changes to the shader code.
• [ ] Determine how to unit test this code.
• [ ] Set up performance tests before and after fog
• [ ] Determine how best to test this. Unit tests may be difficult since this requires rendering things far from the camera. Playwright testing might be a better idea.
• [ ] Plan a deprecation strategy for #11681 and document it, as the change to the GlobeFS is not exactly trivial given the differences in uniforms and defines.

[ptrgags]

There were a couple typos so a couple uniforms were NaN, fixed them. Now the coloring looks better, but now I get this artifact where everything above the horizon looks darker:

I'll have to investigate further.

[ptrgags]

It looks like the scattering code is likely where it happens, though I'm still trying to figure out why.

Along the way, I'm finding some corner cases with some of the variables that seem a bit off to me, though these don't seem to be a problem for terrain. Documenting them for future reference.

First, the variable w_stop_gt_lprl (weight for representing whether [ray intersection].stop greater than length(primaryRayLength)). This is a measure of "sky or horizon?" It seems to be sensitive to the difference between an ellipsoid and a sphere since it uses a sphere for the intersection test.. It seems to hover around 0.5 due to the 0.5 + 0.5 tanh(x) almost everywhere, but when the camera zooms in near the poles it jumps to 1.0 due to the camera being above the ellipsoid but underneath the sphere used for the test.

Second, the variable w_inside_atmosphere has a weird behavior on the horizon but only for certain camera views. it's 1 wherever it is yellow. The yellow at the bottom seems consistent with the behavior in most views when the camera is hovering close above the earth, but not sure why it's also 1 on the horizon:

Still poking around the code a bit. I also want to do a bit of reading up on atmosphere rendering to understand the scattering functions better.

[ptrgags]

This article was a helpful resource at understanding the concepts here about Rayleigh and Mie scattering, with plenty of good diagrams.

[ptrgags]

Well that's one thing that seems off. For fragments above the horizon, the lightHeight computed in the inner loop seems to be negative somehow :thinking:

lightHeight = length(lightPosition) - innerAtmosphereRadius so either the reference point (which is based on positionWC for the fragment) is wrong, or the light position calculation is wrong for fragments above the horizon.

[ptrgags]

Huh, when rendering (innerAtmosphereRadius / 1e7, length(lightPosition) / 1e7), both variables seem to get noticeably larger above the horizon. not sure why.

[ptrgags]

Hm, rendering positionWC = computeEllipsoidPosition() (same exact function as in GlobeFS.glsl)

I get weird values. the reddish colors above the horizon would mean +x here which is definitely wrong (+x is eastern hemisphere, this view is California). Also not sure why there are two colors.

Terrain for comparison for a similar view. All blue, even above the horizon.

So something about the coordinates used for the computation must be different between globe and Model... though this function uses only built-in uniforms so not sure what's different here.

vec3 czm_computeEllipsoidPosition()
{
    float mpp = czm_metersPerPixel(vec4(0.0, 0.0, -czm_currentFrustum.x, 1.0), 1.0);
    vec2 xy = gl_FragCoord.xy / czm_viewport.zw * 2.0 - vec2(1.0);
    xy *= czm_viewport.zw * mpp * 0.5;

    vec3 direction = normalize(vec3(xy, -czm_currentFrustum.x));
    czm_ray ray = czm_ray(vec3(0.0), direction);

    vec3 ellipsoid_center = czm_view[3].xyz;

    czm_raySegment intersection = czm_rayEllipsoidIntersectionInterval(ray, ellipsoid_center, czm_ellipsoidInverseRadii);

    vec3 ellipsoidPosition = czm_pointAlongRay(ray, intersection.start);
    return (czm_inverseView * vec4(ellipsoidPosition, 1.0)).xyz;
}

[ptrgags]

In SpectorJS, the only difference in uniforms is the frustum is quite a bit different. Model on the left, terrain on right:

[ptrgags]

Today I've been focusing on other details of 3D Tiles fog and thinking towards the future of atmosphere rendering.

See https://github.com/CesiumGS/cesium/issues/11681#issuecomment-1865080064 for more details on what I think the atmosphere setting structure should be (although getting there requires quite a few deprecations). For now, I'm sticking to non-breaking changes until it can be discussed further.

Then I wired up the flags that enable/disable the fog pipeline stage. fog.enabled/fog.renderable are the main settings that control this. This is simpler than for terrain which is tightly coupled to whether the globe exists.

[ptrgags]

I added an enum to select between the 3 dynamic lighting modes (OFF, SCENE_LIGHT, SUN_LIGHT like I described in the other issue (as this is cleaner than 2 separate flags that depend on each other). Here's it in action:

• black fog: scene light. I don't think I had the right direction vector for this view.
• sunset fog: sun light
• pale blue fog: dynamic lighting off; fragments are lit from above.

For safekeeping, here's a Local Sandcastle for WIP branch 4196-3d-tiles-fog

[ptrgags]

@ggetz I clarified the product details of 3D Tiles fog with @shehzan10 today, and he clarified that right now the priority is focusing on 3D Tiles performance for world-scale datasets like Google P3DT. The visual rendering details are at a lower priority for now. So I'm splitting this issue up into the following smaller issues:

• https://github.com/CesiumGS/cesium/issues/11715 (high priority)
• https://github.com/CesiumGS/cesium/issues/11716 (lower priority)
• https://github.com/CesiumGS/cesium/issues/11717 (lower priority)
• https://github.com/CesiumGS/cesium/issues/11681 (lower priority, involves breaking changes)

I'm closing this issue in favor of the 4 above.