Augmenting Nu with 3d capabilities.

[bryanedds]

A while back, I started working on this in the 3d branch - https://github.com/bryanedds/Nu/tree/3d

So far I've mostly just 3d-ified the Transform type and converted the 2d portion of the engine. Since everything has gone so well so far, I wanted to take a moment and talk about the features I am planning for 3d.

Planned Features (while all planned features may not be available immediately, all of the data structuring and engine scaffolding will be in place to support them) -

• [x] Add OpenGL .NET Bindings
• [x] Convert Color to 32-bit floats and add #RRGGBBAA parse.
• [x] 2d Renderer Threading
• [x] 2d Renderer OpenGL reimplementation.
• [x] 3d Transform System (converted from 2d)
• [x] 3d Rendering, using
  • [x] Physically-Based Materials (Highest Priority)
  • [x] Deferred Drawing (Highest Priority) (ex. https://github.com/GameTechDev/OpenGL-ES-3.0-Deferred-Rendering)
  • [x] Assimp Integration (Highest Priority)
  • [x] Discriminating Octree (Highest Priority)
  • [x] Sky Box (Highest Priority)
  • [x] Specular IBL (Highest Priority)
  • [x] Light Parameters (Highest Priority)
  • [x] FBX Scene Import (Highest Priority)
  • [x] Material Shader Coloring Support, plus Import (Highest Priority)
  • [x] Animated Billboards Sprites (Highest Priority) (for Final Fantasy Tactics-like characters)
  • [x] Material Emissive Support (rename existing fields names 'Glow' to 'Emission'?)
  • [x] Per-object relative light sorting for forward rendered objects. (Highest Priority)
  • [x] Point Lights
  • [x] Directional Lights
  • [x] #490
  • [x] #491
  • [x] #481
  • [x] #488
  • [x] #489
  • [x] #500
  • [x] Opaquing Translucency (alpha in forward render lerps to 1.0 as approaching > 32m distance from eye)
  • [x] #503
  • [x] #565
  • [x] #562
  • [x] #523
  • [x] Depth Shadow Mapping
  • [x] Variance Shadow Mapping
  • [x] Exponential Shadow Mapping (details on artifact mitigation - https://learn.microsoft.com/en-us/windows/win32/dxtecharts/common-techniques-to-improve-shadow-depth-maps?redirectedfrom=MSDN)
  • [x] Animation parallelized and from main thread (one frame delay incurred. need job system?)
  • [x] #650
  • [x] #725
  • [x] Streaming Texture Loading (during renderer down-time since OGL isn't multithreaded)
  • [x] Screen Space Reflection for up-facing and down-facing surfaces (https://lettier.github.io/3d-game-shaders-for-beginners/screen-space-reflection.html)
  • [x] Volumetric Fog (https://web.archive.org/web/20230623235630/http://www.alexandre-pestana.com/volumetric-lights, https://interplayoflight.wordpress.com/2015/07/03/adventures-in-postprocessing-with-unity/)
  • [x] Contained light map subsumption
  • [x] Per-light-probe ambient properties
  • [x] #885
  • [ ] #833
  • [ ] #796
  • [ ] Shadow Caching (by detecting (Box3 Set) bounds changes per shadow w/ exception case for animating objects)
  • [ ] Decals
  • [ ] #884
  • [ ] #695
  • [ ] Algebraic Foliage with some decent means of Occlusion Culling
  • [ ] Dynamic / adaptive tone mapping
  • [ ] #873
  • [ ] #903
  • [ ] #379
  • [ ] #352
• [x] 3d Interactivity (Very Basic) in Gaia (Highest Priority)
• [x] 3d Interactivity (Widget-Based) in Gaia
• [x] 3d Physics w/ Bullet Physics Integration (Highest Priority)

I'm hoping to constrain my implementation to just <= OpenGL 4.1 for direct Mac support, but if I need a greater version to implement a critical features, we'll have to consider other options for supporting Mac (I understand there is an OpenGL-in-terms-of-Metal layer...)

Anticipated Features (not planned per se, and the engine will not necessarily have the data structuring and scaffolding to easily slide them in, but they are almost certainly necessary for someone) -

• [ ] Bloom - for far away lights whose emissiveness + bloom stands-in for real light sources (https://learnopengl.com/Advanced-Lighting/Bloom)
• [ ] Subsurface Scattering (Burley model or https://developer.nvidia.com/gpugems/gpugems/part-iii-materials/chapter-16-real-time-approximations-subsurface-scattering)
• [ ] Large Water Body Shader (using projective reflection)
• [ ] Depth of Field Post-Process
• [ ] Some sort of generalized rendering occlusion solution such as https://medium.com/@Umbra3D/introduction-to-occlusion-culling-3d6cfb195c79 or #518
• [ ] Probe Volume as an alternative to Baked Light Maps - https://youtu.be/DlxuvvYZO4Q?t=1070
• [ ] Voxelized GL Probes as another alternative - https://docs.godotengine.org/en/3.5/tutorials/3d/gi_probes.html (DotRecast already voxelizes our scene for nav purposes - could that be leveraged here?)
• [ ] #694

[bryanedds]

A vital item is the ability to import from Unity and / or Unreal. Implementing a USD import process for Nu is one possibility -

https://docs.unrealengine.com/4.26/en-US/WorkingWithContent/USDinUE4/

https://github.com/unity3d-jp/USDForUnity

Related to assimp and USD -

https://github.com/assimp/assimp/issues/1332

[bryanedds]

Looks like there's some .NET USD bindings here -

https://github.com/unity3d-jp/USDForUnity/blob/master/USDForUnity/Assets/UTJ/USDForUnity/Scripts/usdi.cs

Along with the example code for dealing with Unity, this could be a great start!

[bryanedds]

Newer repo here -

https://github.com/Unity-Technologies/usd-unity-sdk

[bryanedds]

Mesh Distance Fields can buy several advanced rendering features - https://docs.unrealengine.com/5.0/en-US/mesh-distance-fields-in-unreal-engine/

Their expense might be worth it since I've heard they could be used in place of cascaded shadow maps. // TODO: verify this.

[bryanedds]

Compile all shaders in a package when hint-loaded?

[bryanedds]

How will we do mesh picking in the editor? Triangle picking? EDIT: Yes, triangle picking from the metadata pipeline.

[bryanedds]

Unfortunately, it appears that mixing SDL rendering with OGL rendering is unsupported. This means I will probably have to start by rewriting the 2d renderer in OGL.

First, I need to design the 2d renderer. I am thinking that I will have a finite number of material parameters for any 2d renderable. That way sprite sorting / batching can be a static process. Unlike 3d rendering, 2d render messages are not composed of materials, but will stay pretty much the same API. It will be implemented with painter's algorithm rather than using z testing since, due to transparencies, we have to keep the sorting algorithm anyway. Like the 3d renderer - or perhaps even moreso - there will only be one rendering model for 2d. While I may add features to the one 2d material over time, I won't provide a way for users to extend it with additional parameters. Nor am I immediately going to provide any pre or post-processing 2d API. Again, that could be added later without restructuring the API.

This leaves a lot of power and features on the table for 2d rendering, but the fact that I have to whip this out in a week or two in order to meet my deadlines helps disambiguate that decision.

Fortunately, it does seem the sdl_ttf will work with opengl, so I won't have to figure out another way to render text - https://stackoverflow.com/questions/5289447/using-sdl-ttf-with-opengl.

[bryanedds]

Even with such a simple 2d rendering model, there's still a couple interesting things I can do basically for free (in terms of implementation cost). 2d normal maps, for example, as well as specular maps (tho that would require a light source render message). Pre and post passes actually would be basically free in terms of implementation. An Elevation buffer could also allow for some interesting effects.

Also, I need to design how the 2d and 3d images come together if 2d is going to have pre and post passes. I would need a destination alpha channel on the target 2d buffer and then render the target 2d buffer on top of the 3d buffer.

With an elevation buffer, some forms of 2d shadowing would be possible, but I dontt think general enough to implement.

[bryanedds]

While I can't necessarily allow the user to extend the 2d material interface, I could allow them to replace it and the rendering process itself with a 2d equivalent for https://github.com/bryanedds/Nu/blob/c3f908cba821f3407c574757cc78f9482df2c957/Nu/Nu/Render3d.fs#L48

[bryanedds]

Approximate render pass order -

shadow map -> geometry -> decals -> lighting -> screen space reflection -> tone mapping -> fxaa

[bryanedds]

For several reasons, mostly due to lack of tooling support, USD files aren't likely going to be our target scene format, but rather FBX. Unity FBX does a pretty good job. Just take the objects you want to export, put them under a single GameObject, then use GameObject -> Export to FBX... (presuming you have Unity's FBX Exporter plug-in installed).

[bryanedds]

FBX export from Unity is great and all, but the immediate question is how to attach user-defined data to an object and retrieve it in Nu. I'm guessing we can build an little custom exporter in term of Unity's FBX exporter API adds a UserProperty string field of a custom UserProperty component like so -

            // Import the scene to make sure file is valid
            using (FbxExporter exporter = FbxExporter.Create(fbxManager, "myExporter"))
            using (FbxImporter importer = FbxImporter.Create(fbxManager, "myImporter"))
            {
                // export the scene to a temp file...
                exporter.Initialize("TempFile.fbx", -1, fbxManager.GetIOSettings());
                var sceneTemp = FbxScene.Create(fbxManager, "SceneTemp");
                exporter.Export(sceneTemp);

                // re-import the scene to an FBX object we can manipulate...
                importer.Initialize("TempFile.fbx", -1, fbxManager.GetIOSettings());
                var sceneFinal = FbxScene.Create(fbxManager, "SceneFinal");
                importer.Import(sceneFinal);

                // search through scene, correlating nodes with game object that have a UserProperty component,
                // then add user property like so -
                var node = null as FbxNode;
                var userProperty = FbxProperty.Create(node, Globals.FbxStringDT, "UserProperty");
                userProperty.Set(userPropertyComponent.UserProperty);

                // then finally, export the scene to a non-temp file.
                exporter.Export(sceneFinal);
            }

...then we should be able to load it from Assimp via -

node.Metadata.["UserProperty"].Data :?> string

[bryanedds]

A few asset packs for testing the renderer - https://assetstore.unity.com/packages/3d/environments/urban/atmospheric-house-modular-192712 https://assetstore.unity.com/packages/3d/environments/urban/hq-abandoned-school-modular-106665 https://assetstore.unity.com/packages/3d/environments/urban/qa-police-station-162515

[bryanedds]

For the metadata pipeline, I might allow users to specify a .meta file for each asset like Unity, but unlike Unity, .meta files are completely optional. The user-defined .meta files are then fed into the metadata pipeline process, resulting in either a single monolithic metadata output file in the bin folder or one .meta file per asset in the bin folder. The important thing is to make .meta files not being a huge PITA like in Unity.

[bryanedds]

I have so far only found one seemingly usable Physx wrapper, but it seems questionable as to how cross-platform it will be -

https://github.com/stilldesign/PhysX.Net/blob/master/PhysX.Net/PhysX.Net/Source/StdAfx.h#L25-L26

[bryanedds]

Bullet physics used instead of Physx.

[bryanedds]

We should consider shadow caching as an optimization feature.

[bryanedds]

We could also consider giving lights contact shadows if ESM makes things look too floaty.

[bryanedds]

We probably need to prioritize the following features, only one of which are on the current roadmap -

• ssr (done)
• volumetric lighting (done)
• per-light probe ambient properties (done)
• light map subsumption (done)
• dynamic / adaptive tone mapping

[bryanedds]

There are .NET bindings for an OpenGL-based fork of FSR2 -

https://github.com/BoyBaykiller/FidelityFX-FSR2-CSharpBindings/

(OpenGL-based fork of FSR2 here - tho there seem to be some unresolved performance problems...)

https://github.com/JuanDiegoMontoya/FidelityFX-FSR2-OpenGL/

Tho ideally FSR3 would be better to use than FSR2. However, it may not be worth taking a dependency on either one and so we might instead implement a TAA ourselves.