Name a first-class scene format (glTF or USD) and adopt physically based rendering

Discussed at the end of dev club today. We started with talking about updating Assimp.NET (or replacing it with our own) for our scene loading infrastructure. I suggested it might be better to name one preferred format that Bonsai declared first-class support for (as opposed to partial support for several older formats as we do today.)

Within the games industry there is wide support for glTF and growing support for USD. I personally have a lot of experience with glTF and have been very happy with it, I've been looking for an excuse to learn more about USD.

I'm creating this issue both to ramble out my thoughts and to track this potential effort.

Physically based rendering (PBR)

Defining physically based rendering (PBR) up front (this might seem unrelated but it falls out naturally from supporting either of these):

Essentially light in a PBR model behaves much more like it does in real life compared to Phong shading, energy conservation is properly modeled and as a result it's easier to create natural-looking materials
In contrast to classic lighting techniques like Phong shading, models light much more how it behaves in real life.
- Energy conservation is properly modeled (you can't have a material that reflects more light than it receives), this inherently makes it much easier to create natural-looking materials
- It's possible to model lights with real-world units like lumens etc.
Great free material libraries like Poly Haven exist
PBR is inherently pretty modular, for example adding a clear coat light transfer function (think car paint) or anistropic materials (think brushed metal) is just a matter of changing how a few parameters are calculated
- This means we only have to support a subset we care about with future room to grow
- (That giant Filament article is a great resource on PBR.)
Blender's material system is focused on PBR
Unity's high definition render pipeline is focused on PBR
The glTF sample viewer is a good tool for getting an idea of what PBR capabilities look like in a real-time context

A downside of PBR is that it's more computationally heavy to render. It's not a problem for desktop-class GPUs but can be for mobile-class GPUs. I think if this becomes a problem we can just offer a simplified rendering toggle somewhere.

glTF

Created by Khronos Group (IE: the maintainers of OpenGL/Vulkan/etc)
- v2.0 was released in 2017 and is widely supported by a wide variety of tools and rendering
- (v1.0 was released in 2015 as a spiritual successtor to COLLADA but was not adopted very widely)
It's tailored towards the needs of real-time rendering, specifically designed with OpenGL in mind
- It's a last-mile format, not ideal for interchange purposes (like moving between Blender and Houdini or something similar)
- In my expereince it's more widely used for single objects, but it does support full scenes of objects
  - All objects in a scene are effectively embedded within the glTF file, which is either a plus or a minus depending on your needs (my feeling is that for Bonsai this is mostly a plus? Our users aren't doing AAA-quality levels, duplication isn't the end of the world.)
  - Multiple scenes can be defined in a given file
In addition to just meshes, glTF also defines materials, lights, a scene graph, animation (both skeletal and rigid nodes), and cameras
- The glTF poster provides a nice overview of the format
Open specification
Has a well-documented extension model
- This maps well to the previously mentioned modularity of PBR
- The previously mentioned "extended" PBR features are provided by the KHR_materials_clearcoat and KHR_materials_anisotropy
- Extensions can be optional or required, files define what they need vs what they want.
- Bonsai can easily tell people when their scene/mesh can't be rendered or won't look quite right.
Textures can optionally be embedded in glTF files (so a single file is all you need for the whole scene)
glTF is well-supported by Blender (added to the Blender core in 2.8)
- I can't find this statement now, but I believe the Blender Foundation has declared glTF as their recommended export format before
- A wide variety of glTF extensions are supported
- Note that due to its extreme flexibility, not all Blender materials can be exported to glTF (I think as long as you stay away from nodes or use simple node graphs it's fine though.)
- Adding support for other extensions (including our own custom ones) can be done through Blender add-ons and is pretty simple from what I remember
The previously mentioned sample viewer is a great way to get an idea of the capabilities of the material definition and common extensions
- The assets are available on GitHub if you'd like to poke at them directly. (.gltf files are just JSON. .glb files are binary as they contain embedded mesh data and textures too.)
- The sample viewer is open source and permissively licensed, so we could potentially just use their shader implementation for Bonsai since it's GLSL
- As mentioned during dev club, I also have my own open source implementation of the core glTF PBR spec for ThreeL, but it's HLSL rather than GLSL
  - You can also use ThreeL to get an idea of the capabilities of the baseline PBR spec supportted by glTF
  - (Note that the smoke is technically not PBR, I fudged the math of light transmitting through it.)

USD

Created by Pixar and published as open source in 2016
- Now controlled by the Alliance for OpenUSD
Open specification
Not widely used in games and other real-time contexts, but there's growing interest (particularly in the AAA space)
Widely used in film contexts, also used in some industrial CAD contexts like automotive
Robin-Yann Storm has a nice 10 minute overview over USD
Treated more as an interchange format (ideal for moving work between Blender and Houdini)
- For Bonsai's needs that kinda means USD is expected to be ingested into a different format internally for actually rendering (this is why AAA gamedevs like it since their last-mile format tends to be engine-specific.)
Compared to glTF, it's more suited to complex scenes (scenes can be made up of multiple USD files that reference eachother, they can be versioned and authored independently.)
Blender started supporting it in 4.0
- Documentation
- Similar to glTF, not all materials can be exported to USD. (Again, I think it's fine as long as you stay away from nodes.)
My understanding is USD is less opinionated about material representation, but UsdPreviewSurface (which is what Blender exports) is PBR-friendly.

What Bonsai supports today

Right now Bonsai only supports Wavefront Objects (.obj) for meshes.

To contrast with the above, it supports the following for scenes:

COLLADA files (.dae)
- Spiritual father of glTF
- Very old (last updated in 2008)
- Blender documentation describes support as being a work in progress
- Materials are defined as Phong, legacy fixed-function pipelines, or as GLSL shaders (both dated and overly flexible)
- Mesh data is stored in plain text, files get huge quickly
Blender files (.blend)
- The internal format of Blender
- Format is not documented and evolves rapidly
- New versions of Blender being broken with Bonsai are why this subject came up in the first place
- If we continue to support importing Blend files directly (which I think is reasonable) we should have Bonsai transparently invoke Blender to have it export whatever format we want to consume.
  - This is basically how Unity works with .blend files (it exports them to FBX.)
  - Essentially you just make a small Python script that uses the Blender API to load and export the file.
    - Blender has a mode where it loads a script and executes it without showing the UI, it's not complicated at all.
FBX files
- Autodesk's closed, proprietary format
- Open implementations like the one used in Blender are made with reverse engineering
- Only official implementation is the Autodesk FBX SDK
- Widely regarded as bloated and messy
Wavefront .obj
- Extremely dated format from the mid-late 80's (couldn't find an exact date)
- Cannot represent scene graphs, it's just one big mesh
- Materials are stored separate from the mesh
- Mesh data is stored in plain text, files get huge quickly
- The format wants to be a last-mile format but doesn't align with how rendering is actually done
  - Format supports n-gons rather than just triangles
  - Vertex data does not need to be indexed uniformly (IE: normals can be in a different order than positions)
- Trivial to parse, the spec fits on Wikipedia
- Materials rooted in dated lighting models, although a PBR extension exists (and is supported by Blender.)
- Does not support scenes or animations
STL files
- Extremely dated format from 1987
- Designed with the needs of 3D printers and other computer-aided manufacturing systems in mind
- Does not support scenes, animations, or materials

Conclusions?

My gut says that glTF is probably the best fit for Bonsai's needs. USD is more powerful, but a lot of what makes it attractive in the AAA games space doesn't feel relevant to Bonsai users. (That being said, I'd greatly welcome the opportunity to mess with USD 😁)

A big benefit of glTF is that the spec is fairly simple, you can implement a parser for it in a weekend. (As I mentioned in DCM I've created 2 parsers and 3 different renderers with varying levels of conformance for it in the past.)

In my opinion COLLADA/FBX/OBJ/STL should all just be deprecated and we should put our energy into a single format instead. (I don't think we need to remove them, but if we ever add more complex scene manipulation stuff like moving objects around we shouldn't make any effort to support those features with them.)

Blender support should be migrated towards transparent export to either glTF or USD.

PBR isn't truly necessary for Bonsai, but I think the way you can get nice results from PBR with relatively little effort is pretty compelling. It's a natural fit for glTF in particular and as mentioned earlier Poly Haven provides a lot of great public domain PBR assets people could use for their scenes. (I focused on the materials above but they offer models too.) I think it'd be nice for people to get similar results out of Blender and Bonsai as well.

bonsai-rx / bonsai