This issue tracks design consideration for the v4 simplification. Instead of keeping separate design documents for each part of the library, we keep everything into one for simplicity.

Color Images

There have been two problems we tackled in v3 and before. The main problem is that different image formats makes sense for different applications, but we do not want to support that as discussed in other issues. Instead we want to cut down code size while still allowing to support everything we do now. Here are the things a few things that we care about:

keep float and byte images
perform all computation in float, exception for image sampling which can be done in bytes
we seldomly save data in byte for speed
we compute in linear and non-linear spaces depending on the application
makes sense to keep color computation in vec3f, but store always in vec4f or vec4b

So this means we have 4 variants

float / linear (HDR)
float / non-linear (computed values)
byte / linear (normal maps)
byte / nonlinear (albedo maps)

A major issue seems to be that we started the project thinking byte=non-linear and float=linear and that was a big mistake. That lead to weird things everywhere in the pipeline, typeing to perform conversions in places where none was desired. We have to try to fix it now. A major issues that needs to be addressed is loading should never change color space, and saving should always respect it.

Given all these constraints, and the discussion below, the proposed implementation is

[x] implement a shared type for float and bytes, for now with only 4 channels
[x] stored a flag for llinear/non-linear (in leu of proper color spaces)
[x] access pixels only with set_pixel()/get_pixel()
[x] add functions width and height
[x] make images by default as non-linear floats
[x] load and save images by respecting the color space
[x] sample images within their color space, but with a flag to force linear
[x] resize images in their natural representation
[x] tone map from linear to non-linear, default as float, but with a construction flag for bytes
[x] color grade from linear to non-linear, default to float, but with a construction flag for bytes
[x] possibly wrap this into a class wo we get proper behavior and add color spaces later
[x] implement set/get region

Follows a discussion on design choices

Here are some operations we perform and how that relates to image formats.

tone mapping: HDR to LDR, but more importantly linear to non-linear clamped
color grading: HDR to LDR or LDR to LDR, i.e. linear/non-linear to non-linear clamped
loading: either float-only or float/byte based on filename
- should loading ignore color space or marked images with it?
texturing: all the above, but generally all converted to linear depending on use
resizing: same color space and flaot/byte type
diffing: same color space and float/byte type
viewing: all cases, although we could skip the float/byte conversion in the view API

The question now to decide how to handle all these cases in code.

One option is to have a templated type as we have now
- library is easy to but apps are painful to write
- many duplications on types
The other option is to have a single type with both byte/float and a linear flag
- loading: always as is and keep color space; can force to float
- saving: float converted appropriately based on color space and file formats, bytes as they are
- resizing: as is
- tone map: linear to non-linear; input only float; output byte or float, but default to float
- make proc: can choose to output floats, but mark linear/non-linear
- colorgrade: input as float, output as float; linear/non-linear to non-linear
- eval: float/byte input, float output
- bytes images are used little, but maybe helpful at times: textures (so eval), eval, output for tonemap/grade only for speed

Generic Shapes

In v3, Yocto did not have a concept of shape explicit within the library. Only low-level functions were used throughout. That makes it possible to build custom apps very easily. But this is at the price of usability and simplicity. In this version, we are planning to introduce two common data types for shapes that are either indexed or face-varying. The discussion here will mostly focus on the indexed shapes, since the face-varying ones are nearly identical.

The main features we want to support are:

generic shape data that is, for now, not extensible to arbitrary data
methods for building shapes with attributes
query methods for shape types and properties (no more !triangles.empty() idiom)
vertex data getter and setter, to be used optionally
vertex data interpolation for the built it properties (these were in the path tracer before)
loading and saving shapes
high level helpers: smoothing normals and targets, subdivision, etc

It seems that to do these, there is very little design that still needs doing, besides naming conventions that are now not properly designed now. So we can just make a list of things to implement.

[x] generic shape data
[x] initialization functions for shapes
[x] query functions for shape types and vertex property
[x] vertex and face setter/getter
[x] vertex interpolating functions with eval_XXX methods
[x] loading and saving shapes, possibly with forced convertions
[x] shape convertions placeholders (quads/triangles, but maybe also lines/points to surfaces)
[x] normals smoothing and faceting
[x] subdivision

For subdivision, things are very similar, so we just add things here.

[x] generic face-varying shape data
[x] initialization functions for face-varying shapes
[x] query functions for face-varying shape types and vertex property
[x] vertex and face setter/getter
[x] vertex interpolating functions with eval_XXX methods
[x] loading and saving face-varying shapes
[x] shape convertions placeholders (face-varying to indexed and vice-versa)
[x] normals smoothing and faceting
[x] subdivision

Scene creation

We have gone back and forth on scene creation, mostly to handle the complexity of either too many parameters, the pointer usage and so on. Right now, we are settling on a simple design that is natural to a data-oriented C++ stack. This design preserves the following goals:

scene creation should be natural to avoid having to write tons of getters and setters
data is accessed directly, so we should do so when we can
in one alternative, scene creation is just manipulation of the various arrays in the scene
add_XXX functions helpful to build a scene in a programmatic apron since they are potentially shorter to use and allow for presets; they also hide the complexity of the materials, at least for now
names are optional and do not have to be set; add methods will take them always for simplicity
add methods should for now take only simple params, even if they eventually will take the full description
add methods should be the exact same in the glscene

Given this design, the translation to todos is easy

[x] remove all add_XXX function from scene that do nothing
[x] add add_XXX function that create full objects
[x] possibly add make_XXX functions that create full objects, useful for cameras, materials, environments and instances
[x] add_XXX functions should add names when necessary
[x] handle name creation in IO and presets
[x] eventually move all presets to the loading of images, shapes and materials; but for presets we should probably decid about how to expose these to the outside
[x] update the glscene to have the same methods, although for now it may be required to have some clutches for untyped materials and shapes

Names in IO

Names in scenes are quite helpful but at the sam time not strictly needed. It seems that the most helplfull way of doing this would be to have a scene_metadata structure where we can keep all this data. We should investigate this further or decide fatly what to do.

[x] investigate the use of scene_metadata as either internal to the scene, derived from it or packaged into it.

Library organization

After a few brainstorming discussion is was decided to refactor the library "by type". We list here the main action to be taken and also what remains to be decided.

The overall theme is to split the library by the type of object thy handle to make browsing code easier and establish a better foundation.

We tried initially to have all functionality in its own libraries. But it seems it would overcomplicate a few things. Instead we could handle this initially by moving the main structures into scene. Later on we can split this but we should remove the array-like functionality.

[x] move shape into scene
[x] have texture duplicate an image
[x] shape io is duplicated between yocto_shape and yocto_sceneio?
[x] yocto_commonio -> yocto_cli
[x] remove embeddd bunny since it screws up compilers
[x] remove all tests
[x] remove scene add_XXX methods
[x] blackbody simplification
[x] shape type
[x] simplify commonio away
[x] consider moving image into scene
[x] remove mesh io
[x] remove mesh

Simpler Material

The material model of v3 and before is derived from the Disney model. That has worked well so far, but has two main drawbacks: complexity in creation and complexity in rendering. In the years this has been hard to use for students, which should not have happened. We are now proposing to following pbrt and use a tagged material model. We want the following features:

materials should have intuitive types types
use types instead of flags to change major beavior
if possible, match pbrt materials in the definitions
convert materials to/from other formats: maintain compatibility with old format and focus on glTF for forward compatibility
reduce the use of textures significantly and if possible to two textures
consider whether to set colors as srgb or rub linear --- it is possible than srgb is simpler to handle overall
in the renderer, use switches to handle brdf parameters

Given these goals, the first issue to solve is to deign the material types and parameters. We would base it on the current versions and then normalize it after.

From this discussion, we now need the following parameters:

matte: color, opacity, color_tex
plastic: color, opacity, color_tex, for, roughness, specular_tex
metal: color, opacity, color_tex, roughness, specular_tex
metallic: color, opacity, color_tex, metallic, roughness, specular_tex
thin glass: color, opacity ior, roughness, specular_tex
thick glass: color, opacity, ior, roughness, specular_tex, scattering, phaseg
subsurface: color, opacity, ior, roughness, specular_tex, scattering, phaseg
volume: only volumetric param

Additional property might include:

light: emission, emission_tex
normal map, normal_tex

Given these goals, the following changes are needed:

[x] review pbrt materials
[x] define new material parametrization
- [x] start by adopting metallic as default
- [x] introduce all material types
- [x] remove old parameters, but keep 3.0 compatibility
- [x] finalize old parameters
- [x] decide about emission
- [x] decide about scattering
[x] add eval/sample/pdf methods for all new materials
[x] remore params from OBJ
[x] change renderer to hande these
[x] change add_XXX_material functions to match
[x] change glrender to handle them, probably with no materials types for now, but doing a simpler translation: use metallic as the main type

Material naming is another complex issues. All renderers have different kind of nomenclature, so there is no common ground to use. There are three main variations for this

names of common materials: matte, plastic, glass
- used in pbrt-v3
- looks fine to me but gets confusing when we have glass for water or leaves from diffuse translucency
names from physics: dielectric, conductor, coateddiffuse
- usd in tungsten, pbrt-v4
- to me it is not good
- it implies that we support correctly physics behaviors
- it implies that you need to know physics to understand this, namely it scares people away
names from phenomenological appearance: diffuse, reflective (metal), transmissive, refractive, translucent
- used by nobody, but the one I prefer

Another issue is parametrization of subsurface scattering and emission. What do we do here is less clear than in all other. I will open an issue for discussion with others elsewhere and than update this design document with the results.

Rationalization of IO

Right now Yocto is mostly independent since many of its IO needs are served within Yocto libraries. This has served us very well in the past. But it is beginning to take a toll in our ability to move forward. All these IO needs require significant effort and a lot of maintenance, that it is unlikely to scale well in the future.

In the end, our goals are

make it easy to write Yocto apps, command line and gui
keep the IO the same through all yocto libraries
reduce maintenance costs

These goals are well served with this refactoring

[x] command line
- [x] add_argument instead of add_optional/add_positional, arguments instead of options
- [x] support vecXX
- [x] simplify calls
- [x] remove command/cli_state difference
- [x] consider using CLI11 internally
- [x] consider using macros to autogenerate cli commands
- [x] consider using labels for enums
[x] remove Yocto/JSON
[x] gui
- [x] add a generic parameter structure
- [x] consider macros to go backs and forth
- [x] otherwise consider serialization methods
[x] modelio
- [x] consider wrapper for STL
  - since what we have works, we will keep this as is
- [x] evaluate more carefully OBJ
  - still unclear what to do
  - the current implementation is slow-is but fine
  - the real issue is that it is really complex and it is not clear how to make it better
  - fast_obj is a great alternative, but it only allows for reading files
  - [x] investigate possible simplifications
  - [x] investigate fast_obj writing
  - [x] provide get_shape() function
  - [x] remove instances
- [x] evaluate more carefully PLY
  - decision is to keep our implementation for now since others have some issues
  - [x] add face reading callback
- [x] evaluate more carefully PBRT
  - we will keep this for now
  - [x] evaluate possibile simplifications
  - [x] evaluate explicit objects
  - [x] evaluate pbrt-v4 scenes
[x] sceneio
- [x] evaluate if doing all glTF internally
- [x] use cgltf for glTF output
  - tried to do and it was too hard to be useful
- [x] parallel loading
[x] progress
- [x] sceneio
- [x] trace

OpenGL Objects

This is a placeholder for discussions about OpenGL in the library.

Large tests

With all changes to Yocto/GL so far, we should probably start to rerun all tests. This will require selecting tests, updating the tests scripts and providing rendering configurations for all.

To run tests we first need to update some infrastructure of the various scripts we are using.

[x] flatten Disney scene for simplicity
[x] flip hardness of pbrt scenes
[x] remove logo
- [x] remove logo
- [x] update scene script
[x] params from files
- consider if doing this in the script for now
[x] rendering from different cameras
[x] when moving camera, params are lost in trace
[x] decode on texture v orientation
- [x] pbrt
- [x] obj
- [x] disney
- [x] default
[x] review all scripts
[x] consider scenes links in the main repo
[x] scene adjustments
- [x] pbrt3
  - [x] crown materials
  - [x] landscape
  - [x] simpleanim
  - [x] squared cameras
  - [x] transparent
  - [x] only save used textures and skip all others
- [x] mcguire
  - [x] bistroexterior: lamp, glass, darker textures
  - [x] bistrointerior: green, metal, glass
  - [x] rungholt, voxelia: sea material darker environment
  - [x] San Miguel: glass material, plant material, darker env?
  - [x] remove all opacity maps from scenes

Apps

The current apps structure works better than in the past. We should converge on making this better still. Some of it is lost functionality or bug fixes w.r.t. the old interfaces. Others are strengthening to integrate more app examples.

Let us list first the known bug fixes.

[x] ytrace crashes when no lights are present
[x] ytrace should support switching cameras in the ui
[x] yscene should support switching cameras

Path tracing

This section lists improvements to the path tracing API. Overall, the current feature set works well for yocto. Improvements may be needed in the API, more uniform code, and small additions to support denoising.

Here is a list of changes to evaluate

[x] use image_data instead of image
- [x] consider passing in an image buffer instead of returning it via callbacks
[x] support denoising by adding normal and albedo buffers
[x] support opacity in envmaps
[x] simpler sample_camera
- [x] maybe move the tent filter somewhere else
[x] integrate denoising into render
[x] falsecolor show material type
[x] unique_ptr for embree
[x] material use of fresnel_schlick?

Material Naming

This issue tracks a discussion of materials naming and parameters.

There are four main concerns:

material names
what to do with emission
what to do with eta/ior
what to do with volumetric properties

Material Types

The current implementation supports the following materials:

matte: Lambert brdf with color as diffuse
metal: specular brdf with color as reflectivity at grazing angles (conductor fresnel for now) and roughness
plastic: labert + specular with color as diffuse, ior and roughness for specular
thinglass: thin transmission and reflection with color for transmission color and ior and roughness for specular/fresnel
glass: same as before but with refraction and homogenous volume; volume uses color and depth for density and scattering and anisotropy for albedo
metallic: glTF material with diffuse + specular, uses color, metallic and roughness
subsurface: same as glass above with scattering texture and to allow a different way of entering the surface; it is ignored for now but I think it is better to keep a subsurface material for skin rendering
volume: a purely volumetric material with bo surface interface, for now uses only the volumetric properties

Material naming

Material naming is another complex issues. All renderers have different kind of nomenclature, so there is no common ground to use. There are three main variations for this

names of common materials: matte, plastic, glass
- used in pbrt-v3
- looks fine to me but gets confusing when we have glass for water or leaves from diffuse translucency
names from physics: dielectric, conductor, coateddiffuse
- usd in tungsten, pbrt-v4
- to me it is not good
- it implies that we support correctly physics behaviors
- it implies that you need to know physics to understand this, namely it scares people away
names from phenomenological appearance: diffuse, reflective (metal), transmissive, refractive, translucent
- used by nobody, but the one I prefer

I prefer the last naming convention.

Material emission

Right now emission is kept as a separate property, just like in all renderers. This may or may not be a good idea.

Pros: Simpler IO
Pros: Other do it this way
Pros: color is a color in [0,1], while emission is in [0,inf]; we could fix this by adding an exposure param
Cons: can be confusing for new people looking for "lights"
Cons: need more setup since we should not use a black BRDF by default, maybe
Cons: 1 more param, 1 more texture
Cons: cannot simulate funky behaviors, like coated emission (glow stick)
Cons: less obvious where lights are (very minor)

Here I am not sure what to do.

Material parameters

The following parameters are mostly reasonable, besides possibly coming up with better names.

color: this is the main object reflectance, but it really is different in all materials; generally this is the one that gets the texture
- called sometimes reflectance in pbrt-v4
- I would probably leave it like this
- color_tex for texture, but could be color_map
opacity: clear what it is, I would leave it like this
- alpha from color_tex
roughness: clear what it is, I would leave it like this
- roughness_tex for texture, but could be roughness_map
metallic: used only for glTF, I would leave it as is since it matches those specs
- texture in roughness_tex
emission: see above
- emission_tex as texture
normal_tex for normal maps; works like it ism but maybe call it normal_map

The more controversial parameters are ior and all volumetric settings.

For metals, we can use either Schlick's or conductors' Fresnel terms. I refer to introduce all the complexity of complex for frankly since we have no data and since it cannot be textured. So a simple reflectivity setting, as a color, works well enough.

For dielectrics it is harder. Right now we use ior for two things: coating/plastic and transmission/refraction. For coating, a reflectivity settings works fine and in fact is required in glTF. Before we used to have specular for that, but it was used in awkward ways, so it was changed to single ior. For transmission/refraction, ior is used in dielectric fresnel and refraction direction, but has two issues: (1) we do not do spectral rendering, so we need single value ior, and (2) nobody changes it since who knowns what it does.

WE can leave this as single ior, basically a read-only property. for now, I use dielectric Fresnel, but it is impossible to teach and explain (in fact all renders have the same cut&pasted code). I would not have a 3 channel ior since it is completely strange (see pbrt-v4 scenes). Th other option is to use specular set to {0.04, 0.04, 0.04} and be done with it (gt for when needed). In the future, that can be used for other effects, like colored coatings.

Not sure what to do here.

Vometric properties

The other hard issue are volumetric properties. These are nasty since they are not intuitive no matter what one does. For this we have three materials now:

glass: meant for refractive glass
subsurface: meant for skin
volume: meant for smoke and clouds, i.e. no interface

In the end, we need to set absorption, albedo and anisotropic (phase g), but it is helpful to consider the there cases separately.

glass: typically absorption only, so make sense that color control this, maybe
subsurface: need two set absorption but main color controls scattering, for example by using the Sony Parametrization to go from diffuse color to scattering
volume: need to control both absorption and scattering: main color from scattering, I am guessing (colored smoke?)

This already would suggest different parametrizations for at least glass and subsurface. Or maybe it make sense to keep the same one, which is what is done in production (glass: color -> density, subsurface: mfp -> density).

BTW, the renderer should also be changed based on these materials a bit. For example if no scattering is there, a simpler early exit is useful, but is for another day.

Here I am really not sure what do. Noe I know what to do with volumetric textures since I have not tried any data yet.

The bottom line is that we could follow two way:

current paraterization:
- color -> density
- scattering -> albedo, probably named subsurface
different parametrization for glass and subsurface where color sets different properties
- glass though would remain as is now

I am not sure what to do.

Feedback on Material Naming

Naming materials based on the appearance is probably better idea for our goals.

matte: matte is ok. It communicates quite well the final appearance.
metal: metal is also ok, even though it does not describe the phenomenological appearance but rather the material. However metals have such a unique look that it is fine for this case.
plastic: This looks like the most common case and can model a wide range of real materials, far beyond plastic. For these reasons this is the most important and hardest material to name. I propose glossy, which should communicate a colored surface with reflections. It is quite generic, but it must be so.
glass: The word that people would use to describe both glass and water is transparent. I would exclude transmissive/refractive, they are still technical terms that describe the effect but not the appearance.
thinglass: For the same reasons, I propose transparent_thin.
subsurface: subsurface is ok, maybe translucent is better. My only worry is that someone can confuse it with the word with transparent.
volume: This should model gases, aerosols and particles like smoke, dust, fog, clouds, but none of these words is good. I guess volume is ok.
metallic: This is too similar to metal, having high potential to confuse people. Since we have it only to support Gltf, I would go for gltf_metallic which is intentionally ugly to warn the user and be clear about its usage.

In the code there's also an unimplemented "leaf". I'm not sure about what it does, but a material with a reflective lobe and a in-place single-scattering lobe can be named translucent_thin.

To sum up: matte, glossy, metal, transparent, translucent, transparent_thin, translucent_thin(?), volume, gltf_metallic.

We can keep emission as it is now, but we probably need to add emissive to the list of materials type.

Material parameters and their naming are ok.

Prepare for Release

This is a list of final things we need to do to get the release ready.

[x] documentation
- [x] bvh
- [x] path manipulation
[x] names in io
[x] fix all tests
- [x] decide whether to automatically generate tests
[x] resolve the scene names issues
[x] resolve material parametrization for volumes
[x] resolve skin rendering
[x] path tracing api cleanup
- [x] pass in image instead of callback
- [x] trace_samples
- [x] consider moving code to viewers and avoid async in the backend
- [x] render name
[x] scenes
- [x] convert all scenes
- [x] eyelight all scenes
- [x] render all scenes
- [x] pbrt
  - [x] glass color
  - [x] crown
  - [x] square camera
  - [x] skin rendering
[x] consider scenes links in the main repo
[x] naming issues
- [x] render
- [x] scene
- [x] materials
[x] finish mesh cleanup
[x] decide about logging

Here are the things that can be postponed.

[x] path tracing improvements listed above
[x] path tracing API cleanup
[x] bvh API cleanup
- [x] embree should be compact or high-quality only
- [x] check high quality bvh speed
[x] log instead of progress callback

Postponed

[x] support denoising by adding normal and albedo buffers
[x] support opacity in envmaps
[x] simpler sample_camera
- [x] maybe move the tent filter somewhere else
[x] integrate denoising into render
[x] material use of fresnel_schlick?
[x] glrenderer support new materials
[x] cli support vecX

The release is ready.

xelatihy / yocto-gl

Version 4 - Design Documents #1151