Switch to "OpenGL" normalmap direction?

This is a thread to decide whether we should keep the default interpretation of normalmaps the way it is now (so-called DirectX style), or switch to OpenGL style, which interprets the y-axis (green channel) as pointing the opposite direction. Note that the direction chosen doesn't really have any relation to the rendering library you use - it's up to the GPU shader code to decide how to interpret things.

Advantages of switching to OpenGL direction:

More intuitive: y-component points up, as expected, and the coordinate system is right-handed
Matches most OpenGL normal mapping tutorials
Improves compatibility with Xonotic assets

Advantages of staying on DirectX:

Our assets continue to work unchanged - we don't have to add q3shaders to reverse the direction of every existing normalmap
Maybe it's more common among games in general? (unclear)

About the engine, the internal representation in glsl shaders can be switched to the OpenGL representation or any format we want from now. The issue is about the default format expected from a material described in a file stored as scripts/<somename>.shader and described using the syntax we use.

Maps from games using another way to describe materials, another material file naming scheme or another material file syntax would allow us to import their normal map the expected way whatever the internal format we choose and the format the use.

This distinction is very important, because it means we can have an engine having an OpenGL-like internal normal representation, even if we expect our scripts/<somename>.shader written with our syntax to describe DirectX-like normal maps.

For other material .shader-like formats, it does not prevent us to load normal maps described in hypothetical materials/<somename>.mtr the DirectX way and normal maps described in hypothetical <somepath>/<somename>.gltf file the OpenGL way.

DarkPlaces does not use .shader files to describe normal maps. This engine uses suffixes. So we are free to use any format we want internally, we would be able to read DarkPlaces's files the expected way. The big issue would come if one day we want to add support for assets from another engine which uses the same technique an relies on the same suffix but expects another format.

The biggest issue I see is that other idTech 3 engines may use DirectX normal map format by default and they may use the same syntax we use… If that's true, it would be sad but we may have to accept they chose for us.

I'm doing some investigations.

Also, we have to consider the normal format conversion to be free. The GLSL code computes a normal scale even if there is no scaling applied (scaling factor of 1) because in GPU world, an if just discards the result of an operation but does it anytime, so we even don't write the if and multiply by 1 instead. The alternative would be to compile a variant of every shader for any representation, something we don't want. So, we have to assume a normal format transformation is free.

One important thing to know is that XreaL did huge efforts to read Doom3 assets, especially material files. Because of that, XreaL chose the Doom3 formats as default formats for many things, and basically idTech3 renderers are likely to mimic XreaL in their implementations, hence choosing what Doom3 chosen (spoilers: DirectX normal map format).

One important reason about agreeing on formats is not only the ability to port data, it's to reduce work required to make compatible third party tools. Many games would not see any interest in reading other's file and may do choices they think they would be the only ones to have to assume them… It's not true. When some cool tool happens, everyone wants it, and everyone wants the same tool, so they wants the same tool to know all formats. That will is difficult to fulfill when people do not agree on formats and every ones reinvent the wheel.

One good example of such third-party tool is https://github.com/SomaZ/Blender_BSP_Importer who just came to life days ago. This is a BSP level importer for Blender, with work in progress texturing support. This tool is a project by @SomaZ who is also working on OpenJK, so this tool supports OpenJK first then others games. By reducing differences between games we increase the possibility of being properly supported by such tools.

That Blender BSP importer is a huge step towards lightmap baking in OpenCL engines like Cycles.

This is awesome we know we can do that:

Solarium Xonotic Map in Blender:

Solarium Xonotic Map in Dæmon:

Solarium Xonotic Map in DarkPlaces:

Then, it would be good to avoid to introduce undecidable problems about format textures because we would use different format with same material syntax. Even without the need to port assets between engines or games we need to avoid that.

So, that said, the issue is basically about the differences between:

legacy assets that are supposed to not have to be edited to be loaded right;
maintained assets that can be edited to be loaded right.

Since maintained assets can be edited, there is nothing to mind about them, we can use whatever format, the assets would follow.

The real question is about interoperability with other engines, and in that topic, the biggest concern would be interoperability on import. So, what we would take care about would be to make the best choices to load legacy unedited assets.

So, I investigated some games.

idTech1-based DarkPlaces-based games like Xonotic

DarkPlaces, so Xonotic, uses OpenGL normal map format.

As DarkPlaces normal map loading is very different from our engine (it does not rely on Quake3 shaders but on filename suffix), it's very easy to implement an engine that would load the DP's filename-suffix-based normal map one way and the shader-based normal map another way. Our engine already reached that point.

DarkPlaces so Xonotic is a kind of engine that uses another way to describe their materials than we do. So there is absolutely no issue to have different formats. DarkPlaces and Xonotic can use idTech3-like .shader files but they do not describe normal maps in them. So we don't have to care about that.

So, while it would be cool to support the same format internally, we don't have to. If a Xonotic game on Dæmon engine becomes a thing, legacy assets would be loaded the DarkPlaces way, with the proper format, and newer assets, hence maintained, can be converted, or use special keywords to switch format.

Note: Xonotic uses IBSP46 for its official maps while DarkPlaces support other formats, I guess Quake 1 and Quake 2 formats but I've not checked.

idTech2-based QFusion-based games like Warsow

There is no plan for any port, but it is one major game out there using .pk3 archives, .bsp files not very different than ours, .shader files like ours etc. Those assets deserve a look. Also a game like Xonotic may be interested in loading maps from Warsow as they are both arena shooters.

Warsow uses OpenGL normal map format.

Warsow also stores materials in scripts/<something>.shader files, but fortunately the way they define normalmaps is very different than us:

textures/wsw_city1/tech_concrete_pattern_inv
{
    qer_editorimage textures/wsw_city1/tech_concrete_pattern_inv.tga

    {
        material textures/wsw_city1/tech_concrete_pattern_inv.tga textures/wsw_city1/tech_concrete_pattern_norm.tga textures/wsw_city1/tech_concrete_pattern_gloss.tga
    }
}

They basically have a material keyword with a list of textures as values and the nature of the texture is defined by the order in the list.

We can stick to DirectX format for the default format for normal map found the way we always did (normalMap keyword), and if we want to grok the Warsow assets, we would have to enable the OpenGL format for the normalmap described this way.

They also use _norm suffix, but nothing enforces it.

Note: Warsow uses the FBSP1 format, which is very close to IBSP46 (major difference is that the internal lightmaps are larger).

idTech2-based CRX-based games like AlienArena

AlienArena uses normal map OpenGL format.

When the normal map suffix is _nm (while heightmap is _hm) it seems the file is autodetected. The _normal suffix can also be found, those looks to be explicitely sets in material files named like scripts/<something>.rscript. Their format are similar to the idTech3 .shader format but uses their own keywords, like this:

players/martianenforcer/red
{
    {
        map players/martianenforcer/default_normal.tga
        normalmap
        map2 gfx/defaultmartianfx.tga
        glow
    }
}

Since it's an arena shooter game, and this genre is popular, it may be possible one day someone wants to port such maps to another game.

What seems to look like an suffix for autodetection looks to be pretty unique, and the material files use an unique naming pattern and an unique wording pattern. An engine would be able to select the right normal map format very easily.

Note: map format is IBSP38.

idTech2-based Quetoo-based games like Quetoo

The Quetoo game running on Quetoo engine uses the DirectX normal map format. The _nm suffix is used but does not look to be enforced.

Materials are defined in files named like materials/<something>.mat, which is very unique, and while the format is close to the idTech3 one, it uses different keywords in a wording that looks very unique:

{
    material chastity/br_arch
    normalmap chastity/br_arch_nm
    bump 1.2
    hardness 0.8
    specular 1
    parallax 0.5
}

I've already seen Quetoo maps ported to Xonotic, which is not surprising since they are both arena shooters.

Note: map format is IBSP69.

idTech3-based games like World of Padman

Quake III Arena, Quake Live, Reaction, World of Padman, do not use normal maps at all.

Note: those games usually uses IBSP46 while Quake Live uses IBSP47. Those formats are almost, if not, the same. Our engine supports both.

idTech3-based games like Smokin' Guns

Smokin' Guns does not use normal maps at all. In any way, if a port on Dæmon becomes a thing, they want to port the assets. So, there is nothing to care about.

Note: map format is IBSP46.

idTech3-based games like Bumpy UrbanTerror

Classic UrbanTerror did not used normal maps at all.

The Bumpy version of it used DirectX normal map format. There is no material file describing them, they are loaded a bit like the DarkPlaces engine does, looking for files with given suffix. The suffix is _n like the one we recommend (but not enforce), but for that game, it is enforced. So, a code detecting Bumpy UrbanTerror normal maps to select the format would also detect our textures.

Note that their heightmaps are embedded in normal map alpha channel and are a bit weird: they seem to define an elevation (black being the floor and white an elevation above the floor level) while parallax is usually implemented the other way (white being the floor and black being a lowering under the floor level).

That Bumpy version was just a showcase I highly doubt it had been played on a large case, we don't have to take care that much about this.

Urban Terror Resurgence will be an UT4 game, a complete move in all file format, we don't have to take care about that.

Note: map format is IBSP46.

idTech3-based games like RealRCTW

Return to Castle Wolfenstein, the WildWest mod etc. did not used normal maps at all.

The RealRTCW project which improves the renderer and ships new textures seems to not ship any normal map at all. I don't know what's their plans.

idTech4-based games like TheDarkMod

Doom 3, Quake 4, TheDarkMod use normal map DirectX format.

They store materials in files named materials/<something>.mtr. So whatever format we use in our scripts/<something>.shader file, it would be easy to interpret those normalmaps described in idTech4 materials the DirectX way whatever the default format we chose for the ones described in our scripts/<something>.shader files.

Those maps are not distributed the .bsp way, so in all case, in an imaginary world where Xonotic runs on Dæmon and people wants to play Quake4 community maps on Xonotic, they would have to be ported. This is doable: there are patches out there for q3map2 to compile idTech4 .map files to q3bsp. In that process, the map porter can also convert the material file or the normal file itself. A port is required in any way.

And even if Dæmon was able to load maps and materials the idTech4 way, that would not conflict with our own material language.

They also use _local suffix, but nothing enforces it (and variants like _local2 have been seen).

As we remember XreaL was attempting to use native idTech4 formats, our engine may be already able to partially parses idTech4's .mtr files with it's .shader parser.

idTech3-based games like ET:Legacy

Wolfenstein: Enemy Territory did not used normal maps at all.

ET:Legacy work in progress assets are using the DirectX normal map format. They have not released any normal map officially so they can still decide what normal map format to chose.

Their assets are using the DirectX normal map format because their renderer was also based on ET:XreaL, like us.

As material file naming and syntax, they used the exact same file naming and syntax we used in 0.51.1 and before. They seem interested to use the pre-collapsed stage syntax (see #216).

So, it would be very bad if we do not agree with ET:Legacy on a normal map format as there would be no way to guess the normal map format.

ET:Legacy renderer is based on XreaL like us. They are not against merging our renderer if it's possible. They haven't yet released any asset with normal map. So the best way to make sure there is no issue is to work together.

idTech3-based OpenJK based games like Jedi Knight

They use almost the same shader format we introduced in master (pre-collapsed stage, see #216). We may have differences with them, but the normal map syntax is exactly the same.

This is their syntax:

textures/pbr_materials/metal01
{
    qer_editorimage textures/pbr_materials/metal01/metal01
    {
        map textures/pbr_materials/metal01/metal01_d
        normalHeightMap textures/pbr_materials/metal01/metal01_nh
    }
    {
        map $lightmap
        blendfunc GL_DST_COLOR GL_ZERO
    }
}

This is our new syntax:

textures/test-pbr_custom/metal01
{
    qer_editorImage textures/test-pbr_custom_src/metal01_d
    {
        diffuseMap textures/test-pbr_custom_src/metal01_d
        normalHeightMap textures/test-pbr_custom_src/metal01_nh
    }
}

And our engine supports also this (uncollapsed normal map):

textures/pbr_materials/metal01
{
    qer_editorimage textures/pbr_materials/metal01/metal01
    {
        map textures/pbr_materials/metal01/metal01_d
        normalHeightMap textures/pbr_materials/metal01/metal01_nh
    }
    {
        map $lightmap
        blendfunc GL_DST_COLOR GL_ZERO
    }
}

Which… is the same. Exactly the same format.

If we don't use the same normal map format together, we are screwed: there would be no way to guess the normal map format only by reading the files.

OpenJK games stores shaders in files named like shaders/<somename>.shader and we stores them in files named like scripts/<somename>.shader. We can use that difference to guess the normal map format, but since the syntax is the same, agreeing on the same format would need a port from one engine to another to just require a rename of a directory, no more. Third-party tools would be easily fooled if we use same syntax but different format, only varying by directory name.

Because that's one important thing to mind: even if the dæmon engine does not load assets from those listed games, we can imagine map editors, and other tools for artists that manage all or part of them, this making a requirement for a way to know what is what.

So, to sum it up:

We can use any default format we want in renderer itself;
We have to take care of things in the material parser;
We can support various syntax expecting one or another format;
We can add a special keyword to tell the format whatever the default of a given syntax (see #307);
It looks like we have to stick with DirectX (X -Y Z) format for the normalMap keyword inherited from XreaL because many projects just did it like XreaL in the past ten years.

The status is:

The Quake 3/Doom 3 material parser assumes DirectX normal map format if nothing else is said because of the huge legacy of XreaL and Doom 3 but anyone can use normal map in OpenGL format with the use of NormalFormat X Y Z keyword in material, and anyone can do variant of X/-X, Y/-Y, Z/-Z to set arbitrary formats.

Most Unvanquished normal maps are known to use DirectX format, some having been designed for XreaL/Dæmon in mind, some may have been ported directly from texture packs targeting engines with DirectX convention like Unreal Engine, some others following the DirectX convention for unknown reasons. Some normal maps like the ones from the Vega texture set use custom format with appropriate material keyword being used (maybe that custom format was an export mistake).

The Darkplaces compatibility layer of the Quake 3 material parser assumes OpenGL format because that's what does the DarkPlaces engine.

As seen as a blackbox, the renderer considers that all per-channel factors being unset means 1, 1, 1 and means OpenGL format (X Y Z), the material parser may then switch those values according to the convention used by the material being read (Quake 3, Doom 3 or Darkplaces) or by the material using explicit keywords like normalFormat.

Unlike normalScale material format that can invert channels relatively to their current orientation (by setting negative values like normalScale 1 -1 1), the normalFormat is absolute : setting normalFromat X -Y Z will not invert the Y channel if the Y channel is already considered as reversed for the corresponding conventions (for example if it's an XreaL or Doom 3 material).

The GLSL code itself is written the DirectX way for historical purpose and no one took the risk to modify it. Because the renderer as a blackbox assumes OpenGL if per-channels factors are set to 1, 1, 1 (X Y Z), the GLSL code inverts the Y channel before processing to keep the validated historical code unmodified.

This issue is about the internal engine representation, not the input format for shaders.

Never mind, I think I confused this issue with something else.

The thread started about internal representations but now all aspects if this problem have an answer and can be considered as “decided”.

We would have to turn my previous comment into documentation written in the wiki.

Someone can still pick the task to revert orientation in GLSL shader internal representation and validate the change if he wants. But for now, the engine can already feed the GLSL shader with normal map data using OpenGL orientation without needing to tell the GLSL shader to revert some channel.

DaemonEngine / Daemon