Ensure proper compatibility with FNV

mklokocka commented 1 month ago

Hi, as discussed, there are several things that must be handled to have full compatibility with FNV (and other older titles I assume).

Crucial

[x] Archive flags - As per Geckwiki, for a BSA to be usable by the game, it must be flagged correctly. There are two sets of bits, archive and file flags. The relevant (for FNV) are the following:

Archive flags:
00000000000000001 - Named Directory - always set
00000000000000010 - Named Files - always set
00000000000000100 - Compressed - use zlib compression
00000000000010000 - Retain Names - set with sound effects
00000000010000000 - Startup Strings - set with meshes
00000000100000000 - Embed Name - set with textures

The first three are self-explanatory. The other three are important for the given asset type. That is, if the archive in question contains either of the noted types, it must have the given flag set.

The file tags are self-explanatory. The important thing is that for the BSA to be usable, it must have the correct file flags set depending on the content - otherwise, the game will never check the archive for the given type. Full mapping of file extensions to file flags (pulled from engine) in comments.

~Disable BSA compression completely~ (CAO) - Compressed BSAs load very slowly due to old zLib version. Updated zLib version is still slower than using uncompressed BSAs and there is no real benefit to compressing them in terms of game performance/memory.
~Disable mesh and animation optimizations~ (CAO) - Animation optimizations are irrelevant and mesh optimizations cause breakage from preliminary tests. As per discussion with a mesh optimizer, it's not really worth it to try to figure out an automatic process. Best choice is to disable both completely.
[x] Ignore packing the following files - A list of extensions to ignore for BSA packing, as they could have trouble loading correctly/working in game.

KF - This is a priority. Most animations are build with an engine extension that currently does not support BSA packed KF files.
MP3
XML
JSON
INI

Necessary improvements

[x] Dummy plugins generated by CAO are unusable by the game - Dummy plugins must have at least FalloutNV.esm set as master. Even when added manually the attached BSA failed to load, so something is probably wrong with how they are generated, must debug further.
~Do not process LOD textures at all~ (CAO) - LOD textures are processed elsewhere and compression and mipmaps are handled there. These are files with the suffix _lod.dds or _lod_n.dds (potentially others, but I am not sure if such are ever used).
[x] Do not recompress files with different algorithm - Mostly connected to the next point.
[x] Use correct compression algorithms - bethutils currently uses the wrong compression algorithms for older Beth games (defined here).

The game supports DXT1, DXT3, and DXT5. DXT3 is useless. The current logic causes problems - textures without any alpha will be compressed with BC5 which is uncompatible with Dx9 the game uses (as far as I can tell). Textures using a single bit of alpha get processed with BC3 which fills out the alpha (for some reason), which can for example mess up the specular on normal textures. The correct logic is:

if texture has up to 1 bit of alpha use DXT1 (BC1),
otherwise use DXT5 (BC3).

In fact, using DXGI_FORMAT_R8G8B8A8_UNORM when not compressing does break the specular as well (specular - alpha component of normal texture). Wonder if there is some explicit alpha adjustment being done by bethutil itself or it's due to the format.

~Mipmap generation - Mipmaps should be generated for everything but UI textures (textures/interface/*).~

Important note: Gamma correction should be applied (and only ever applied) to diffuse textures. Assuming standard conventions (described here) this means gamma correction should be applied to any texture not ending with _n/_m/_em/_g/_sk/_hl/_e.

Scratched. Turns out DirectXTex has no direct support for gamma correction of mipmaps. It could probably be done manually, but not worth the effort.

Nice to have

~Automatic .override generation~ (CAO) - .override files are empty text files used by an engine extension to load the corresponding BSA at the same "load order" as the plugin (similar to newer Bethesda titles). If we have a plugin test.esp and archive test.BSA, this means generating test.override. Similarly, if splitting archives, this would mean having test 0.bsa, test 0.override, test 1.bsa, test 1.override.

Not crucial, but would be very helpful.

mklokocka commented 1 month ago

Engine mapping of file extensions to file (archive) types:

"kf" -> ARCHIVE_TYPE_MESHES
"nif" -> ARCHIVE_TYPE_MESHES
"egm" -> ARCHIVE_TYPE_MESHES
"egt" -> ARCHIVE_TYPE_MESHES
"tri" -> ARCHIVE_TYPE_MESHES
"cmp" -> ARCHIVE_TYPE_MESHES
"lst" -> ARCHIVE_TYPE_MESHES
"dtl" -> ARCHIVE_TYPE_MESHES

"dds" -> ARCHIVE_TYPE_TEXTURES
"tai" -> ARCHIVE_TYPE_TEXTURES
"tga" -> ARCHIVE_TYPE_TEXTURES
"bmp" -> ARCHIVE_TYPE_TEXTURES

"wav" -> ARCHIVE_TYPE_SOUNDS

"ogg" -> ARCHIVE_TYPE_VOICES
"lip" -> ARCHIVE_TYPE_VOICES

"xml" -> ARCHIVE_TYPE_MENUS
"txt" -> ARCHIVE_TYPE_MENUS

"spt" -> ARCHIVE_TYPE_TREES

"fnt" -> ARCHIVE_TYPE_FONTS
"tex" -> ARCHIVE_TYPE_FONTS

"vso" -> ARCHIVE_TYPE_SHADERS
"pso" -> ARCHIVE_TYPE_SHADERS
"vsh" -> ARCHIVE_TYPE_SHADERS
"psh" -> ARCHIVE_TYPE_SHADERS
"lsl" -> ARCHIVE_TYPE_SHADERS
"h" -> ARCHIVE_TYPE_SHADERS

mklokocka commented 1 month ago

The striken-out points will better be handled by CAO (i.e. disabling compression, mesh/anim optimization, skipping LOD textures, override creation).

Guekka commented 2 weeks ago

Can we close this now @mklokocka?

mklokocka commented 2 weeks ago

Yeah, I think so. 👍

Guekka / bethutil