search

niftools / nifxml

A repository for the nif.xml file, which contains the nif file format description.
http://www.niftools.org
GNU General Public License v3.0
37 stars 43 forks source link

Num UV Sets fix for Bethesda games #51

Closed neomonkeus closed 7 years ago

neomonkeus commented 8 years ago

@niftools/nifxml-reviewer

This PR is a result of a conversation between @ttl269 & @jonwd7 about setting ExtraVectorsFlags + Num UV.

The problem with the nif.xml currently is that in the case of bad (unexpected) values entered into "Num UV Sets" Nifskope produces an error and the nif doesn't load/render. He originally entered value 4095 which caused error with new nif.xml.

Initial discussion was whether or not it is better to keep the value split a a pair of bytes "ExtraVectorsFlags" and "Num UV Sets or BS Num UV Sets" or revert it to original ushort "BS Num UV Sets", so the values like 4097 would be back again.

The commit proposes how to keep two separated values with "protection" against unexpected values. Update to ensure 'Num UV Set's for Skyrim and Fallout nifs are immune from unexpected values.

neomonkeus commented 8 years ago

This will also need a commit to update the version to 0.7.1.2

neomonkeus commented 8 years ago

Updated issue summary with issue background as provided by @ttl269

hexabits commented 8 years ago

@ttl269 @neomonkeus There is actually a regression taking place from the previous UV changes. I did a full scan on all FO3 meshes and got this: https://gist.github.com/jonwd7/390c5d88b3f64766e8f0dc301fc1f7a9

The files listed are:

meshes/architecture/megaton/megatonrampturn45sml.nif
meshes/architecture/megaton/megatonrampturn90sml.nif
meshes/armor/raiderarmor01/m/glovel.nif
meshes/characters/head/headfemalefacegen.nif
meshes/dungeons/office/rubblepiles/offrubblechunka03.nif
meshes/dungeons/office/rubblepiles/offrubblechunka01.nif
meshes/dungeons/office/rubblepiles/offrubblechunka06.nif
meshes/dungeons/office/rubblepiles/offrubblechunka02.nif
meshes/dungeons/rivetcity/exterior/rcexbridge.nif
meshes/dungeons/rivetcity/exterior/rcextaircarrierback.nif
meshes/dungeons/rivetcity/exterior/rcexaircarrierfront.nif
meshes/dungeons/rivetcity/exterior/rcaircarriertower.nif
meshes/dungeons/rivetcity/exterior/rcextaircarrier01.nif

Other regressions:
meshes/dungeons/office/rubblepiles/offrubblechunka04.nif
meshes/dungeons/office/rubblepiles/offrubblechunka05.nif

And then several at the bottom of the output that don't seem to be related to this issue. They are all in meshes/triggers. After some more testing I will narrow those down and create an issue.

I tracked it down to the changes done between 55792903ed55d67532553fa7ae905c03232d1724 and bbd0d5e2511e4c6f8cc31fd2e84905ff6233a9e4 . The XML from one commit will load these meshes fine, the other will not.

So, the changes from this PR should be verified on the above files as well. I will try to get to it and leave an update. In the meantime I will be creating a ticket for the other two regressions.

neomonkeus commented 8 years ago

@jonwd7 & @ttl269, marking this as mergeable based on comments in #52?

hexabits commented 8 years ago

Well @ttl269 and I discussed this over PMs, but I posited that in Skyrim NIFs it's actually 1 = "Has UVs" and 4096 = "Has Tangents/Bitangents". Because if you take a look at the preview scene graph, even if I set "Num UVs" for Skyrim NIFs to 0/1 there is still the value "4096". It tells me that maybe it's a ushort bitflag with only two used values.. 1 and 4096.

The left is with the standard values and the right is turning off UVs (but leaving Tangents/Bitangents on). You can see it still says 4096 on both so that leads me to believe that the flag setup is not the same as what ttl269's changes are doing.

Unfortunately this preview app doesn't display the value of the other bit for that area but if it did I would assume it'd be "Has UVs" and not "Num UVs" as we already know there can only ever be one UV set.

neomonkeus commented 8 years ago

General usability observation, if be nice have the options as flags in some form. 1 & 4096 mean nothing to users.

hexabits commented 8 years ago

I propose something like this instead (irrelevant stuff removed):

    <bitflags name="BSVectorFlags" storage="ushort">
        <option value="0" name="Has UVs" />
        <option value="12" name="Has Tangents" />
    </bitflags>

    <niobject name="NiGeometryData" abstract="1" inherit="NiObject">

        <add name="Num UV Sets" type="byte" vercond="((Version &gt;= 10.0.1.0) &amp;&amp; (!((Version == 20.2.0.7) &amp;&amp; (User Version &gt;= 11) &amp;&amp; (User Version &lt;= 12))))" calculated="1">Number of UV texture sets.</add>
        <add name="Vector Flags" type="VectorFlags" vercond="((Version &gt;= 10.0.1.0) &amp;&amp; (!((Version == 20.2.0.7) &amp;&amp; (User Version &gt;= 11) &amp;&amp; (User Version &lt;= 12))))">Bit 4: Has Tangents/Bitangents</add>
        <add name="BS Vector Flags" type="BSVectorFlags" vercond="((Version == 20.2.0.7) &amp;&amp; (User Version &gt;= 11) &amp;&amp; (User Version &lt;= 12))" />

        <add name="Normals" type="Vector3" arr1="Num Vertices" cond="Has Normals">The lighting normals.</add>
        <add name="Tangents" type="Vector3" arr1="Num Vertices" cond="(Has Normals) &amp;&amp; ((Vector Flags &amp; 16) | (BS Vector Flags &amp; 4096))" ver1="10.1.0.0">Tangent vectors.</add>
        <add name="Bitangents" type="Vector3" arr1="Num Vertices" cond="(Has Normals) &amp;&amp; ((Vector Flags &amp; 16) | (BS Vector Flags &amp; 4096))" ver1="10.1.0.0">Bitangent vectors.</add>

        <add name="UV Sets" type="TexCoord" arr1="((Num UV Sets &amp; 63) | (BS Vector Flags &amp; 1))" arr2="Num Vertices">The UV texture coordinates. They follow the OpenGL standard: some programs may require you to flip the second coordinate.</add>

    </niobject>

Which gives you this:

fdsfdsfdseeed

For Skyrim meshes.

Also, I should easily be able to scan all of Skyrim NIFs to see if that 2 bytes ever differs from or exceeds 4097 and if so, why.

hexabits commented 8 years ago

@ttl269

So I realized that this commit still doesn't work if you are setting BS Num UV Sets to 2 or 4 or 8, etc. because the 0th bit will still be 0 and the | (BS Num UV Sets &amp; 1) will fail. If anything it should be BS Num UV Sets &gt; 1 ... which still isn't right because we shouldn't be letting them set it anything to 0 or 1 in the UI.

I scanned all of Skyrim meshes and using my above method there were no values other than 0/1/4097 for the BS Vector Flags bitflag.

I began seeing if FO3 and Oblivion meshes could also adapt to using the BS Vector Flags, and I only ran into two issues:

D:/Oblivion/Data/meshes/meshes/clutter/middleclass/middlechestbrokenbottom02.nif (20.0.0.5)
"array Data Layers much too large. 1788635711 bytes requested"
"failed to load block number 5 (bhkNiTriStripsShape) previous block was NiTriStripsData"
D:/Oblivion/Data/meshes/meshes/clutter/middleclass/middlechestbrokenbottom03.nif (20.0.0.5)
"array Constraints invalid"
"failed to load block number 5 (bhkRigidBodyT) previous block was bhkNiTriStripsShape"

Looking into it, these are the only two meshes across Oblivion, Fallout 3, and Skyrim that have UV Sets > 1. The issue is that this above commit still displays the data incorrectly to the user:

cdsedfsdsf

It says Num UV Sets == 66 when secretly it's only 2 because of the bit masking. The confusing thing is that these two UV Sets are on Havok collision data. What are UVs doing on a bhk* block anyway?

@ttl269 Also, if it's really masking off with & 63 how did you arrive at that conclusion? Are there non-Bethesda NIFs that use that many UV sets?

hexabits commented 8 years ago

With still treating the Num UV Sets as a separate byte for Oblivion here is everytime it != 1: https://gist.github.com/jonwd7/3e93da7348b273bb86aae4e7ecd5c608

Block number in brackets.

As you can see it's clearly a part of a larger bitflag.

If I use this

    <bitflags name="BSVectorFlags" storage="ushort">
        <option value="0" name="UV_Set_1" />
        <option value="1" name="UV_Set_2" />
        <option value="2" name="Unk2" />
        <option value="3" name="Unk3" />
        <option value="4" name="Unk4" />
        <option value="5" name="Unk5" />
        <option value="6" name="Unk6" />
        <option value="7" name="Unk7" />
        <option value="8" name="Unk8" />
        <option value="9" name="Unk9" />
        <option value="10" name="Unk10" />
        <option value="11" name="Unk11" />
        <option value="12" name="Has_Tangents" />
    </bitflags>

    <niobject name="NiGeometryData" abstract="1" inherit="NiObject">

        <add name="Num UV Sets" type="byte" vercond="((Version &gt;= 10.0.1.0) &amp;&amp; !((Version == 20.2.0.7) || (Version == 20.0.0.5)))" />
        <add name="Vector Flags" type="VectorFlags" vercond="((Version &gt;= 10.0.1.0) &amp;&amp; !((Version == 20.2.0.7) || (Version == 20.0.0.5)))">Bit 4: Has Tangents/Bitangents</add>
        <add name="BS Vector Flags" type="BSVectorFlags" vercond="((Version == 20.2.0.7) || (Version == 20.0.0.5))" />

        <add name="Normals" type="Vector3" arr1="Num Vertices" cond="Has Normals">The lighting normals.</add>
        <add name="Tangents" type="Vector3" arr1="Num Vertices" cond="(Has Normals) &amp;&amp; ((Vector Flags &amp; 16) | (BS Vector Flags &amp; 4096))" ver1="10.1.0.0">Tangent vectors.</add>
        <add name="Bitangents" type="Vector3" arr1="Num Vertices" cond="(Has Normals) &amp;&amp; ((Vector Flags &amp; 16) | (BS Vector Flags &amp; 4096))" ver1="10.1.0.0">Bitangent vectors.</add>

        <add name="UV Sets" type="TexCoord" arr1="((Num UV Sets &amp; 63) | (BS Vector Flags &amp; 3))" arr2="Num Vertices">The UV texture coordinates. They follow the OpenGL standard: some programs may require you to flip the second coordinate.</add>

    </niobject>

I can scan all of Oblivion, FO3, and Skyrim NIFs with the same bitflags. I can also print out a list of NIFs that use Unk7/Unk8/etc. There are several.

Edit: I must have missed something because apparently & 3 does not work for Fallout 3, only & 1 ... Meaning the games can not all use the same bitflag.

Anyway, if maybe I could get a good sampling of non-Bethesda NIFs I could actually see how they handle Num UV Sets as well.

ttl269 commented 8 years ago

@jonwd7 Thanks for your exhaustive nif testing. I think that these two Oblivion meshes with 2 UV sets in NiTriStrips used in bhkNiTriStripsShape are Bethesda's mess. There is no use for UV sets in havok mesh. Unfortunately we must react to them too. So @jonwd7, your last code proposal seems to be final and right for Bethesda nifs. Although it is still weird to me, it seems much better to represent that discussed value for modders as ushort bitflags for Bethesda games even with chanciness of knowledge of other bits. All we can state is:

So we must count with number of 0,1 or 2 UV sets for Bethesda's meshes for Oblivion, Fallout 3, Skyrim.

I propose only minor modification of jonwd7's BSVectorFlags:

    <bitflags name="BSVectorFlags" storage="ushort">
        <option value="0" name="Has UV Set 1" />
        <option value="1" name="Has UV Set 2" />
        <option value="2" name="Unk2" />
        <option value="3" name="Unk3" />
        <option value="4" name="Unk4" />
        <option value="5" name="Unk5" />
        <option value="6" name="Unk6" />
        <option value="7" name="Unk7" />
        <option value="8" name="Unk8" />
        <option value="9" name="Unk9" />
        <option value="10" name="Unk10" />
        <option value="11" name="Unk11" />
        <option value="12" name="Has Tangents/Bitangents" />
        <option value="13" name="Unk13" />
        <option value="14" name="Unk14" />
        <option value="15" name="Unk15" />
    </bitflags>

And assuming that you renamed ExtraVectorsFlags to VectorFlags.

Also, if it's really masking off with & 63 how did you arrive at that conclusion? Are there non-Bethesda NIFs that use that many UV sets?

This AND 63 is not my adding. It was already there before I started to make changes to nif.xml. Although it is true that I don't remember to any other game nif with Num UV sets higher than 2, I still think that it is better to keep it as it is (speaking about non-Bethesda nifs).

hexabits commented 8 years ago

@ttl269 Sorry, you may have missed it but I found out that (BS Vector Flags &amp; 3) wasn't actually working for all FO3 NIFs because I guess the 2nd bit is used for something else in FO3. Meaning the 2nd bit being set does not mean 2 UV Sets. :(

So until I figure out what exactly the issue is we're kind of back where we started. I will scan the FO3 meshes and print out which NIFs have that 2nd bit set and try to find some correlative data to figure out its use in FO3.

But at the very least now we need a different condition just for FO3 because my current proposal only works for Oblivion/Skyrim.

hexabits commented 8 years ago

Here are the FO3 NIFs where BS Vector Flags & 2 is set:

D:/Fallout 3/meshes/architecture/megaton/megatonrampturn45sml.nif (20.2.0.7)
"[10] 1027"
"[13] 1027"
D:/Fallout 3/meshes/architecture/megaton/megatonrampturn90sml.nif (20.2.0.7)
"[10] 1027"
"[13] 1027"
D:/Fallout 3/meshes/armor/raiderarmor01/m/glovel.nif (20.2.0.7)
"[29] 4099"
D:/Fallout 3/meshes/characters/head/headfemalefacegen.nif (20.2.0.7)
"[6] 4099"
D:/Fallout 3/meshes/dungeons/office/rubblepiles/offrubblechunka01.nif (20.2.0.7)
"[4] 3"
D:/Fallout 3/meshes/dungeons/office/rubblepiles/offrubblechunka03.nif (20.2.0.7)
"[4] 3"
D:/Fallout 3/meshes/dungeons/office/rubblepiles/offrubblechunka02.nif (20.2.0.7)
"[4] 3"
D:/Fallout 3/meshes/dungeons/office/rubblepiles/offrubblechunka06.nif (20.2.0.7)
"[4] 3"
D:/Fallout 3/meshes/dungeons/office/rubblepiles/offrubblechunka04.nif (20.2.0.7)
"[4] 3"
D:/Fallout 3/meshes/dungeons/office/rubblepiles/offrubblechunka05.nif (20.2.0.7)
"[4] 3"

And the blocks where these occurred in brackets. The 2nd number is the integer value of the bitflag. None of these blocks have 2 UV Sets, the bit means something else.

So, I believe for 20.0.0.5 ONLY, can (BS Vector Flags &amp; 3) be used for the arr1 value for UV Sets. Unfortunately FO3 cannot use the same bitflags enum as Oblivion and Skyrim either. And after thinking about it some more, the only known valid values for Skyrim are bits 1 and 4096 so the other bits that are set in Oblivion and Fallout should not be selectable while in a Skyrim mesh.

I will do more analyzing of Oblivion/Fallout NIFs to try to find some correlative data with the unknown bits.

hexabits commented 8 years ago

FO3 Unknown Vector Flags (including & 2) https://gist.github.com/jonwd7/d2f32b4cf57128f486d69d2b905efb2e

OB Unknown Vector Flags https://gist.github.com/jonwd7/2282328ca352d580781c7cbb946cf026

Other than the bit & 2, I am hoping that the FO3 and Oblivion uses of the other bits are the same. I should also verify that every Vector Flags is actually a bitflag but at this point I am fairly certain of that.

ttl269 commented 8 years ago

Two+ hours of opening/watching in various Fallout 3 meshes but still no idea about any use of other bits of Vector Flags. I don't see any understandable rule of their presence or their realationship with any other parameter in the nif file. Everytime I start to think that I found some meaning it comes other nif which break my idea :(

ttl269 commented 8 years ago

@jonwd7

Anyway, if maybe I could get a good sampling of non-Bethesda NIFs I could actually see how they handle Num UV Sets as well.

I have collected some samples for you.

hexabits commented 8 years ago

@ttl269 Thanks. Actually while looking at endianness in NifSkope I noticed you mention something in this issue:

Note: there is also one issue related to Extra Vectors Flags and Num UV Sets. Due to authors of that big endian nif file assumed that these two parameters are one (stored in short - as two-byte), these two bytes are also swapped in their nif file. And saving that nif in little endian order causes loosing Tangents, Bitangents and UV Data - even that this reversed order is covered in nif.xml (by conditions), changing Endian Type to little endian in Nifskope fills them with 0. But it is not Nifskope error. This will have to be solved separately and I am afraid that on Nifskope's side, because there is no support for bit shift operations in conditions in nif.xml.

Given that Num UV Sets and Extra Vectors Flags needed to be swapped for big endian, doesn't this prove that even non-Bethesda NIFs read that area as one ushort? You said "authors of that big endian nif file assumed that these two parameters are one" but I'm pretty sure it's our assumption that they were two parameters at all. I'm guessing the big endian was either an official NIF for that game or created by an official tool for that game. And if the NIF file worked in the game, then that means the bytes are in fact not swapped by mistake and it's our mistake with the XML by treating them as two bytes.

Just a small update, if I go back to treating the area as one ushort instead of two bytes, I can read that area just fine in big endian without XML changes.

I haven't looked for non-Bethesda NIFs between 10.x and 20.x where the actual UV Sets are > 2. I have found ones with 2 UV Sets though, and treating them exactly like Oblivion is working.