BadDogSkyrim / PyNifly

Export/Import tools between Blender and the Nif format, using Bodyslide/Outfit Studio's Nifly layer. Supports Skyrim LE, Skyrim SE, Fallout 4, Fallout New Vegas, Fallout 76, and Fallout 3.
GNU General Public License v3.0
179 stars 16 forks source link

Vertex Colors changes on export #242

Closed VessperiaS closed 2 months ago

VessperiaS commented 2 months ago

Currently modding Fallout 4 hairstyles. Noticed that after export mesh changes it's colors. The thing is hairstyles are colored by gradient texture that applies it's colors by vertex colors (the whiter the vertex color the lower color is taken from palette texture of hair colors). If color shifts a bit > you are getting almost completely different color on hairstyle since the numbers are very limited (5 shades per one hair color avaliable). Due to this problem of distorted colors on meshes that use gradients (in Fallout, these are not only hairstyles, but sometimes clothes and armor), the mesh can be severely damaged on export. For example, if the rgb value was 251,251,251 on the hair roots (the extreme available value for applying a darker color for hairs) > the vertexes have a darker color on export, below 251,251,251 > then the standard darker ginger hair color on the roots becomes completely different, since it takes an adjacent gradient from the hair color palette which is called "light brown" ingame.

image Ginger and Light Brown at the very bottom.

This is very simple to replicate if you take male hairstyle's hairline (since it is informative to visual comparison), import to blender and export right after. изображение_2024-09-06_181451082 _Male Hair26Hairline.nif for example. Colors distorted, but but they do not go below 251,251,251, since the original mesh has maximum color values closer to 255,255,255. There are other meshes that have too low values for the correct appearance on export with PyNifly.

image And here is actual comparison of colors applied to the same vertex (see the position to verify its the same).

My guess that this might happen because of the type of element the color attribute is stored on - for now it is Face Corners. I may be wrong, but I've never heard of models storing colors this way in games. Most likely, the vertex should be specified as the type. Maybe on import vertex colors are distorted because they converts to face corners and it mixes the colors a bit. image If I try manualy convert the color layers to vertex type, the nif simply refuses to export and shows an error.

In addition, this whole color distortion problem has almost no effect on the alpha channel. Alpha affects transparency in the game, but in fact the problems are not noticeable because alpha uses the entire range of black and white values from 0 to 255.

BadDogSkyrim commented 2 months ago

Yes, probably colors should be on vertices. I don't think that was available when I implemented them. Face corners should be correct as long as all corners have the same colors. If they don't you should see a seam in Blender.

I think you're saying there's a bug where the vertex color value isn't the same on export as it was on import. I'm not sure what the problem might be, but I did accept a change having to do with calculating values that would show up correctly in the viewport, so there might be a bug there. But it will be about handling the color value, regardless of face corner color vs vertex color.

k5tQp2ET commented 2 months ago

Yea, anything with vertex color is a pain to work with.

BadDogSkyrim commented 2 months ago

Fixed in next release.

VessperiaS commented 1 month ago

Hello! Thank you for your work!

I tested the new update but faced another bug. Everything is complitely fine with how "Col" attribute layer exports, but not with "VERTEX_ALPHA". Something should be tweaked a little. For example I am using FO4's model - Meshes\Actors\Character\Hair\Female\FemaleHair05_Hairline.nif.

Here is what happens: image

Here's how VERTEX_ALPHA looks in viewport. Completely fine. image

I can almost fix that by converting "VERTEX_ALPHA" to store it's info on face corners back again. Right after it exports fine, but the appearance still changes! As you can see, vanilla mesh is more subtle where transparency fading hapens. Exported mesh is harsher around the ears especially. image

BadDogSkyrim commented 1 month ago

Grr. K, will check.

BadDogSkyrim commented 1 month ago

OK, 18.2 should fix it.