search

Mutagen-Modding / Mutagen

A .NET library for analyzing, creating, and manipulating Bethesda mods
GNU General Public License v3.0
121 stars 32 forks source link

Support writing /w Compression flag on #235

Open Noggog opened 3 years ago

Noggog commented 3 years ago

Right now mutagen doesn't export /w any compression mechanics. Just not high priority feature as all it does is save space on-disk, as far as I'm aware. We can eventually add it for completeness

The main aspect to getting this done will be doing any research to know that we're using the correct compression when writing out. Not sure either way if that would cause problems, but theoretically could.

In the meantime, if you want to export a mod that has compressed records, this is an easy fix to make them all not compressed before exporting:

foreach (var rec in mod.EnumerateMajorRecords())
{
    rec.IsCompressed = false;
}
ghyujkilop commented 2 years ago

I might be wrong in saying this, but when using something like Remove Landscape Vertex Color the loading time is DRASTICALLY increased. And if you look at the records it's only removing vertex colors. However, when you compare the data size difference https://i.imgur.com/OBDUIMW.png you can see that this would be a primary cause of wildly increased loading times. Could this priority of being added be raised due to increased loading times?

Noggog commented 2 years ago

Good points. Thanks for the feedback. Will be seeing if/when I can slide in the feature

JMPZ11 commented 4 months ago

What is the status on this issue?

For starfield, I can't really use the output if it doesn't compress the cells, as I think Starfield always compresses them.

It isn't a huge deal, but it does mean I must open the output in xedit to fix the cells, so I can't really automate yet.

Thanks!

Noggog commented 4 months ago

I think the main blocker is just researching the proper compression choice, and finding the right C# library and call to use to get a resulting compression that the game can read. I know there's a few out there, so I didn't want to just slap any zip call I could find on the system

If anyone has any research on the proper compression choice to use, that'd help move it along!

JMPZ11 commented 3 months ago

I think the main blocker is just researching the proper compression choice, and finding the right C# library and call to use to get a resulting compression that the game can read. I know there's a few out there, so I didn't want to just slap any zip call I could find on the system

If anyone has any research on the proper compression choice to use, that'd help move it along!

How does mutagen read the cells that are compressed? Could that provide insight into the algorithm being used maybe?

Noggog commented 3 months ago

Here is the file that is doing the work, currently https://github.com/Mutagen-Modding/Mutagen/blob/dev/Mutagen.Bethesda.Core/Plugins/Binary/Streams/Decompression.cs#L71

The main question at hand is whether this Zlib library able to decompress many incoming alternatives.. but the reverse might not be true.. where it's exporting a specific compression algo that the game might expect

It might not be a problem at all, but I would want to do some tests to show that before pushing out an update that suddenly starts CTDing peoples mods.

Avenues for solution that I see:

JMPZ11 commented 1 month ago

I asked the xedit folks and got this:

"that would be zlib or lz4" "Same as in tes4"

Does this help? Sorry for the delay responding - this is the one thing I'm aware of that causes the output to require manual correction for Starfield.

Is zlib able to be configured with verbose logging maybe? Or are the .net binding capable of reporting the compression used?

If I could just get a sample of the data being decompressed I'd be happy to do the legwork -- I just don't know how to extract the relevant bits to test myself...