Mutagen-Modding / Spriggit

A tool to facilitate converting Bethesda plugin files to a text based format that can be stored in Git
GNU General Public License v3.0
88 stars 10 forks source link

Latest update fails to serialize many of my Starfield esms #57

Closed JMPZ11 closed 3 months ago

JMPZ11 commented 3 months ago

I noticed in the release notes that it was made "pickier" by vomiting on "unexpected records"

It is choking on REFL, which it never did before - I depend on that fact specifically.

Especially considering Bethesda is somewhat regularly adding new sub record types - could it not simply store unknown parts raw like I thought it used to?

It just feels like a big step backwards. My old version stopped working for some reason "System.ArgumentException: Object of type 'Mutagen.Bethesda.Plugins.ModPath' cannot be converted to type 'Mutagen.Bethesda.Plugins.ModPath'."

Like, it was working two days ago.

This is just my opinion so grain of salt - refusing to serialize a single esm that the game considers valid (i.e. any content from the vanilla files) is one too many.

As always, appreciate everything you do, and sorry for complaining - I have something I need to do I was able to do 2 days ago. I'm getting the feeling this new .net isn't as solid as the framework. And now I'm a dino lol.

Thanks so much!

JMPZ11 commented 3 months ago

I can work around it by using the 0.18.0 model to dump the esm, but the app tries to use the Skyrim model if you populate the version field.

My opinion stands though - the 0.19.0 model is nerfed to oblivion :-P

Noggog commented 3 months ago

It's intentional, as the mechanisms that Spriggit uses to do the translations does not handle unknown content well. The older versions were simply skipping those fields, which would lead to silent data loss (even worse, imo). The extra pickiness is a canary in the coal mine that the definitions need updating.

If you have mods that it cannot handle, I would recommend reporting the details (and including the file, when you can). I'll be adding a warning message in the UI to the same effect. I'll make sure to get any new definitions into the system, and push a new spriggit version (which is the intention to do decently often until everything is known)

JMPZ11 commented 3 months ago

I'm actively trying to work with things xedit doesn't support or allow. Spriggit has been a Godsend. I'm able to export large cells and use vscode to find things that would otherwise be needles in the stack.

Starfield 1.11.37 came with some significant updates - new base record types, and what would appear to be the ability to extend formlists without overriding them entirely. If I want to add a new class of ship part properly (supporting the new empty habs), I will need to be able to clone things xedit cannot yet read.

I built the nifskope addon that saved the community from "meshageddon" - I'm more than willing to get my hands dirty is what I'm getting at, and I'd really rather not have to bother you every time I bump into something new --

When I run into an unsupported field, what is the process to get it working? If it's documented feel free to point the way and I apologize for my poor searching skills lol. I'm happy to share my findings if I'm even capable.

I appreciate you taking the time to reply, and thanks for doing what you do!

Noggog commented 3 months ago

Spriggit is built on the back of Mutagen, which is built by generating hundreds of strongly typed classes /w known fields and types.
https://mutagen-modding.github.io/Mutagen/plugins/ These are great for coders that get well structured classes /w strong typing to use in their code.

Spriggit extends off this and looks at these class definitions, and generates serialization code from their fields. This is where it knows to get the label Health from, and that it's a floating point number to show as 1.234. It's because the Npc class has a field float Health

Because it's driven off these classes that are hyper aware of their contents, it doesn't do well with unknown fields. Previously it just skipped them; Now it's throwing an exception

That's just the reality of the tool, at the current moment. Another tool written on top of a different less structured library might be able to handle unknown fields better.

When I run into an unsupported field, what is the process to get it working? If it's documented feel free to point the way and I apologize for my poor searching skills lol. I'm happy to share my findings if I'm even capable.

I would say reporting unparsable mods to the Mutagen repo.
https://github.com/Mutagen-Modding/Mutagen/issues It's also important to know how the mod was made. If it was made from the CK, then it can be seen as "kosher" and its format should be supported. If it's made by hand or via some other 3rd party tool, then it's a bit more suspicious, as the formats might not be the "truth" for how things should be done.

Unknown fields need to be analyzed and folded in officially, as it's important for a number of reasons. We need to know which are FormIDs, fundamentally, or we can make corrupted mods. Certain fields are string fields that bounce off the .strings files, etc.

Spriggit is meant to be updated with latest definitions frequently https://mutagen-modding.github.io/Spriggit/translation-packages/

So reporting things it cannot parse is natural, and after a few initial waves, should settle down

JMPZ11 commented 3 months ago

Cool, so the problem here is Starfield 1.11.37 added quite a bit of new stuff xEdit can't parse yet - one of those being (what seems to be a static) ANAM subrecord on formlists. I think it may be necessary.. not sure.

I wouldn't want to go so far as upstreaming the findings until I was more sure - for now I just need to be able to include the one-off field here and there to try and diagnose a problem the latest update created - Sounds like this may not be the best forum to discuss anyhow - is there a discord or other better place to discuss? Thanks!

JMPZ11 commented 3 months ago

Also, by setting the serializer version to 0.18.0 using the 0.19.0 UI, I have been able to get around this issue, so I'll go ahead and close it. Thanks!