esfxml with Warhammer 2 repack

[Sharptsa]

Hi,

I'm trying to edit a save file for Warhammer 2 (EditSF can't do what I want) and I wanted to use esfxml for that.

Unpacking the save seems to work fine, only with semantic conversion failures.

Warning: Semantic conversion failures (low level conversion performed instead)* MAPS: (1 records, 1 failures, 0 quiet failures)

(this is OK if converting ESFs for game newer than Empire):

LZMA 22.01 (x86) : Igor Pavlov : Public domain : 2022-07-15

Input size:  6926814 (6 MiB)
Output size: 37801775 (36 MiB)
Warning: Semantic conversion failures (low level conversion performed instead)
* ANCILLARY_UNIQUENESS_MONITOR: (274 records, 274 failures, 0 quiet failures)

(this is OK if converting ESFs for game newer than Empire):
* CAI_REGION_HLCI: (1246 records, 1246 failures, 0 quiet failures)
* CAI_SITUATED: (717 records, 717 failures, 0 quiet failures)
* CAI_TECHNOLOGY_TREE: (272 records, 272 failures, 0 quiet failures)
* CAMPAIGN_LOCALISATION: (65803 records, 65803 failures, 0 quiet failures)
* COMMANDER_DETAILS: (3413 records, 3413 failures, 0 quiet failures)
* FACTION_FLAG_AND_COLOURS: (781 records, 781 failures, 0 quiet failures)
* MAPS: (1 records, 1 failures, 0 quiet failures)
* POPULATION CLASSES: (705 records, 705 failures, 0 quiet failures)
* RELIGION_BREAKDOWN: (235 records, 235 failures, 0 quiet failures)
* TRAITS: (10283 records, 698 failures, 0 quiet failures)
* UNIT_LIST: (3 records, 3 failures, 0 quiet failures)
* UNIT_RECORD_KEY: (3413 records, 3413 failures, 0 quiet failures)

And I can do the modification I want on the extracted XML files.

But when trying to repack the save files, it goes smooth without errors but generate a very light save that the game cannot load (7.75MB originally -> 1.13MB after repacking)

Any idea why that would fail like this?

I know the script wasn't specifically design to handle Warhammer 2 saves, but is it possible to make it work?

Tell me if you need more informations.

Here are the save files (original and repacked) : https://drive.google.com/file/d/1hN7hJvONB6kgWV2VVgwqZXJozYwWdfW7/view?usp=sharing

Thanks!

[taw]

I'll look into this at some point. I'm guessing it's probably related to compression method. I need to disable all "semantic" warnings, they only make sense for Empire and most people would use it with newer games.

[Sharptsa]

After a little bit more investigating, it seems that converting the compressed data back to esf fails silently somewhere.

./xml2esf export/compressed_data compressed_data.esf

It generates an esf file of only 16B.

I can try to look into it myself if you have an vague idea where the problem could be and point me in the right direction.

[taw]

I'm guessing it's something to do with compression, but I'd need to take a look.

ESF to XML to ESF to XML loop should be nearly perfect (except for a few things that have multiple possible encoding, and originals don't always use the shortest one iirc).

[taw]

I disabled printing semantic conversion errors (unless --semantic argument is given).

[taw]

I added proper XML error reporting, now this fails with:

XML Error: PCDATA invalid Char value 2

So at least that gets us closer to the issue.

[taw]

And going one step further, some strings in that ESF save file contain raw byte U+0002 - in <bad character here>missions_localised_title_wh2_dlc16_great_vortex_wef_sisters_dragon_stage_4.

XML 1.0 does not allow that. XML 1.1 would, but somehow that never became popular, and Nokogiri doesn't support that. So the clean fix of using XML 1.1 is not possible.

I'm pretty sure I had to do a workaround for this before - in UI converter. Whichever hack I used there needs to be ported to esfxml.

[taw]

OK, the workaround is there, it should work now. U+0002 converts to &#xE002; (as &#x0002is technically not XML 1.0), then it gets magically converted back toU+0002`.

You'll need to unpack it again from the start. Let me know if it worked for you, so we can close this issue.

[Sharptsa]

Works like a charm! 👍

Thank you for taking the time to look into it and fix it.