MS2 - Importing, Exporting, or injecting certain JWE2 files in CustomBuildings

[hexabits]

Describe the bug

Varied success in doing a round-trip of files in Content0\Buildings\CustomBuildings (only buildings I've tried so far).
For context: https://discord.com/channels/680909673607463131/906797640124080190/978417857866137640

To Reproduce Unpack, Import, Export, Inject files:

Content0\Buildings\CustomBuildings\Footprints\GST_CustomFootprint_Large\gst_customfootprintlarge.ms2 Content0\Buildings\CustomBuildings\Footprints\GST_CustomFootprint_Medium\gst_customfootprintmedium.ms2 Content0\Buildings\CustomBuildings\Footprints\GST_CustomFootprint_Small\gst_customfootprintsmall.ms2

These files do not successfully import with error Bone info 0 failed.

Example Log: blender_plugin.log

I can however at least unpack and import these files successfully using an older version circa 2ced7f50 as linked here by ViraL Cyclopes after posting about my issues.

It's so convoluted my memory may be incorrect at this stage, but iirc, if I take the unpacked MS2 using the old revision, then import that with the plugin from the same revision, then export that, it will change the header from 20 0C to 18 0C, and then the newest revision plugin will at least import this old-new-new re-exported MS2 into Blender.

At this point, the re-exported MS2 will have truncated strings on import, e.g. customfootprint_large -> omfootprint_large as the scene name, or customfootprint_large_armature -> omfootprint_large_armature for the armature name. So then I hex-edited the re-exported MS2 to add 00 00 before the string table, but did not add the FF FF the 20 0C header files have, and this results in a file I can import into Blender without truncated strings. (Edit: Actually it was truncating the full 8 bytes, from gst_customfootprint_large)

However, OVL Tool doesn't get the memo, and it believes the mdl2 filename to be customfootprint_large_.mdl2 instead of gst_customfootprint_large_.mdl2 after I inject the new MS2 and save the OVL. There were no errors and a warning or two during this last step, but the OVL it creates breaks both OVL Tool and the game.

So at this stage I then hex edited a second time, adding the FF FF in front of the 00 00 in the header, and then I finally get an MS2 file that I can round-trip in the latest revision.

I also edited MS2.py at some point mid-process to change assert first_materials to if first_materials: and indenting the remaining scope. I believe a core issue may be that these MS2 files just have an armature and no materials.

---

Content0\Buildings\CustomBuildings\BLDG_Custom_Buildings_Shared\ent_physicsnavigation.ms2

Cannot export file. blender_plugin.log

---

Content0\Buildings\CustomBuildings\Footprints\GST_CustomFootprint_Attraction\gst_customfootprintattraction.ms2

This I can round trip so long as I remove the ms2, then drag it in. I do not know if dragging it without first removing it (or using the Inject file menu option) is meant to be supported, so I'm still bringing it up. I get no warnings or errors (in log or UI) when dragging the file and saving, but then upon reopening the OVL, it is completely broken and also crashes the game.

Expected behavior Files to import, export, inject without error. Game to not crash when there are 0 errors reported during a basic round-trip process.

Version Info: Blender 3.1.2 INFO | Running python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)]

For the files that can import: INFO | Running cobra-tools 2.3.8, 46682543 - Sat May 21 23:36:58 2022 +0200

For the files could not even import in version above: INFO | Running cobra-tools 2.3.8, 2ced7f50 - Sat Mar 19 08:50:48 2022 -0700

Log Files:

• See above

[hexabits]

OT: Also it was wild opening the data definition files (e.g. ovl_base.xml) and seeing my nifxml syntax being used. 😅 I hadn't a clue Cobra Tools was using it.

[hexabits]

At this point, the re-exported MS2 will have truncated strings on import

I've simplified the process of getting working files by exporting with the old revision, then immediately adding FF FF FF FF 00 00 00 00 to the header, then opening it in the latest plugin revision and re-exporting that.

[hexabits]

Footprints.zip

Here are the requested files.

[hexabits]

@HENDRIX-ZT2

Re: the recent commits, I still get an exception when dragging ent_physics_navigation_.ms2 to the OVL.

When not removing the original first:

ERROR | An exception occurred while injecting ent_physics_navigation_.ms2
ERROR | 'NoneType' object has no attribute 'struct_ptr'
Traceback (most recent call last):
  File "C:\Users\Jon\Desktop\cobra-tools-master\generated\formats\ovl\__init__.py", line 735, in inject
    file.loader.load(file_path)
  File "C:\Users\Jon\Desktop\cobra-tools-master\modules\formats\MS2.py", line 211, in load
    self.create()
  File "C:\Users\Jon\Desktop\cobra-tools-master\modules\formats\MS2.py", line 125, in create
    self.ptr_relative(model_info.first_materials.frag.struct_ptr, first_materials.struct_ptr)
AttributeError: 'NoneType' object has no attribute 'struct_ptr'

When removing the original first and resaving the OVL:

WARNING | Could not create: ent_physics_navigation_.ms2
Traceback (most recent call last):
  File "C:\Users\Jon\Desktop\cobra-tools-master\generated\formats\ovl\__init__.py", line 775, in create_file
    file_entry.loader.create()
  File "C:\Users\Jon\Desktop\cobra-tools-master\modules\formats\MS2.py", line 125, in create
    self.ptr_relative(model_info.first_materials.frag.struct_ptr, first_materials.struct_ptr)
AttributeError: 'NoneType' object has no attribute 'struct_ptr'
INFO | Updating hashes for STATIC
INFO | Updating pool names
ERROR | No entry found for pool 3!
WARNING | MemPool [Size: 0, Address: 0]
        * zero_1 = 0
        * size = 0
        * offset = 0
        * zero_2 = 0
        * file_hash = 0
        * disney_zero = 0
        * num_files = 0
        * ext_hash = 0
        * zero_3 = 0
 has no name assigned to it, cannot assign proper ID
Traceback (most recent call last):
  File "C:\Users\Jon\Desktop\cobra-tools-master\ovl_tool_gui.py", line 506, in inject_files
    self.ovl_data.add_files(foreign_files)
  File "C:\Users\Jon\Desktop\cobra-tools-master\generated\formats\ovl\__init__.py", line 799, in add_files
    self.update_hashes()
  File "C:\Users\Jon\Desktop\cobra-tools-master\generated\formats\ovl\__init__.py", line 1233, in update_hashes
    archive.content.update_hashes(file_name_lut)
  File "C:\Users\Jon\Desktop\cobra-tools-master\generated\formats\ovl\__init__.py", line 160, in update_hashes
    raise KeyError(f"Can't find '{entry.name}' [{entry.__class__.__name__}] in name LUT")
KeyError: "Can't find 'ent_physics_navigation_.ms2' [RootEntry] in name LUT"

It does it with both the freshly unpacked and the imported/exported versions.

Edit:

Actually I get an exception just when opening the OVL that I didn't notice. Nevermind on this, the OVS file got broken by a previous attempt and I didn't realize.

There is the vanilla OVL, the stuff I extracted, and the MS2 I exported. I guess the dragging doesn't work because of the archive breaking, not the MS2.

Edit 2: Sorry actually maybe the OVS file just got broken by a previous attempt at injection/etc. I'll try to look into it more. I just noticed the date on the OVS file is wrong and might not be vanilla anymore.

BLDG_Custom_Buildings_Shared.zip

Edit 3: I still get the issues with dragging ent_physics_navigation_.ms2 to the OVL whether I first remove it from the OVL or not. I updated the ZIP to make sure the OVS was vanilla.

[HENDRIX-ZT2]

Hey Jon, great to have you here. Nifxml syntax has proven to be really useful and expandable.

I've pushed some more fixes, now ent_physics_navigation_.ms2 is properly added to the OVL. However, the pointers to empty data are not pushed to the end of their mempool to actually get 0 bytes assigned on OVL loading. I don't think that causes any trouble in game or for OVL loading, though.

[hexabits]

This is good to close.