Open obi0ne opened 1 year ago
note, the 24th above might be the mission name or something.
Hi, please see #21 for details on why XCOM2 isn't supported. It's not so much a problem with understanding the format as that it can't be unambiguously parsed without also having information from the script packages and that is a big job to implement.
I am not actively working on this project other than bugfixes, so that isn't something I have any plans to take on myself. I would definitely accept PRs that add full xcom2 support, though.
Currently I have no custom mods installed, so that's fine by me. Also I assume, have partial support is better than no support at all... I intend to invest some time with this & see if I can help with this.
I would appreciate if you could help or point me in the right direction as to the actual header & inlined blocks structures. How can I verify the actual structure, did you relay on some other format unreal engine reverse, or EXE reverse (IDA pro) ? or it was more trial & error, and see what happens when you modify it ?
Also, regarding the scripts mentioned on your reply, can you elaborate further on those ? where are those scripts & what is exactly require, should i decide to provide full support ?
Can anyone help identify the yellow fields ?
The scripts I mentioned are the unrealscript packages for the game and for any mods that have custom code. The big one is xcomgame.upk, and you can build it yourself by installing the xcom2 mod tools SDK. A big chunk of the save file is serialized versions of these unrealscript objects.
You don't necessarily need to understand all the fields in the save, as long as you know how big they are you can just read that many bytes and treat them as some unknown byte values that need to be put back in exactly the same way when regenerating the save. Even for EW many of the fields are marked 'unknown' and the values are just passed through as-is. It does make it harder to use if those bytes represent something that you might want to change in the save, though. I don't remember exactly if we even know the size of all these unknown structs to be able to treat them as opaque, it's been many years since I looked into this.
Some helpful tools for investigating unreal files are UE Explorer (https://github.com/UE-Explorer/UE-Explorer) which you can use to dump info about a script package and these can be helpful to try to interpret what the serialized form of the data will look like. There are also some useful wiki pages on nexusmods (https://wiki.nexusmods.com/index.php/Modding_XCOM:EU_2012) although that's EW specific XCOM2 uses the same engine and the formats are only slightly different.
thanks for reply with info. I will try to "play" with it, and see how far will I get.
weird, neither UE Explorer or UE viewer is able to view xcomgame.upk file.
Keeps getting errors:
Not sure if it's because somehow compressed, or a change in serialization format made during move to XCOM2 (i've read Unreal engine is 3.5, compared to version 3 on enemy unknown).
If i try to read enemy unknown files, it's goes little bit further but also "dies"
Are you able to view XCOM2 files on your pc ? Maybe something in my desktop environment ?
If you're using the ones from the shipping game you need to decompress them first, there's a decompress tool here: https://www.gildor.org/downloads
If you build the packages from the mod tools SDK they are uncompressed by default.
If you're using the ones from the shipping game you need to decompress them first, there's a decompress tool here: https://www.gildor.org/downloads
If you build the packages from the mod tools SDK they are uncompressed by default.
Thanks for that, you were right, it was compressed.
The tool however only seems to work on XComGame.upk file for Enemy Within & Enemy unknown & not for XCOM2 :( they must have modified serialization format during move to UE v3.5...
Will attempt to contact Gildor author & see if he can help with new's format...
Is the decompression failing or ue explorer failing to read the decompressed file? I was able to use ue explorer on the uncompressed ones I built myself.
If it's the decompression failing it may be that xcom2 uses a different compression algorithm. I do remember seeing this but thought it was only Chimera Squad that used the different compression type, but it might be XCOM2 too. There may be another version of the decompressor that supports the newer algorithm somewhere.
Is the decompression failing or ue explorer failing to read the decompressed file? I was able to use ue explorer on the uncompressed ones I built myself.
If it's the decompression failing it may be that xcom2 uses a different compression algorithm. I do remember seeing this but thought it was only Chimera Squad that used the different compression type, but it might be XCOM2 too. There may be another version of the decompressor that supports the newer algorithm somewhere.
decompression of the original (XCOM2 War of the chosen) XcomGame.upk fails.
Searching on the tool's authors (Gildor) forum, found some 2nd approval of that: https://www.gildor.org/smf/index.php/topic,7127.new.html#new
Attempted to bump it there, will see what he answers.
Seems that the decompressor detect the file as if it's UE1 (which obviously it is not) then it fails...
found another thread there with hope: https://www.gildor.org/smf/index.php/topic,2942.0.html try to understand if this specific build is decompressor or viewer only.
EDIT: working partially with this build: https://drive.google.com/file/d/110USn24A_rL2wytRu9B1-E_O2kYxFI8E/view
if i filter out texture, some texture there seem to crash it, not sure why
EDIT2: now crash after complaining cannot find SDL2.dll, will debug this one, not sure why not finding it...
EDIT3: got it decompressed with original decompressor using "-game=xcom2" tag.
I need some help.
You can see working extractor on code here: https://github.com/obi0ne/xcomsave/blob/61191112d2e5dea1f5059b62b0047a695b23c74a/xcomreader.cpp#L125 (on my fork).
That's output correct "output.dat" file with actor's table. However, i'm getting some issue's decoding actor table.
begin of file looks like:
First number is number of actors (845 on my case, so not modulu 2, maybe same as android).
But followed by two numbers: 120(dec) & 2738 (dec).
Do you have nay idea what can be those ?
Attaching zip of decoded output for my case.
output.zip
Ah, magic numbers! I took a quick look at a couple of saves I had handy and I also see the same 0x34d and 0x78 numbers in every single one so they are not likely to be an actor count. The 3rd number changes from save to save, and looks to me like it might be the actual actor count (the number tends to grow if you look at saves much later in a campaign, and the size of the table also seems to grow so that tracks). Opening up xcomgame.u in UE explorer and looking at the package I can see "Version: 845" and "Licensee version: 120", so that explains those values, although it's interesting that they get emitted as part of the actor table (then again, the actual file structure is more or less all guesswork too so they might be a header in front of the actor table, or what I call the actor table might not even be an actor table).
Just by looking at the format of the entries I can already see they are different from the ones in EU and EW but each entry does appear to have the same size other than the variable string length. My advice would be to just keep plugging away with it and recording the values without trying to figure out what they all are ahead of time. As long as you can decode the layout of the table without necessarily understanding what each field really is you may still be able to make progress.
yea, it's looks to be like: XCOM_STR | INT(4Bytes) | FF FF FF FF | INT (4Bytes) | FF FF FF FF
Weird thing is there seems to be 108 same entires, with following: XComGameState | 00 00 00 00 | FF FF FF FF | 2C 01 00 00 | FF FF FF FF
up till offset 0xFE0 Does this make any sense ??
108 missions ? having some place holders ?
Afterward i begin seeing some different entries, such as: XComGameStateContext_ArchiveHistory , XComGameState_Item etc.
I'll try to play with it further, thanks for your help.
Yeah that is a little weird. From what I remember from EW saves the number following the class name was the instance number for that object, and every one was different (maybe not entirely accurate, it's been a while). These ones being identical copies is a bit strange, and I don't know why that would be. When I looked at one save yesterday it did seem to change further down: if you scroll down to where the pattern starts to change and look at the last few entries they did seem to have increasing instance numbers. I didn't check to see where the pattern changes, though, or if it really does look to be an instance number.
XComGameState
is a class that represents information about what happens in the game, you'd have to read up about how XCOM2 models data to get a sense of how the history and game states work. Which reminds me of another reason why this tool is not a great fit for XCOM2: In EW the state of the game is generally represented by just single objects that are serialized into the save, so when you decode the save to json you can search for those particular interesting values and modify them, and generally there is only one copy.
In XCOM2 the save game encodes not just the current state but all game states, almost like a git repo that records the entire history of the whole campaign. So you won't have just one representation for a particular entity (like a soldier) but a whole bunch of those states for the entire history of that soldier. [There are some exceptions, generally the state transition history for a tactical mission is not kept in strategy because it's so big, but some parts are and the details of how this works actually changed between the original xcom2, the dlcs, and wotc]. So when you convert to json you'd see many different copies of that soldier, and only one of them is the current state that you'd want to change if you are trying to modify something. The flat json structure does not know anything about xcom's history data structures.
ok, here's what i got for this actors table. You are right, the first two numbers ar emagic & the third (2738) is number of items in list. Currently in the following structure:
XCOM_STR | INT(4Bytes) | FF FF FF FF | INT (4Bytes) | FF FF FF FF
decoded the entire actor table, attached here: actors_value.txt
the numbers don't make sense, likely to be representnig enums ?
here's for example snippet of values from table:
[0] ("XComGameStateHistory", -1, 0)
[1] ("XComGameState", 0, 300)
[2] ("XComGameState", 0, 300)
[3] ("XComGameState", 0, 300)
[4] ("XComGameState", 0, 300)
[5] ("XComGameState", 0, 300)
[6] ("XComGameState", 0, 300)
...
[119] ("XComGameStateContext_ArchiveHistory", 118, 216)
[120] ("XComGameState_Item", 118, 316)
[121] ("XComGameState_Unit", 118, 3400)
[122] ("XComGameState_Item", 118, 316)
[123] ("XComGameState_Unit", 118, 3400)
[124] ("XComGameState_Item", 118, 316)
[125] ("XComGameState_Unit", 118, 3400)
[126] ("XComGameState_Item", 118, 316)
...
[270] ("XComGameState_Unit", 118, 3400)
[271] ("XComGameState_Cheats", 118, 168)
[272] ("XComGameState_BattleData", 118, 1636)
[273] ("XComGameState_Item", 118, 316)
[274] ("XComGameState_Item", 118, 316)
[275] ("XComGameState_Item", 118, 316)
[276] ("XComGameState_Item", 118, 316)
[277] ("XComGameState_Item", 118, 316)
[278] ("XComGameState_MissionSite", 118, 1236)
[279] ("XComGameState_Reward", 118, 192)
[280] ("XComGameState_Unit", 118, 3400)
[281] ("XComGameState_Reward", 118, 192)
[282] ("XComGameState_Reward", 118, 192)
[283] ("XComGameState_HeadquartersProjectResearch", 118, 252)
[284] ("XComGameState_Item", 118, 316)
[285] ("XComGameState_Item", 118, 316)
[286] ("XComGameState_Item", 118, 316)
...
[291] ("XComGameState_HeadquartersProjectRecoverWill", 118, 256)
[292] ("XComGameState_HeadquartersProjectRecoverWill", 118, 256)
[293] ("XComGameState_HeadquartersProjectRecoverWill", 118, 256)
[294] ("XComGameState_HeadquartersProjectHealSoldier", 118, 256)
[295] ("XComGameState_HeadquartersProjectHealSoldier", 118, 256)
[296] ("XComGameState_Item", 118, 316)
[297] ("XComGameState_Item", 118, 316)
[298] ("XComGameState_Item", 118, 316)
[299] ("XComGameState_Item", 118, 316)
[300] ("XComGameState_Item", 118, 316)
[301] ("XComGameState_Item", 118, 316)
[302] ("XComGameState_Item", 118, 316)
[303] ("XComGameState_Unit", 118, 3400)
[304] ("XComGameState_XpManager", 118, 200)
[305] ("XComGameState_Item", 118, 316)
[306] ("XComGameState_Item", 118, 316)
[307] ("XComGameState_Unit", 118, 3400)
[334] ("XComGameState_Unit", 118, 3400)
[335] ("XComGameState_HeadquartersProjectBuildFacility", 118, 248)
[336] ("XComGameState_StaffSlot", 118, 216)
[337] ("XComGameState_StaffSlot", 118, 216)
[338] ("XComGameState_FacilityXCom", 118, 308)
[339] ("XComGameState_Item", 118, 316)
[340] ("XComGameState_Item", 118, 316)
[341] ("XComGameState_Item", 118, 316)
[342] ("XComGameState_Item", 118, 316)
[343] ("XComGameState_HeadquartersProjectHealSoldier", 118, 256)
[344] ("XComGameState_HeadquartersProjectHealSoldier", 118, 256)
[345] ("XComGameState_Item", 118, 316)
[346] ("XComGameState_Item", 118, 316)
...
[2281] ("XComGameState_RegionLink", 117, 284)
[2282] ("XComGameState_RegionLink", 117, 284)
[2283] ("XComGameState_RegionLink", 117, 284)
[2284] ("XComGameState_RegionLink", 117, 284)
[2285] ("XComGameState_Continent", 117, 260)
[2286] ("XComGameState_Continent", 117, 260)
[2287] ("XComGameState_Continent", 117, 260)
[2288] ("XComGameState_Continent", 117, 260)
[2289] ("XComGameState_Continent", 117, 260)
[2290] ("XComGameState_Continent", 117, 260)
[2291] ("XComGameState_Haven", 117, 400)
[2292] ("XComGameState_WorldRegion", 117, 832)
[2293] ("XComGameState_Haven", 117, 400)
[2294] ("XComGameState_WorldRegion", 117, 832)
[2295] ("XComGameState_Haven", 117, 400)
Alot of these "magic numbers" are somehow repeating, mostly 117, 118, 400, 300, 316.
unlike previous game versions, the number represented instance index. Here, it looks like they're serialized, as a collection, without any unique identifier. Unless someelse is pointing to them with some index i'm not aware of.
EDIT: to add to this,
i've yet to decode the part wafterwards, but peeking into output.dat file in hex & into UE Explorer,
it's begining of dumping each object public fields.
so afterwards I can see clearly:
History
and so on.
Only question come to mind is why some fields serialized & other don't ?
From explorer, eg:
Why for example they are not serializing "CurrentIndex" ?
the numbers don't make sense, likely to be representnig enums ?
I don't think they're enums. The string is a class name, the first number after that is relatively small, and then the next number after that is also relatively small. It also looks like the 2nd number is always the same for the same class name, but the first number changes.
My guess: The first number is the index of the parent object. The history is the root at index 0 (parent -1), the children of the history are a bunch of game states (all parent 0), and the other entries after that all link their parent back to one of the game states. This makes sense to me: the history contains XcomGameState
instances, and the XcomGameState
instances refer to the state objects (units, items, etc) that change in that change state.
The second number being fixed per class type made me suspect it might be a size, but that might not be correct (in particular I wouldn't expect all XComGameState_Unit
s to have the same size, since they have different names). But I could be wrong.
Why for example they are not serializing "CurrentIndex" ?
XComGameStateHistory is a native class (see the native(Core)
specifier near the top), which means it's partially implemented in the engine native code, it's not pure unrealscript. If you look at the class definition in the SDK it has a cpptext block that defines a Serialize
function, so I suspect this uses very custom serialization code rather than just dumping out the full object contents. Even for non-native classes that wouldn't have super custom serialization code the game history is often stored in terms of deltas rather than full objects: only the fields that changed are recorded. You can see this by playing around with the console command x2debughistory
and looking through how things are recorded (try it on a fresh campaign so history is fairly small, and do things like change soldier loadouts or promotions to see how they're recorded). The save game probably reflects this too to reduce save size and kind of "rebuild" the game state by loading the save and then replaying history to apply all the deltas to get back to the final position, rather than saving a full copy of the object at each state position.
looks like you are right, what throw me off, the index of parent isn't serialized sequential, the first object is 118, but looking indeed on list again, I do see all indexes running from 0 to 118, where for root, parent is -1, as you mentioned.
The second number could be size, even for different soldier name, if each class is serialaized with place holders after names. I will attempt to start decoding the 3rd part of the save game, and i'll see if this matches the second number or not.
Thanks again for your help.
question: between chunks, after i read first chunk, the chunk size was 932, but after finished reading properties for read propertries & diff with the offset before read, size was 924.
Now, looking on Hex editor, i see indeed those extra 8 bytes:
followed by 4 FF FF FF FF to separate to next object, probably.
However, this does not behave as previous xcom games, this does not seem to be padding, as half of it it non zeros.
First number is zero. second number is 0x1286 (4742dec).
Any thoughts on what should i do with this ? another magic number ?
EDIT: I think the 0x1286 is the props_size of the next chunk, and possible now the size is counted before the property size. as if i ignore the 8 and attempt to read next chunk it's crashes & does not match up.
EDIT2: first 1 bytes after are zero padding & I can continue to parse file. Currently debugging why at chunk index 121 I get exception when parsing properties...
EDIT3: found source of exception, I need prop builder type of "InterfaceProperty", which does not exists. Probably was not available on XCOM1 games.
EDIT4: seems UE explorer has no info about that type:
Ideas on how to decode this one ?
EDIT5:
it's crashes on decoding of property named "VisibilityInterface" of prop_type "InterfaceProperty",
in hex dump looks like this:
EDIT6: Current code can be seen on my fork, branch named "support_xcom2_wotc" : link: https://github.com/obi0ne/xcomsave/tree/support_xcom2_wotc
To debug, just go into xcomreader.cpp & conditional breakpoint inside function named "read_actors_state_table_xcom2()"
and place conditional breakpoint on code line 735:
with condition of iActorIdx == 121, then step into read_properties(), and condition on code line #453 with condition of:
strcmp(&prop_type[0],"InterfaceProperty") == 0
Did you add code to support InterfaceProperty
in read_properties
? I didn't see it in your branch. This looks to be fairly simple with a 4 byte property value but I'm not sure from only one example what the actual value represents (here 0x79). From the property name I'd guess this is perhaps an index into one of the UPK tables to locate an interface class, but that's just a guess.
Hi, thanks for reply. I've added some partial support, as you mentioned (made push to git), but it seems to be broken, as padding now don't align. Perhaps it's not just one number. Still debugging this one. Will try to progress more on weekend.
I have some problem decoding the binary file:
after the InterfaceProperty i get the None (which ends the loop for "read_properties").
and then there is gap of 59 (9881 - 9822) bytes, that, seems way too big,
that are expected to be padding, then it's reads 0x2 & throws.
Do you think there is a chance the "None" belongs to the InterfaceProperty ?, and then we're simply needs to browse further for the next proeprty(ies) ?
Next data isn't a normal propertries:
I have "Team Assignment", and afterwards something named "Move this turn" padding looks like 1 byte, if i try to see where next structure begin...
Both 'MovesThisTurn' and 'TeamAssignment' are special unit value names. UnitValues are stored on units in a special native map data structure mapping strings to floats:
var private native Map_Mirror UnitValues{TMap<FName, FUnitValue>};
The float value 2 is 0x40000000 and value 1 is 0x3f800000 so that is what those values are. This may be using special custom serialization logic since it's not a regular unrealscript array.
should I assume this is the missing 59 bytes & attempt to decode as some property field (UnitValues), or should i add special logic to treat it as "standalone objects" & ignore the expected size to previous object ?
I don't know, you'll probably have to examine several different saves to see if there is anything in common for these extra bytes to be able to process them correctly. The 2 value is interesting because it looks like an int 2, immediately followed by the length-prefixed string "TeamAssignment". But after the float value 0x40000000 there is a byte value 1 immediately followed by the next length-prefixed string for "MovesThisTurn". Those don't really line up, so maybe the 2 is not an int but either a byte or a short and indicates the number of elements in the map. Then either 1 or 2 bytes unknown, then the last 0 is not part of the int but is actually an index? All speculation. If these are 1-byte index values that precede the map entries then I'd expect the total size to also only be a byte, so that leaves 2 zeroes of unknown use after the size. It's hard to guess from only a single instance, it might be more useful if you can locate more entries of this kind in the save preferably with different unit value counts to see if you can find any patterns. From the SDK sources "TeamAssignment" seems to be used only for the lost, but player soldiers often have many different unit values on them so if you can find a soldier (try searching for the name of a known soldier and look around there) you might be able to get a better idea.
If you don't have the SDK installed i do recommend you install it so you can get all the base game unrealscript sources. Looking at those source files can give you a much better understanding of what you should expect to see in the save.
Just to let know i'm still on it - currently on hold till I get some spare time. for last few months had no time to play almost any game(s) due to "stress load" on real life work, working coding late nights. I hope to get more spare time towards middle of August to continue playing & investigate this further :)
Hi,
Found out your awsome work, thank you for that. I've been pooking around the binary in attempt to support XCOM2: Wa of the Chosen save data as well.
Figures, i'll share what I have, would appreciate if you guys could share more knowledge on how you've obtain the knowledge, and hoping togther we can support new format. And i'll up Pull Request with required changes.
Heres what's i've come by from what i've seen, regarding header info: 1) 4B : file version, for war of chosen it's 0x16 0x00 0x00 0x00 2) 4B : not sure, might be uncompressed size (different saved files had different values, eg. 3E 02 00 00) 3) 4B: unknown 4) 4B : unknown, seems always be 00 00 00 00 5) 4B: suspecting it to be game number 6) 4B: suspecting it to be save number 7) xcom_str: save description (with futurestic date) 8) xcom_str: time 9) xcom_str: map_command 10) 4B: suspecting it to be "tactical_save" (1 - tactical, 0 - Geo) 11) 4B: unknown ? 12) 4B (bool) : Auto Save ? 13) 4B: unknwon, seems 00 00 00 00 on my save games 14) xcom_str: language ("INT" for my case) 15) 16B: unknown ? 16) xcom_str: looks like some description ("UILibrary_MissionImages.Missions_Strategy_Generic", not sure what it's representing...) 17) xcom_str: again looks like description: "AUTOSAVE: Campgain 1, Mission 3" in my case, except it has no "game futuristic date" inside it (unlike field 7) 18) 4B: number of xcom_str ahead, has the number 5 on my end, and i have 5 xcom_strings 19) arr of xcom_str: sequence of xcom_str based on previous number, contains list of DLCs 20) 4B: number of XCOM installed PACKS 21) arr of xcom_str: sequence of xcom_str based on previous number, contains list of PACKs 22) 4B: unnown ? (contains 3 on my save) 23) 4B: unknown ? contains 1, possible some boolean ? 24) 8B : unknwon contains zeros on my side. 25): xcom_str : unknown ? has "NeutralizeFieldCommander" on my side. maybe invetory llisting ?
Any owner can shed light on how he obtained the other knowledge, i might help with this one...
cheers.