Fix Issue #3: explicity set the File Write Mode for files saved with …

[tomysshadow]

This is a fix for Issue #3 where scripts are hidden in files saved with Virtools 3.0+.

As you likely already know, Virtools distinguishes NMO files from VMO files by using a flag in the File Info at the start of the file. This flag, the File Write Mode, corresponds to the CK_FILE_WRITEMODE enum in CKEnums.h.

A value of 0x00000008 corresponds to the CKFILE_WHOLECOMPRESSED flag, which indicates the file is compressed. A value of 0x0000000C corresponds to a combination of the CKFILE_WHOLECOMPRESSED (0x00000008) and CKFILE_FORVIEWER (0x00000004) flags, the latter of which indicates the file is for viewers only (such as the 3D Life Player.) Modifying a VMO file to remove this flag makes it behave as an NMO file, which allows it to be opened in Virtools (with scripts hidden.)

However, in files saved with Virtools 3.0 and later, the File Write Mode is stored redundantly. In the Variable Manager, there is a setting for File Options/Compression. This setting is implemented by storing the File Write Mode. If the setting is set to Compressed and the file is exported for viewers, then the File Write Mode is stored as 0x0000000C, separately from the flag in the File Info.

As a result, when loading the file - although it does load - the global File Write Mode is set back to CKFILE_FORVIEWER, and the scripts are hidden, because their interface is not intended to be accessible within the 3D Life Player. The solution is to explicitly unset the CKFILE_FORVIEWER flag in the File Write Mode after loading the file.

Attached is a VMO file - modified into an NMO file - for Hammer: The Game, which may only be decoded with this fix applied.

hammer_thegame.zip

[yyc12345]

It seems reasonable to me. However, I have checked my IDA workspace and couldn't find any evidences to prove this. This PR may beyond my capacity. So, I suppose it need to be reviewed by the real author of this project before merging. I mean, instr3 or chirs241097. It will help us to confirm it if you like to provide more detail about your research steps, such as some important function call address or anything else.

[tomysshadow]

Certainly, I can provide more evidence. I will use CK2.dll from Virtools Dev 3.5 to prove this. You can also test for yourself with the NMO I provided to see for yourself that it only works with this change.

The problem occurs on the following line of our project [Parser.cpp:404] CKStateChunk *chunk = bb->Save(file, 0);

In IDA Hex-Rays, we can see the implementation of CKBehavior::Save at address 0x24018B96.

In this code, we can see it starts writing, and soon after, it gets the context of the file (using the member m_Context of CKBehavior.)

After this, there is an if statement which checks two conditions. The DLL import names pretty clearly spell out what happens next: if both conditions are met, it writes the Identifier 0x10 and a SubChunk. Afterward, regardless of the condition is met, it writes the Identifier 0x20 followed by more data. This is, pretty clearly, the Interface and Tail data. So we can see the Interface data is not always written: there are two conditions that must be met first.

The first condition is irrelevant; if we run this code in a debugger we can see it's already met. It corresponds to the IsInInterfaceMode property of CKContext, which as far as I can tell is always true in Virtools Dev.

The second condition is the problematic one: we can see it looks at a property of the context we just got, at *(BYTE*)(context + 0x2BC). It tests if it has the flag 0x04, and only if the property does not have that flag, the interface is written. Hmmm.

Furthermore, if we set a breakpoint at this location in a debugger and look at the value of this property in hammer_thegame.nmo, we can see that it is set to 0x0C. Hmmmmmm.

Spoiler alert: that member is the File Write Mode. And I know it's the File Write Mode because if we look at CKContext::SetFileWriteMode (at address 0x24007EEB in CK2.dll) we can see it takes in a context (as this) and writes to the same member (the one at 0x2BC.)

However, if we set a breakpoint on CKContext::SetFileWriteMode in a debugger, it's never actually called. So what does set the File Write Mode in this case? Well, try this: set a breakpoint in Custom.dll before the load occurs, and set Hardware Breakpoint on Write... (notice how the default value is 0x08 before CKContext::Load is called)

The first write just resets the value to 0x08, so it's irrelevant, but on the second write we end up here, at address 0x24032A51, and now the value is 0x0C. The function we've arrived in is called CKVariableManager::SetVariableValue. And notice how x32dbg points out, in the right column of the CPU view, ESP + 8 is set to "File Options/Compression?"

Well, that setting can be seen in Virtools! If we go to Editors > Variable Manager, we can see for ourselves the File Options/Compression option.

So we can guess that under the hood, this setting is implemented as a File Write Mode, the same as the kind in the File Info. And we know from the definition of CK_FILE_WRITEMODE in CKEnums.h that the value of 0x04, which causes the Interface data not to be written, is CKFILE_FORVIEWER.

typedef enum CK_FILE_WRITEMODE 
{
    CKFILE_UNCOMPRESSED        =0,  // Save data uncompressed
    CKFILE_CHUNKCOMPRESSED_OLD =1,  // Obsolete
    CKFILE_EXTERNALTEXTURES_OLD=2,  // Obsolete : use CKContext::SetGlobalImagesSaveOptions instead.
    CKFILE_FORVIEWER           =4,  // Don't save Interface Data within the file, the level won't be editable anymore in the interface
    CKFILE_WHOLECOMPRESSED     =8,  // Compress the whole file
} CK_FILE_WRITEMODE;

So it is obvious what happened: the File Write Mode is stored redundantly by the Variable Manager. The reason it's stored redundantly is to allow developers to turn the compression off through the Variable Manager interface, and this setting is then copied to the File Info when the file is saved. However, that means when exporting a VMO file that this setting's value is 0x0C, so upon re-importing, that changes the global File Write Mode to include flag CKFILE_FORVIEWER, and the Interface is never written in the call to CKBehavior::Save. This causes the Script Hidden upon importing the NMO normally, and the assert in BBDecoder (because the 0x10 identifier is missing.)

And just to confirm that is the case, we can extract a hammer_thegame.nmo with offzip, and right near the beginning of the second zlib compressed block (at 0x00072ef6 in the file,) we see the File Options/Compression variable as represented in the raw file, followed by its value, 0x0C.

So the solution is simple: to call SetFileWriteMode on the context ourselves after loading, before calling the Save method, restoring the Interface data to the buffer.

BTW, Virtools Dev 2.1 either doesn't have this option or doesn't save it this way. If you look in a file saved with Virtools Dev 2.1, the "File Options/Compression" string is simply not there, so the only File Write Mode value is in the File Info at the start of the file (but that one prevents the file from opening in Virtools outright, so this issue never occurs.)

[several edits made for clarity]

[yyc12345]

Nice work! Thank you for your contribution and research. I also have found the registration of variable "File Options/Compression" at 0x240086BD in CK2.dll. That instruction calls CKVariableManager::Bind. It actually binds this variable to the address of (DWORD*)context + 0xAF. It is exactly the same as your provided address (BYTE*)(context + 0x2BC). I will give approval for this PR. However, the merge may need to be done by instr3 or chirs241097.