UNOBB: Add support for KotOR2 Android obb archives

[DrMcCoy]

KotOR2 for Android from Aspyr is out. Our unobb tool doesn't handle those obb data files.

To recap: obb files are big data files used by Android apps. They don't have a set format; what's inside them is handled by the apk itself.

For the first KotOR game, Aspyr just used normal PKZIP files. For Jade Empire, they made them a virtual filesystem, with zlib compressed blocks of 4096 bytes.

Looking at the files in KotOR2, they seem to be still that virtual filesystem Aspyr introduced with Jade Empire. With a key difference: some files are not zlib compressed, but uncompressed. Might have been that Jade Empire already supports that too in principle, but that just no file there was uncompressed.

This has two implications for our unobb reader:

• The file identification fails, because we depend on there being a zlib header (78 9C)
• The compressed resource index list can't be found, since we depend on there being a compressed chunk beforehand

Manually ripping a resource list block from the KotOR2 obb files yields a zlib compressed resource list of the usual format.

This means, I need to revise our resource index list searching for obb files. Instead of searching backwards for a zlib header and then checking that the previous block metadata looks okay, why not just go by the metadata right at the end of the file (the last 16 bytes)? That should contain the offset of the resource list. Throw on inconsistent data there, and we don't need to check for the a zlib header at the start either.

Another thing I need to fix: check if compressed size == uncompressed size. If so, it's an uncompressed file. Read it directly instead of running it through zlib. Open question: what if the file is larger than 4096 bytes? Is there a block split somewhere or can we just read the whole thing as is?

There's also liblzma in the apk. At first I suspected Aspyr uses that in this obb format version, but I might be wrong. Open question: are all files in the obb either uncompressed or zlib? Or is there lzma too?

In either case, that looks like a nice little puzzle to solve/fix over the holidays for me :)

[Dhamp1r]

Thanks for working on this. But will it be possible in the future to import files back to .obb (all or some)?. This would be very useful, otherwise unpacking is practically meaningless, because the files on the iOS are identical and are not packed to a virtual file system.

[DrMcCoy]

Well, the problem is that I'm not yet sure about all of the features of the format. Each compressed block has 16 byte of metadata (not there on uncompressed blocks) that I don't know the contents of (well, the first 4 bytes are the offset to the start of the block, that I know).

I know enough to extract files, but I'm not sure I can create files yet. That needs a lot of experimenting. But I take pull requests, so feel free to write an obb tool, test it to heck and back and throw us a PR! :)

otherwise unpacking is practically meaningless, because the files on the iOS are identical and are not packed to a virtual file system

Except that you need an iOS device to buy the game on iOS and transfer the files away. I haven't even looked at what KotOR2 and Jade Empire iOS do, to be honest.

[DrMcCoy]

As for the 16 byte of metadata, this could be:

• Offset to the block, as an uint64_t
• Size of the uncompressed block, as an uint64_t

All Aspyr OBB files I've seen are < 2GB, and Android itself only allows files up to 4GB. In all blocks I've inspected, those upper 32 bits are 0, which fits. On the other hand, it might actually be some flags or some other information.

[DrMcCoy]

I made some changed to my local checkout, using the last 16 bytes at the end of the file to find the offset and uncompressed size of the resource index. With that, I'm able to correctly extract the two KotOR2 obb files.

...However, that doesn't match up with Jade Empire. The metadata in the obb files in Jade Empire behaves differently. First of all, the last 16 bytes there to not go to the offset of the start of the resource index. In fact, the values doesn't fit anything; it might just not be a metadatablock at all, but the end of the zlib stream.

Looking at the 16 bytes preceeding the resource index in Jade Empire, the first 4 bytes are an offset th obb file, and that is the start of a block, since I always see the zlib header (78 9C) there. But the 16 bytes preceeding that don't match at all, so that might not be the first block of the file at all.

Not sure what to make of this now. I can extract both types (though I'm unsure anything I do is strictly correct and would work with potential other files, like in new updates), but the path of one doesn't work with the other. I could just add a command line switch to make it user selectable and call it a day. Or try to rig some autodection heurstics. Or both, autodection + manual override. Hmmpf.

I'm not sure if this failure is because they changed the format (and if so, is there even a version marker anywhere in the file?) or if I'm just taking shortcuts in the code that fail to work for all possible valid Aspyr Obb VFS files.

[DrMcCoy]

Oh, also, according to my comments in unobb.cpp, there's also a list of (all?) blocks in the obb file, or at least in Jade Empire's version of it. I don't even remember what I found there, but that would be another thing one needs to understand first before an obb writer is feasible.

[DrMcCoy]

Also found a bug in our Archive extractor code. It normalizes file names through the FileType map but doesn't catch the case of an unknown FileType, which leads to files with an empty extension being created (and in the case of the .DS_Store file Aspyr accidentally packed too, even an extraction failure). That's obviously wrong, and possible impacts the existing Jade Empire unobb as well.

EDIT: Even worse, in the case of files with the same name but different extension, like vertex + fragment shader, this leads in one file overwriting another outright.

[shadowslasher410]

I made some changed to my local checkout, using the last 16 bytes at the end of the file to find the offset and uncompressed size of the resource index. With that, I'm able to correctly extract the two KotOR2 obb files.

I don't suppose you would be willing to push your local commit to your fork? I want to get into modding KOTOR 2, but to do that I need the contents of the OBB files...

[DrMcCoy]

It's too messy to really push it into a branch, but here's a patch: https://gist.github.com/DrMcCoy/39d81320fda6ebffce1830f654bb2375

Note that this is makes the unobb tool work with the new KotOR2 OBB files, but will break the tool for the old style Jade Empire OBB files. I hadn't really found any time to properly unify the code to work with both, unfortunately.

And also, I of course give no warranties. If you break something, you get to keep all the pieces ;)

And also also, my previous comment still stands, that our archive extractor code (in src/archives/util.cpp, though fixing will ripple upwards) is broken. I had started fixing that by making all the archives classes, at least in xoreos-tools here, keep the full path, if they know it (some don't). But it looks like I've inadvertently thrown those changes away, because I can't find them anymore in my local repo.

Feel free to fix xoreos-tools up in those respects and send a PR our way. Or alternatively, just take the code more like a study on how the new format works, and write your own tool to extract it.

[bleau2000]

i cant install your patch itried multiple method but it does nothing the only thing that happen is that the command start but only to stop immediatly the only thing that appear in the window is a _ can you help me please

[DrMcCoy]

It's a patch you apply to the source code before compiling xoreos-tools, you are aware of that, right?

You need to apply it with patch(1) and then compile xoreos-tools. I.e. this is not really something a user is supposed to do, this is a developer's workflow.

If you are a dev, try these steps and if they fail, describe in detail what you did and how it failed, including any error messages.

If you just want to use xoreos-tools to extract files and have no experience compiling C++, then I'm afraid this won't help you. You need to wait until I (or someone else) find the time to clean up this patch so that it works alongside the old format, and then I can push a new release.

[bleau2000]

thanks for the quick answer i did not know that

[bleau2000]

I just try applying the patch to the source code but It does the same thing it doesn't give me error the only thing appearing is a "_" by the way im on windows i have tried with gnuwin32 and the official linux on windows 10

[bleau2000]

Do you know any tool i can use to extract kotor 2 obb

[bleau2000]

Beacuse your tool is the only one i find

[DrMcCoy]

I'm sorry, but I can't really give you any technical support here on applying a patch. This is neither the correct venue for that, nor do I have the time.

I don't know of any other program that can extract those archives, but I haven't looked since I last worked at it. I'm pretty sure I was the first who figured out the format (mostly), but others might have found out more since then. You need to research that yourself.

In either case, this is all mostly off-topic for this issue. Good day.

[bleau2000]

Thanks anyway

[bleau2000]

i have tried with patching with visual studio but it says that your patch is corrupt at line 187 i have use an extension called git patch utility by maketes it does the same thing in powershell

[DrMcCoy]

Ah, yes, it seems the final newline character was cut off. Get the patch again, I fixed the gist.

[bleau2000]

thanks

[DrMcCoy]

Oh, also, my comment in https://github.com/xoreos/xoreos-tools/issues/69#issuecomment-749209510 still applies, so the extraction doesn't really work correctly right now even with the patch.

The correct fix would be to allow extraction of files with unknown file extensions. A workaround, however, is to add the new file extensions to our list. I've updated the patch again to do that. If you revert your old changes, get the patch again and apply it again, that should, after compilation, then give you an unobb binary that can successfully extract the KotOR2 obb files.

Well, at least the obb files beginning main.205 and patch.14, which was the state of the game at release day. Might be that there were later updates to the game date files with files with more unknown extensions. You'll see this when you got files without an extension after extracting the archives. if that happens, you might even have some files overwriting another, when you've got files with the same name and different unknown extension. If you check the contents with "unobb l", and you see two files named the same with no extension, then that's the case. I'd need to reinstall the game on my phone and recopy the files to check.

[bleau2000]

also do ineed to repack the obb after modding or can i put the file in data folder if i need to repack how can ido that

[DrMcCoy]

If you want to mod the game, you would need to repack the obb. We do not have an obb packer, though, just an unpacker, so you can't use xoreos-tools for that.

You might be able to put modified files into the override directory. I don't know where that is on Android, though.

[LewsTherinTelescope]

Mod directory is Android/data/com.aspyr.swkotorii/files/dlc/mods_english (can replace english with another language, but most mods tend to be English, I think). Overriding files works similarly to the desktop, you need to put them in the correct subfolders (override, movies, etc) but they should work. Exception to that is sound files, which use a different format from the desktop version and so don't play (think the TSLRCM people might've figured out what it is, but not sure).

[DrMcCoy]

That's good to know, thanks!

[LewsTherinTelescope]

Oh, another exception to that: movie files seem to crash the game (presumably they use a different format). So don't include those, I guess. (I believe they work fine for K1, though?)

[DrMcCoy]

Hmm, no, they don't seem to be a different format as such...

Taking permov02.bik for example, ffprobe returns, for both

  Stream #0:0[0x0]: Video: binkvideo (BIKi / 0x694B4942), yuv420p(tv), 1600x680, 30 fps, 30 tbr, 30 tbn, 30 tbc
  Stream #0:1[0x0]: Audio: binkaudio_dct, 48000 Hz, stereo, fltp

I.e. still Bink (1) and the same revision of the Bink format, as well as same resolution, fps, etc.

However, the Android file is much smaller, 6.9MB vs. 20MB. So higher compressed, maybe different bitrate setting given to the Bink encoder.

No idea why that might crash the game, though. Possible that in additon to changing the bitrate, they disabled an encoding feature that is missing or buggily implemented in the Android port of the Bink library.

[MarcinT93]

Oh, also, my comment in #69 (comment) still applies, so the extraction doesn't really work correctly right now even with the patch.

The correct fix would be to allow extraction of files with unknown file extensions. A workaround, however, is to add the new file extensions to our list. I've updated the patch again to do that. If you revert your old changes, get the patch again and apply it again, that should, after compilation, then give you an unobb binary that can successfully extract the KotOR2 obb files.

Well, at least the obb files beginning main.205 and patch.14, which was the state of the game at release day. Might be that there were later updates to the game date files with files with more unknown extensions. You'll see this when you got files without an extension after extracting the archives. if that happens, you might even have some files overwriting another, when you've got files with the same name and different unknown extension. If you check the contents with "unobb l", and you see two files named the same with no extension, then that's the case. I'd need to reinstall the game on my phone and recopy the files to check.

hello thank you for your work on this great toolset. i have got question and request to you. the request is - could you please share exe version of patched unobb (as optional download somewhere maybe) for windows? i have tried to compile everything on windows (using msys2 and your instruction) but i have never done such things before so i probably fu***d things up. after compilling (ussing auto) it misses some dlls (like boostfilesystem-my and zlib1). i have copied these from Build version of toolset (with name changing as for example -mt version was not included), downloadd zlib1 from web but still its not working (missing function in dll). at beginning i have assumed that after compilation i will get all exes and dll :( so its possible i have compiled this wrongly. your instruction is generally very good but for someone aho Has at least Basic experience with such things. i will be working to compile this still but... the questions - if there is any progress in fixing these 69 issue?

[DrMcCoy]

could you please share exe version of patched unobb (as optional download somewhere maybe) for windows

I'm a bit busy at the moment, and I've broken my mingw cross-compiler recently unfortunately, but I'll see if I can do that this week

after compilling (ussing auto)

Nice, first time doing something like that and you've managed to compile it, that's already quite a big step! That should be the hardest part done and everything to run the binary should be there in msys2.

i have copied these from Build version of toolset (with name changing as for example -mt version was not included), downloadd zlib1 from web but still its not working (missing function in dll)

Yeah, just mixing and matching DLLs like that won't work, because they've been probably build with different compilers, compiler versions or for different libcs. You should have the DLLs you've build against in msys2: they should be somewhere in /usr/lib/ or /usr/bin/, or possibly /lib/, I think.

If you can't find them, you can use the tool find to, well, find files.

find /usr/ -iname "*.dll"

should list all .dll files within the /usr/ directory (and subdirectories) of msys2.

i have assumed that after compilation i will get all exes and dll

you're absolutely right there, just that msys2 simulates a Linux-like environment, and when you installed the packages for the libraries, it put the DLLs into the system paths (comparable to how when installing something on Windows, the DLL might put copied into C:\Windows\System\). You just have to copy them into the directory of the EXE when running them in Windows.

It might be that just running the tools from within msys2, i.e. doing

./src/unobb.exe

in the xoreos-tools build directory could work, but I'm not sure. I'm not using Windows often, sorry.

if there is any progress in fixing these 69 issue?

Unfortunately not, no. I've been pretty busy with RL and wage labour and all that jazz. I've barely touched most of my projects for a good while now :(

[DrMcCoy]

@MarcinT93 Okay, I'm a bit late, but I compiled a modified unobb version for you for Windows: xoreos-tools-0.0.6-kotor2_android-win32.zip