Cold Steel III PKG Files

[BigheadSMZ]

So the demo for CS3 just dropped, and already people are trying to extract the PKG files and textures. It was brought to my attention in the forum post below that CS3 has most likely changed the compression method used in the PKG file structure. https://forums.dolphin-emu.org/Thread-custom-texture-tool-ps-v45-0?pid=503925#pid503925

This is further supported by this tool's error message output: https://i.imgur.com/o9ImVX8.png

Now thesupersonic16's PKGTool does extract the files, but I have a hunch that it is not extracting them correctly. The extracted "asset_D3D11.xml" displays a list of the files when extracted from CS1/CS2 PKG files, but appears to be corrupted when extracted from CS3 PKG files. https://i.imgur.com/JcU3CUu.png

And when comparing the phyre image header data between the two versions, it appears it has either changed completely or there is missing/corrupt information (I am assuming the latter). https://i.imgur.com/nWwxGZc.png

I don't know if you still have interest in further developing this tool, but I figured I would give you a shout out and let you know that it isn't working with the latest game. And again I want to thank you for creating it in the first place as it has been endlessly helpful in developing my texture packs.

And as an addendum: This tool does not work with .NET Core 3.0+, which isn't really a problem since both 2.0 and 3.0 can be installed together. But I figured I would mention that too.

[Sewer56]

Falcom Stuff I actually didn't even know that CS3 was getting a PC port up until, actually just now. I kinda expected NISA to only localise the PS4 version and call it a day.

I've not really played any Falcom stuff outside of Tokyo Xanadu since I voluntarily departed from the community in 2017. I uh ... guess my Burning Heart craft hit its toll after playing a bunch of Falcom.

That said, if there's demand I totally wouldn't mind cracking this.

Regarding the Format:

• Given the screenshots you posted, it still looks like an LZ77 based compression format with Run Length Encoding (RLE).
• Should be fairly easy to crack. I reckon I could do it without much of an issue without the need to disassembly the game executable and finding the decompression function.
• I reckon it would take me 2-3 days to figure out the format and write a compressor + decompressor.

When? I've a bunch of University related stuff to work on right now in real life. I don't think I'd be able to look into this until around the start of April.

That said, if you want to take a look at it, it's probably very similar to the existing format. As such, you could use the current document https://github.com/Sewer56/Sen-no-Kiseki-PKG-Sharp/blob/master/docs/Compression-Explained.md as a starting guide to figuring out the format.

Any programmer worth their salt with some determination should be probably able to crack the compression here as it doesn't look too different.

[BigheadSMZ]

So I looked into this a few times and thought maybe I could figure it out, but it's definitely way over my head. I'm definitely not a programmer "worth my salt" lol, maybe a script kiddie at best and even that is probably a little too generous. I may give it another go eventually, but I literally have no programming knowledge outside of some PowerShell scripts, a teeny bit of C#, and a little bit when it comes to image formats.

But honestly if you don't end up finding the ambition to figure this out I can't say I would blame you and I personally don't have an immediate need to crack open the PKG files. My main interest lies in the phyre texture formats: I read somewhere they are using BC7 textures this time around. Since I managed to (mostly) figure out the headers for all the formats used in CS1 and CS2, I've been itching to see if it's possible to use BC7 textures in those games. The problem is, I have yet to see an example of a "texture.dds.phyre" that is actually compressed with BC7. All the formats require very specific flags in certain parts of the header, and everything I tried by trial and error ended up not working. Using Special-K it was possible to inject BC7 textures, so I know the game can display them properly, but I have no idea the engine can handle them by default. Call it curiosity more than anything.

The other thing I'm curious about is how CS3 will handle higher resolution textures. Will it accept them without issue? Or will it end up like the other two games? When I first figured out how to create phyre textures with higher resolution, it required special fixes by Durante to get them to scale correctly. A huge list of every texture was needed for both games, which unfortunately means that no new textures can ever be added to those lists unless someone somehow reversed the exe files which I don't even know if it is possible to do something like that. I know the lists must be in there, because Durante's first patch that made these work only replaced them.

Broken textures: https://imgur.com/a/ICNMcde

Example list for Durante that fixed them in CS2: https://www.dropbox.com/s/zgvbhpe9kpx35xo/ToCSII_SetSize_r4.txt?dl=0

Sorry I got a little off track there, I just felt like adding that little bit of background since there is nowhere else on the web I ever mentioned any of that stuff. At least now its there for "archival" purposes or something? But yeah, my only interest in new PKG format is potentially cracking phyre BC7 images (which may not even bear any fruit) and seeing if CS3 texture resolution can be increased (which I doubt it even can be). So even figuring out how to unpack these new PKG files may end up being in vain in the long run....

[Dlon93]

Hello! Any news regarding this? Many people are looking forward to editing the texture and other files from Trails of Cold Steel 3.

In any case, thanks a lot for your work so far. It's been very useful.

[CeruleanSky]

According to https://steamcommunity.com/games/991270/announcements/detail/1806446683762590792 at least for the PC versio it is ; "LZ4 ... with the addition of our engine-specific PKA format" that uses " sorting of data within the PKA" which they labeled as LZ4-PKA-S , with the S for sorted.

[Sewer56]

In that case, doesn't look that bad, could probably figure out everything beyond LZ4 with some trial and error.

I'm not sure when I'll get around doing this though, I'm working on a larger project at the moment: Netplay for Sonic Riders PC. (I also don't own CS3 but could probably just work with the demo)

[CeruleanSky]

The 18gb assets.pka file which probably stands for an archive of packages seems to add a slight extra layer to things, but the .pkg packages themselves seem the same as was listed in https://github.com/Sewer56/Sen-no-Kiseki-PKG-Sharp/blob/master/docs/PKG-File-Structure.md , just need to add a case for 04 for LZ4 and your chosen method of implementing LZ4 block decompression/compression.

I may be mistaken, but it seems if the compression type bitwise AND with 2 is true, such as if the 10b bits are set, which changes for example the 04 type to 06 there is a hash that is checked which is probably used in console versions. Also it seems the other compression types are 0x01 = lzss , 0x08 = lzma , 0x10 = Zstd and 0x00 for none, but should be verified as I may of mixed them up.

The most important change for people planning to repackage the files and make repackagers, that they are sorted in the same way and preferably compressed to a similar level as when they were originally packed as according to the announcement linked above, that has a huge performance impact. This probably mostly applies to the 18gb assets.pka file.

I'm sure when dealing with GB using D would be a much better language then interpreted python, but here is a small toy program to decompress and extract a package that people can play with until you have some time to update things and support the PKA type.. It doesn't rename files, so if packages extract the same file name, for example "asset_D3D11.xml" is common inside the data\asset packages, it will just overwrite it with the last one that was extracted. If somebody is really impatient they can rewrite things to reverse the operations and recreate the pkg.

#decompress most packages in D:\The Legend of Heroes Trails of Cold Steel III\data\asset\D3D11_us
#Can use Noesis and https://zenhax.com/viewtopic.php?p=33383#p33383 plugin to view
from lz4.block import decompress
import argparse
from struct import unpack
import glob

parser = argparse.ArgumentParser()
parser.add_argument("-v","--verbose", help="be more verbose", action="store_true")
parser.add_argument("files", type=str, nargs="+", help="file(s) to decompress")

args = parser.parse_args()
files = []
for x in args.files:
    files.extend( glob.glob(x))

for file in files:
    with open(file,'rb') as f:
        pkg = f.read()
        try:
            num_of_files = unpack('L',pkg[0x4:0x8])[0]
            for file_num in range(num_of_files):
                i = 0x8+file_num*0x50
                filename = pkg[i:i+pkg[i:].find(0)]
                unpacked_length=unpack('L',pkg[i+0x40:i+0x40+0x4])[0]
                block_size = unpack('L',pkg[i+0x44:i+0x44+0x4])[0]
                block_offset = unpack('L',pkg[i+0x48:i+0x48+0x4])[0]
                comp_type = pkg[i+0x4C]
                if comp_type == 4:
                    z = decompress(pkg[block_offset:block_offset+block_size],uncompressed_size=unpacked_length)
                elif comp_type == 0:
                    z = pkg[block_offset:block_offset+block_size]
                else: 
                    print("Compression type " + hex(comp_type) + " for filename " + str(filename) + " is unsupported in pkg:" + file)
                with open(filename,'wb') as o:
                    if args.verbose: print(filename)
                    o.write(z)

        except:
            print("Something went wrong such as unsupported file type or variation" + str(args.files))

[CeruleanSky]

I continued to play around with PKA and decided to modify your PKG doc and made one for the PKA in #4 to aid your(or anybody else's) future work.

While it will likely work as is, the only testing I did was manually carving a few spots and verify their SHA256 sums.

To summarize the format: PKA Header with sig & pkg entry numbers repeating package entries --pkg name & number of files ----file entry (name and SHA256 of file) ----... ----file entry --... --pkg name & number of files ----file entry ----... compressionEntries that describe the location and size of the data that needs to be decompressed and the compression type, It also has a SHA256 which matches the file entries the data itself

[CeruleanSky]

And my last planned comment, an extractor for pka files https://gist.github.com/CeruleanSky/c34400510c2b2484d179548f8db4ba41

In case I left any ambiguity this should help clear things up. Have fun everybody and hopefully a repacker and rearchiver will be around sooner or later.

Edit: There is a chance do to similarities and the engine supporting multiple types of compression that you can repackage in the older format and throw them in D3D11_US , didn't try it though.