Repacking does not compress PKG files

[BigheadSMZ]

Hi there, big fan of your PKGTool, along with the one created by thesupersonic16. I am attempting to reach out to both of you by creating an issue here on GitHub. I have been working on texture packs for Trails of Cold Steel I & II for some time now, and use the PKGTools extensively. Unfortunately, replacing the phyre images with higher resolution images, along with the lack of compression when repacking PKG files causes the newly generated PKGs to grow exponentially in size.

So I was wondering if it was possible if you could implement recompressing the PKG files? Currently I have Durante attempting to build them for me with official tools, but this only going to be a one time deal. I have created tools to leverage the PKG tools, convert the phyre images to standard formats, reconvert them back to phyre images, then repack them. My texture tool automates the process completely.

The only thing missing from Cold Steel modding is recompressing PKG files. If you could somehow implement it, I'd be forever grateful and it would really help me in creating the best possible texture packs for these games. Thanks for your time!

[Sewer56]

Some Rambling

Worry not; I know who you are and I would rather say I'm surprised that after all this time; you're still going at this honestly. To get even get assistance from Durante himself - ohohoho.

It's honestly nice to hear that people are still trying to do stuff with Sen no Kiseki; or any of the other Kiseki games for the matter since I stopped poking around with the franchise a bit for a large slew of multiple reasons.

(That being perceived lack of community interest in modding of the games, reversing what others already did but not share, leaving the overall community in general, going back to carry Sonic Heroes modding on my shoulders; you name the reason).

A Response

That said, sure, I'll do it. Not long ago I was doing some cleanup of my older Github projects code quality that did not satisfy me anyway; and it seems I never got around to doing this one.

This should be fairly simple to implement for me; will be easier than the SEGA PRS Compressor/Decompressor I've already written prior.

The original reason as to why I never got around to writing a compressor for this format in the first place (aside from laziness) was quite simply because I didn't believe back then that it would be necessary:

• Although fast to decompress; the compression ratio on the format is low. It's just that the phyre files are very highly compressible.

• LZ4 outclasses the compression format in regards to its purpose in every single way: Even on Durante's end, it would be more efficient in terms of size and decompression speed if uncompressed PKGs were compressed with LZ4 and decompressed on the fly as they are loaded from disk (though something like this would result in a huge patch that may annoy people).

• When distributing mods; I figured that people would rather prefer small downloads over small disk space after extracting the download. For mod distribution, 7z provides much better compression ratio when compressing over the raw PKG than compressing a PKG with files inside already compressed.

Though I can see how you might be concerned given the overall size of a large texture project like this when the files sit decompressed on the hard drive.

Sure, I'll rewrite the project as a set of libraries, one for compression/decompression and the other for PKG; then a simple GUI for it.

Give me around a week for this as I'm binging on a Sen IV translation stream in my spare time in addition to normal life/study routines.

Appendix

By the way; since you're moving away from Special K it seems; you may be interested or get some use from my universal mod loader tech: Reloaded

I also let SS16 know that you opened an issue on his repo; since we're both friends on Discord and originate from the Sonic Hacking Community (though that said; when this and his PKG tool were made, coincidentally in the same time frame - we did not know each other); his Github notifications (or settings) are broken for some reason.

Have a nice day.

[BigheadSMZ]

Sounds amazing, thank you so much for the response and the effort! I've done a lot research into the phyre formats and managed to integrate converting them back and forth to normal formats as well as converting them to other formats. I do have one request, I would prefer if you could also create it so it's possible to run the tool via command line as it is now, as in this way it's easily integrated with my own toolset.

Using your PKG tool, I have an option to extract the PKGs into folders: https://i.imgur.com/6ruq9Vw.png

Convert the extracted phyre images into standard PNG, and save the headers to convert them back. https://i.imgur.com/leWWbx5.png

Convert the images back into phyre, with the specified format. https://i.imgur.com/tharFaY.png

And finally, repack the PKGs with your tool. https://i.imgur.com/vvoW9uC.png

I also have an image viewer where I can directly drag and drop the PKG or PHYRE images to view their contents. https://i.imgur.com/8t6HJrm.png

With all that said I'm not very good at programming or I would have attempted this myself, but I did manage to build your tool and had no issues with it (despite not knowing what I was doing :P). So if and when you finish this up, if you could also share a build that could save me some frustrations!

But yeah, the reason why compression is a concern at this point is because of what a mess this game is internally. The eye textures by frozenwings are spread across multiple PKG files, and currently when my entire pack is converted and repackaged, its sitting just above 6GB which is HUGE compared to the default sizes. I managed to get Durante to build the UI textures which saved quite a lot of space, but the way the build environment for the game is, there is too high of a risk and too much effort to pack the eye textures which is the worst offender when it comes to file size.

Having a public tool that can compress PKG files would be the penultimate addition to modding these games. Durante has very kind to fix many textures that become broken when increasing resolution to the default phyre images, so this opened the door to new possibilities and to avoid requiring Special K.

Also, just wondering where you're streaming the SenIV translations? And if you have done a SenIII translation stream? I'd binge watch them all!

[Sewer56]

Worry not; as with almost all everything I do nowadays I will provide simple code samples right in the readme. In library form; the code would integrate directly into your own program; without the need to remotely execute a precompiled version; much faster and easier for you.

If you'll require/need any extra help; I'll be happy to provide that in due time ;).

As for the Sen IV translations; I'm actually binging on Reiahl's streams: https://www.twitch.tv/reiahl. He's doing a very fine job translating as he's playing along and has also done the same with Sen III; I have no proficiency in Japanese myself

[BigheadSMZ]

Thanks, but like I said I'm not much of a programmer, and my "program" is actually just a really elaborate PowerShell script (you can find the code here: http://bhemuhelp.co.nf/other/ctt/CTT-PS_Code.html ) that leverages .NET Framework classes "System.Windows.Forms" and "System.Drawing". When it comes to writing in a "real" programming language, I'm a complete amateur. I can somewhat grasp C# code since it's similar to PowerShell, but when it comes to actually writing and compiling it, (derp).

When I started this script, I never intended to take it this far, and I only learned what I had to when I needed it to accomplish something specific. It's more of a "frontend" that leverages other programs through the command line, which is why command line support would be the easiest for me. There is some specific code that manipulates images in specific ways to get different results (such as I write my own DDS headers, combine pixel data in DDS textures for custom mipmaps, convert images to raw pixels+ further manipulation, etc.), but beyond images, I don't know too much of anything. :P

I know it's possible to import DLLs into PowerShell and use their internal functions, so I suppose that is also a valid route. I've never personally done it, but I'm sure if I had a need to I could figure it out. If you are writing the code in C#, its probably also possible to convert it to PowerShell (which I wouldn't expect you to do, I would attempt it if it's not too complex), as I have managed to convert C# functions in the past. That way I could just implement it straight into my script, although I think an external EXE would be the path of least resistance (at least, when it comes to me implementing your tool into my script). Speed isn't really too much of a concern to me, just the end result.

And thank you for the stream links, I'll probably end up binge watching them myself as I work on these packs. :)

[Sewer56]

Coming Soon (TM) [~2-3 hours unless I fall asleep];

Or add me on Steam if you want to test this for me while I write the documentation; I still need to mess around with this a bit.

[Sewer56]

Closing as compression has been implemented and tested by both me and the OP.