SWBFSpy
commented
2 months ago what do you guys use to repack games
Quote from: Masquerade
It's too complicated to explain in detail. Basically, they use a program called FreeArc combined with other (de-)compression tools by several coders (e.g. Razor12911). The used tools depend on the game. Most repackers do it like this nowadays.
FreeArc is an archiving utility just like WinRAR and 7Zip.
However, FreeArc has the advantage of allowing you to include external compression methods to what FreeArc provides as standard. This effectively resigns FreeArc to being a simple host process for multiple different compression tools that we chain together.
There are three main types of external tools - precompressors, compressors and decryption tools.
Precompressors are responsible for decompressing input data. Most games nowadays use some kind of compression algorithm, deflate/zlib is arguably most commonly used but LZ4 is becoming more popular especially in Unity games. Furthermore, the Oodle family by RAD Game Tools is quickly becoming more mainstream through Unreal Engine. Even Ubisoft changed from LZO to Oodle in their Anvil Next 2.0 game engine (think Assassins Creed, Immortals Fenyx Rising and Ghost Recon Breakpoint).
One of the basic principles of data compression is that once data is compressed, you aren't going to be able to compress it optimally unless you decompress the data first. This is how the precompressor does its job. By decompressing the data, we then can apply more effective compression algorithms which is how we get ultra-small repacks. Such an example of a precompressor is XTool by our good friend Razor12911. XTool supports many different algorithms like the aforementioned Oodle, Zlib, Deflate, LZ4 and more. This wide range of compatibility means that XTool is the most commonly used precompressor as well as it is quick and multithreaded.
Other precompressors exist too for more specific purposes. The coder ProFrager wrote UELR (Unreal Engine LZO Recompressor) for Unreal Engine 3 games. Edison007 wrote AFR (Anvil Forge Recompressor) for Ubisoft older Scimitar/Anvil games that used LZO, not Oodle.
Moving onto decryption tools, they are very important. Not all games are encrypted, but when they are it's a huge pain. Encrypted data cannot be compressed so it's important that we decrypt that data in order to make the repack. Sometimes, the decryption tool can fit into the FreeArc compression chain so the installed game files will be encrypted again perfectly. An example of this is Razor's Death Stranding tool. Other times, all we can do is decrypt the files in the case of Monster Hunter Stories 2. The game still ran fine without encryption which is what made the repack possible. Another example of encryoted games are TellTale games. All of their games are encrypted with different Blowfish keys. Shegorat wrote TTGD which is a combination of decryption and precompression tool for TellTale games.
Finally, main compressors are often the longest process of the FreeArc compression chain. SREP+LOLZ is the configuration of choice if you want the smallest size. SREP was written by Bulat Ziganshin, the creator of FreeArc and is responsible for removing duplicate data. Why bother compressing dupe data if SREP can use a huge LZ77 dictionary to remove and store those dupes, drastically cutting the size of data sent to lolz. For example, two thirds of precompressed data from Just Cause 4 could be removed by SREP. Just Cause 4 used a combination of three Oodle algorithms - Kraken, Mermaid and Selkie. LOLZ is a program made by ProFrager and is widely regarded as the best compression tool for game data. It has built in raw and header based stream detection for optimal compression of input data as well as support for large dictionaries and multithreading. LOLZ does have a function called LDMF (Long Distance Match Finder) which does a similar job to SREP - however I rarely use this so do not wish to comment on it. LOLZ compression is SLOW. Personally, my LOLZ configuration runs at between 1300kb/s to 1500kb/s depending on input data. You can speed it up slightly by turning detections off, but at that point you might as well just use a different algo. One overlooked but huge advantage of LOLZ is its ultra fast decompression speed. This reduces your repack install times! Because LOLZ compressor is slow for compression, some choose to use other algorithms such as LZTURBO or LZMA(2) which are faster but do not compress as well.
So here I have explained a little bit about the basic FreeArc compression tools and highlighted some of the tools that all repackers use. However to make more advanced repacks like we do here at KaOs, there's also a great deal of work done with custom tools outside of the compression chain to crank that compressed size down and compress everything as best as we can. There's also tools for game audio and Bink videos that I haven't discussed here. I'm sure if data compression fascinates you like it does myself, you won't hesitate to dig deep and build your research yourself.
We have decided to use a new compression and installer tool Phobos has developed for distributing ZeroFront Silver. ZF240716 build is 18GB at 11833 files. Using 7-Zip Ultra LZMA2 with 3840MB Dictionary size, 273 Word size and 16GB Solid Block size it compresses to 3.31GB in 85 mins. Using Phobos' UltiComp Repacker Tool (at 75mb, which includes SREP + LOLZ), it compresses to 2.72GB in 40 minutes. This means we can get even more maps into future versions while keeping the mod builds under 4GB for archival. The 240716 build has 327 playable maps featured.
This is a major breakthrough because UltiComp not only compresses ZeroFront better than 7z, but also in <half the time! The autoinstaller is nice too. We used to use installers back with v0.91 and v0.94. But Inno Setup is way better than InstallForge. 12 languages supported: English, Dutch, Farsi, French, German, Hungarian, Italian, Japanese, Polish, Portuguese, Russian, Spanish
ZF240716 Silver = 18.0GB Zip (Ultra, Deflate, Word size 258) = 7.76GB, 43.1%, 40mins to compress RAR (Best, Solid, Dictionary size 32GB) = 3.37GB, 18.7%, 9mins to compress 7z = 3.31GB, 18.4%, 85mins to compress, 105 seconds to extract UltiComp ARC = 2.72GB, 15.1%, 40mins to compress, 50 seconds to extract
ZeroGold Encrypted Addon is 5.09GB, 222 files 7-Zip = 616mb, 12.1% UltiComp = 546mb, 10.7%
This brings the previous total size of 3.93GB down to 3.27GB, saving almost 700mb of space.
UltiComp Repacker Tool + Compression Tests @ https://swbfmodders.com/index.php?topic=2249