[Feature] PK3 support

[NightFright2k19]

I am aware this is a QSS feature, but since this build already includes some code from QSS/vkQuake, I am trying my luck:

Any chance of getting PK3 support? Newer releases like AD, Alkaline or DOTM are quite huge and the outdated pak format offers almost no compression at all. However, with PK3, file size can be reduced by 50-70%.

IMO it's one of QSS's most useful features and shouldn't break vanilla compatibility if it was implemented. In Quakespasm and vkQuake, authors don't want to implement it, so this is kind of my last hope of getting the feature in Quakespasm without having to use QSS.

[andrei-drexler]

I was going to suggest using the built-in compression on Windows (which is what I'm using), but then I did a quick test on AD's pak1.pak and it's not even close: ~512 MB uncompressed, ~308 MB with NTFS compression, ~174 MB unpacked+zipped, so there's definitely an advantage to supporting pk3's. I think I'll implement this, and while I'm at it, I'll also probably make it so that all the pk3's in the mod dir are used (like in Q3), not just those named pakN.pk3.

[NightFright2k19]

Holy crap, finally found a port author with common sense here! Indeed, tons of disk space can be saved with the zip format. Especially TGAs compress extremely well.

And yeah, loading pk3s regardless of their names is also how it works in QSS. Load order is alphabetical, so something like alkaline.pk3 would be loaded before dotm.pk3 if it's in the same directory. One could even have a pk3 version of quakespasm.pak in their root dir and it would be loaded. (Not that it would gain you much, but in QSS it's possible.)

Thank you so much, really!

[j4reporting]

the default NTFS compression is an on-the-fly compress/decompress operation, so it needs to be fast. The compact command allows you to select better compression algorithms with the /exe argument. see https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/compact

apparently the /EXE option is optimized for files that are read only. PAK files seem to be a good candidate. However, I have not tested whether processes can still access individual files within a pak file quickly without first decompressing the entire pak file in memory. So there may be a noticeable delay in loading times.

here are the compress ratios for ad pak1.pak

default  without /exe         537160200 : 323158016 = 1,7 to 1 [OK]
/exe:XPRESS4K                  537160200 : 259055616 = 2,1 to 1 [OK]
/exe:XPRESS8K                  537160200 : 240979968 = 2,2 to 1 [OK]
/exe:XPRESS16K                537160200 : 231804928 = 2,3 to 1 [OK]
/exe:LZX                           537160200 : 183558144 = 2,9 to 1 [OK]

pk3 support would give one big advantage over pak files. You can access the content w/o a pak_explorer. But I'm not so sure if it's a good idea to use all pk3 files inside a directory, because I tend to disable pak files by renaming them. But that's just me ;)

[andrei-drexler]

Thanks, I didn't know about the new compression options, but at a glance they seem to be exactly the missing ingredient. I couldn't find more technical details specifically about them, but according to some older docs NTFS compression is performed on chunks of 16 clusters (or 64 KB, whichever is smaller). I'd expect the new methods not to change that too much (at least not to the point that random access is no longer possible without decompressing the whole file first), and the recommendation to use them on read-only files to be based on compression speed (which seems to be quite a bit lower). My old Atom x7-Z8700 (Surface 3) took a few minutes to compress AD's pak1.pak with LZX, but there didn't seem to be much of an impact on loading times.

It's a shame Microsoft didn't expose these new options in Explorer, though. If that were the case, then I'd probably recommend using XPRESS16K or LZX over PK3s, simply because I'd rather maintain compatibility with QS and vkQuake. If someone converts their PAKs to PK3s to save some space and then finds an issue while playing, they won't be able to check if the same issue also occurs in QS/vkQ. I'd really like to avoid that situation. On the other hand, though, I do find it hard to recommend a solution that involves using the command line. Quake is already not very user-friendly as it is, and I don't want to make matters worse.

Anyway, long story short, now I'm a bit less sure of the benefits PK3s provide, but I'll probably still end up adding support for them.

[j4reporting]

I knew that LZX looked somehow familiar... it was first implemented as an archiving program for AmigaOS.

There is a big significant disadvantage when compressing files with the /EXE option. A file is immediately decompressed when it is opened for writing. So it is just enough to open a pak file with the PakExplorer.

/EXE | Uses compression optimized for executable files that are read frequently, but not modified.

With pk3 support you may need to get around some issues how paths are stored, i.e. a user may add a file as ./texture.tga.

My 2 cent: Point windows users to NTFS compresion I've found a GUI program for the XPRESS/LZX algorithms. On linux btrfs supports transparent compression.

[NightFright2k19]

I have tried CompactGUI now. With XPRESS8K, I get a 30% compression rate on my Quake folder. While that's not bad, converting all pak files to pk3 with highest zip compression would still double that rate.

[j4reporting]

what's the point in comparing a fast, medium compression setting with zip's highest and slowest setting? If you want NTFS to use the best compression then use LZX. According to my tests, LZX even outperforms zip compression at level 9.

[NightFright2k19]

Didn't try LZX because in CompactGUI, it's stated that mode is not recommended for gaming. Well, guess I'll give it a shot after all, then.

[j4reporting]

then the same statement applies to zip with the highest compression. On a 'modern' HW you should not notice a big difference in loading time.

[j4reporting]

AFAIK ezquake has pk3 support. But the base is QW.

[NightFright2k19]

So, tried LZX, almost the same compression rate as XPRESS8K (35% of my Quake folder). Doesn't beat pk3 with max compression for me, sorry. At least I can say I tried. So yeah, can't close this feature request.

[andrei-drexler]

It's ok, I probably would've reopened it anyway ;) IMO there are other downsides to LZX besides compression ratio that are even more important, like no built-in UI support in the OS (requiring an external program), and files not retaining compression when moved/copied to a USB stick, which make it really hard to recommend using it as an alternative to PK3 support. It seems to me like the feature is not ready yet for general use.

Anyway, just popping in for a quick update to say that I haven't forgotten about PK3 support. I've familiarized myself a bit with the filesystem code when adding Steam detection, and I'll probably have a go at implementing this pretty soon :tm:

[NightFright2k19]

Cool beans! Keeping my fingers crossed for this one!

[Muleke]

PK3s are just better to manage. I am so glad this is a possibility for Ironwail! Thanks for considering that, @andrei-drexler, and thanks for requesting this feature, @NightFright2k19!

[KillPixelGames]

Came here to request .pk3 support and glad to find someone beat me to it!

+1 for pk3

[lockheed]

It's been 2.5 years since @andrei-drexler said Ironwail will support pk3. Is this still really not the case?

[NightFright2k19]

With recent monster releases such as the Brutalist Jam or latest Mjölnir (3 GB), this feature would be more useful than ever.

[gitdrug]

bump

[Moonkeytest]

So, it's been a couple of years now and there's still no mention of this. Has it been abandoned, did it pose to much of a technical challenge?

[andrei-drexler]

It's still on my todo list. It's not technically challenging, it's mostly just tedious because Quake doesn't really have a nice file system abstraction that's used consistently throughout the code and there are a whole bunch of places where it just switches to plain stdio file access. IIRC QSS gets around this by just unzipping files to a temp location before reading them, but I'd rather not wear out people's SSD's. My time and energy are pretty limited lately, so I'd rather focus on things that are either fun to do (which can be a bit random), or that would make more of an impact. That being said, I still want to add this feature at some point.

[KillPixelGames]

That's good to hear -- this would be an excellent feature.

[j4reporting]

I admit i am not very keen on pk3 support IF it leads to even more fragmentation of the community. When MODs are only released in pk3 format, the users of QS and other engines would be left out. Perhaps pk3 support should be introduced in QS first and then distributed downstream?

[tyfighter]

I admit i am not very keen on pk3 support IF it leads to even more fragmentation of the community. When MODs are only released in pk3 format, the users of QS and other engines would be left out. Perhaps pk3 support should be introduced in QS first and then distributed downstream?

This curmudgeonly attitude has been a problem with the Quake scene forever. I am 100% in favor of fragmentation, so that these engines facing irrelevance can be kicked to the curb. Improvements need to happen to keep this scene alive, and it's no ones responsibility to police what can and can't be implemented.

[j4reporting]

Great attitude. Way to go. What are we discussing here? Where is the big progress/improvement with pk3 support, except that someone saves some disc space because he/she might have bought a too small ssd/hdd, or is too cheap to buy a bigger one? :)

We don't live in the mid 90s anymore, when you still paid $600 or more for a 2GB SCSI hard drive.

My vkQuake directory is just 18.4GB uncompressed / size on disk (ntfs compression) 12gb.
Ironwail 10GB/6.5GB size on disk. That's insignificant compared to today's SSDs.

For comparison doom 2016 occupies 69GB and this is for many already a very old and outdated game!

[tyfighter]

PAK files have a 2048 file limit, which necessitates spreading out large mods across multiple PAKs. But more importantly this attitude that huge file sizes are fine ignores that with Ironwail's performance, the largest most complex mods can reasonably played on systems even 15 years old, where it would be a slide show on QS (and sometimes QSS).

The problem as I see it is that this attitude of engine compatibility is an issue of misunderstood priorities. The contributions of John Fitzgibbons, Ozkan Sezer, et. al. are legendary, but their work is only relevant in that enabled the mappers/modders to make more and better content. That's all that's important here.

Quake engines (and their developers) are not the reason the Quake modding scene is still active. They're only in a supporting role. The mappers and mod authors are the only people keeping Quake alive.

[Gabriel-Goldsztajn]

@j4reporting: your way of thinking is why most software gets overbloated and its codebase harder to maintain over the years. Legacy software that is not up to snuff needs to go away.

Who is using Darkplaces anymore, other than the Wrath devs? Almost no one.

Who is using vanilla Quakespasm instead of Quakespasm-Spiked, vkQuake or Ironwail? Almost no one.

What uses PK3s? Mods and Texture Packs, for one.

Guess what other very respected Source Port does support PK3? Quakespasm-Spiked.

Who should care about your opinion on what are reasonable space savings, the size of your Ironwail/vkQuake folders and your Doom 2016 strawman? ONLY YOURSELF.

Lastly, who the hell do you think you are calling people "cheap" based on your quixotian delusions?

[andrei-drexler]

Please keep the discussion civil. No matter how strongly you might disagree with someone's viewpoint, personal attacks are not ok.

[Gabriel-Goldsztajn]

Nevermind j4reporting's derisive attitude towards a request that you yourself considered reasonable (though not too feasible) and him calling "cheap" to everyone based on his segregatory socioeconomic opinions, right? Those are ok, I guess.

Understood.

[mhQuake]

There's nothing in the PAK file format that enforces the limit of 2048 files, and it's just an arbitrary number chosen by ID. That limit could be trivially removed from both engine and tools. The engine reads the PAK directory to a fixed-size stack buffer and then copies that to Hunk memory, but it could just read directly to correctly-sized Hunk memory instead, and remove the distinction between on-disk and in-memory formats for the directory structure. Tools would need whatever is appropriate for each tool.

Does anybody have any metrics on space-saving vs performance for compressed PK3s? I seem to remember that Q3A used uncompressed PK3 but if performance really isn't an issue anymore then it's happy days. All the stdio stuff that @andrei-drexler mentioned would be a tedious (and potentially error-prone) slog though.