PNG/FLAC support?

[ghost]

Hi,

I have been working on an ambitious mod for Doom 3 for quite a while now, targeting dhewm3 from the onset.

What would be great to have is to have the engine support the PNG format for textures and FLAC for sound. Both formats are well supported and offer maximum image/audio fidelity.

This would negate the need to convert textures to the TGA format (which isn't as widely supported as PNG) and audio to OGG (which is a lossy format).

What do you think?

[DanielGibson]

To be honest, I think it's not worth the effort.

For short sound effects, you can just use uncompressed .wav, like doom3 does (it'll be losslessly ZIP compressed when put it in a .pk3) - and even for longer stuff, I doubt you can really tell the difference between high quality (oggenc -q10, or even just -q8) and FLAC.

I think PNG is not that optimal either:

• it takes a lot longer to load than TGA (due to decompression)
• PNG export often fucks up the alpha channel (especially Photoshop's export does this)
• TGA in .pk3 (=> ZIP compression) shouldn't be that much bigger
  • PNG uses the same kind of compression (and some additional filters that are supposed to make it compress a bit better)
  • so .png in .pk3 would be compressed twice (twice the uncompression cost), but that wouldn't make it any smaller, because compressed PNGs (or any compressed file) can't really be compressed further

I'm really surprised you find TGA badly supported, is there any software (besides maybe MS Paint) that supports png but not tga?

[ghost]

Thanks for your arguments, they make good sense. Oh well, it isn't a big deal. I use GIMP a lot which handles PNG flawlessly. I'm not saying TGA is badly supported, it's just that outside of idTech4 I don't know of any programs requiring it, while PNG is pretty much the standard nowadays.

[DanielGibson]

The C4/Tombstone Engine loads textures (only) from TGA (unless you add a PNG importer yourself which is reasonably easy), and converts it into a custom compressed format. No idea about other engines, probably most support a variety of image formats nowadays, including PNG. Generally it's better to use a format that's natively supported by the GPU and already includes mipmaps (like DDS which is supported by Doom3) - this significantly reduces loading times (the GPU driver doesn't have to create the mipmaps itself) and GPU memory usage (but it's not lossless).

No idea what your textures look like and what size they have etc, but maybe you should at least try out DDS? Maybe the compression artifacts aren't noticeable (especially if the textures have higher resolutions than the original Doom3 textures) and it might be worth it? You might be able to use more or higher resolution textures then and still run on GPUs with less memory (and either way have faster loading times). https://modwiki.xnet.fi/DDS_(file_format) has some info

[ghost]

I will look into using DDS. I can't help but think that it would benefit dhewm3 if developers were given the option to use more widely used file formats. If given the choice between faster loading times and using lossless formats, I would pick the latter everytime. One advantage is that it would eliminate the need to keep lossless copies of development files and having to convert them down to import them into the game - that's worth waiting a second or two extra for. It speeds up development time and makes it less complicated, and it's just nice in general knowing that your textures and audio are represented in the best way they can possibly be. Support for the 7zip compression format would be great too - sure, it would increase loading times more, but again, it's worth the compromise. The average PC nowadays is more than fast enough and comes with ample RAM to make it a non-issue in practical terms. If the game is loaded from an SSD (and I have a hard time imagining SSD's wont become the de facto standard within a few years), then the loading times are negligible and being able to compress the mod down to roughly half the size (on average) is desirable.

I hate to push, but I urge you to reconsider nonetheless. GzDoom supports all the formats I mentioned.

[DanielGibson]

I will look into using DDS. I can't help but think that it would benefit dhewm3 if developers were given the option to use more widely used file formats. If given the choice between faster loading times and using lossless formats, I would pick the latter everytime.

I wouldn't, I find even the loading times of vanilla doom3 on modern hardware annoying (BFG edition is much better in that regard)

One advantage is that it would eliminate the need to keep lossless copies of development files

That's the normal thing to do in "real" gamedev ;-)

It speeds up development time and makes it less complicated, and it's just nice in general knowing that your textures and audio are represented in the best way they can possibly be.

It's a nice feature of doom3 that you can keep on using TGAs during development and it would be sufficient to convert them for releases (and it prolly makes sense to ship both the TGAs and DDS files to your players, in case their GPU driver doesn't support DDS).

Support for the 7zip compression format would be great too - sure, it would increase loading times more, but again, it's worth the compromise. The average PC nowadays is more than fast enough and comes with ample RAM to make it a non-issue in practical terms. If the game is loaded from an SSD (and I have a hard time imagining SSD's wont become the de facto standard within a few years), then the loading times are negligible and being able to compress the mod down to roughly half the size (on average) is desirable.

Sorry, but 7zip and its slowness is not desirable at all, especially not with SSDs. Modern harddisks can provide 80-120MB/s (when reading sequentially), SSDs 550MB/s or even more. So reading the file from the disk/SSD is really fast and it doesn't matter that much if better compression can make it 10% smaller or something.

Decompressing 7zip reaches maybe 60-70MB/s on modern CPUs, while decompressing .zip reaches >250MB/s (numbers taken from https://github.com/inikep/lzbench - lzma is the algorithm used by 7zip, while ZIP uses zlib; on highend CPUs both numbers should be a bit higher than in that table, but the relation should be the same). So with SSDs decompression speed will be the most important factor - and that depends on the CPU, not the disk (and even modern harddisks can deliver data faster than 7zip can be decompressed).

7zip may make sense to distribute downloads (once to be decompressed by the user), because download speeds are (usually) <=12MB/s, so the relatively slow unpacking of the 7zip archive still gets you some speed due to faster download (because of smaller file). But for archives holding the game data that is loaded each time the game is started (or a level is loaded) it's not very suitable.

Note however that having the game data in some kind of archive - even uncompressed - is usually better than 1000s of small files, due to filesystem overhead occuring when opening a file: with an archive, from the filesystems point of view the game only opens one file (the archive) and afterwards reads within that file, so the filesystem/kernel doesn't have to find all the small files actually used by the game (contained in the archive) and check their privileges etc.