search

OpenTechEngine / Discussions

The issues of this project are abused as a discussion forum for Open Tech
4 stars 0 forks source link

File compression algorithm (reduce game loading times) #15

Open Anthony-Gaudino opened 8 years ago

Anthony-Gaudino commented 8 years ago

I remember I was searching about compression algorithms some time ago and I found this one to be great https://github.com/Cyan4973/lz4 / https://code.google.com/p/lz4/ it's very fast and was used in Battlefield 4 and many other software.

Do you think it would be a good addition to the engine?

kortemik commented 8 years ago

Friend of mine works with lrzip so I am familiar with lzs .. but might be for another day. pk4 is the easiest for modders.

On Thu, Aug 27, 2015 at 5:31 AM, Anthony Gaudino notifications@github.com wrote:

I remember I was searching about compression algorithms some time ago and I found this one to be great https://github.com/Cyan4973/lz4 / https://code.google.com/p/lz4/ it's very fast and was used in Battlefield 4 and many other software.

Do you think it would be a good addition to the engine?

— Reply to this email directly or view it on GitHub https://github.com/OpenTechEngine/Discussions/issues/15.

ghost commented 8 years ago

I was also looking into the issue of compression when it comes to the texture formats. Currently I'm using Caesium, an open source image compression tool to compress PNG textures even more. There is a very small loss in image quality but it helps cut down on overall size. It cuts down around 30-50kb in size, on average almost half the size of the .tga textures used in Doom 3 when PNG is used.

DanielGibson commented 8 years ago

lz4 makes sense for network packets and things like that which must be compressed and decompressed really fast, even if the compression ratio is not optimal.

zip (pk4) compresses better than that and is fast enough for us. and it has the huge advantage that everyone can easily create and access it, so it's definitely better for assets.

if I didn't care about ease of use, I'd probably suggest using lzham: https://code.google.com/p/lzham/ and https://github.com/richgel999/lzham_codec
It compresses better than zip and is about as fast for decompressing (compressing is a lot slower, but that's only done once anyway)

BTW, the loading times are already very good in the Doom3 BFG engine due to binary models etc (original doom3 was a lot slower because it used textbased formats which take longer to parse).

Also note that we don't yet have PK4 support, but I plan to do it in the future.

ghost commented 8 years ago

I noticed today that saving out PNG from Inkscape gives an even smaller file size than Caesium. I've tested a texture set from Doom 3, the original size of the Diffuse, Specular and Normal maps was 2.28mb together, after Inkscape it came out as 888kb together with almost no loss in quality. Some textures go down to sizes of 4-11kb. It loads super fast in OTE.