Investigate GDeflate supercompression

[lexaknyazev]

• Low-hanging fruit is to support this scheme as a general supercompression scheme.
• Then, it may be interesting to combine this approach with BasisLZ.

https://github.com/microsoft/DirectStorage/tree/main/GDeflate https://github.com/NVIDIA/libdeflate/tree/gdeflate

[MarkCallow]

Questions:

1. Can GDeflate be applied to already Deflated data or does the library combine deflation and re-ordering?
2. If the answer to 1 is yes, is BasisLZ compatible with the Deflated data GDeflate expects?

[lexaknyazev]

1. The library expects uncompressed data.
2. BasisLZ is a highly customized compression scheme with many domain-specific tweaks.

[fintelia]

Some of the press releases mention breaking the GDeflate data into 64 KB chunks and compressing each separately, which enables significantly more parallelism (especially important on the GPU). In fact, the Vulkan extension even takes an array of chunks to decompress in parallel. Oddly however, neither the bitstream format specification nor any of the other documentation I've found specify a standard way of encoding the number/locations of chunks.

EDIT: It looks like the tile format can be reconstructed from the reference implementation, particularly TileStream.h and GDeflateCompress.cpp (plus MSVC's documentation on bitfield layout).

In addition to the question of how to specify the list of chunks, there's also a number of different ways to map chunks to portions of the image. Some that come to mind:

1. Directly map each consecutive 64 KB range of the image data to a chunk
2. Have chunks be rectangular blocks of pixels with the dimensions used by Vulkan's sparse residency feature (i.e. 128x128 for 2D RGBA8)