Compressed SDKMESH / DDS support

[walbourn]

Both the .sdkmesh and .dds file formats are good runtime formats, but on disk they are not space efficient. A simple solution is to use a .zip-like compression on them on disk, and decompress them on load.

For Windows 8 or later, this can be done by using the Compression API. This avoids requiring 3rd party libraries for this feature.

Because this is a Windows 8+ only API, I will add support as well to DirectX Tool Kit for DX11 only when not building for Windows 7 support. In theory it could use an alternative external library, but the data files would not necessarily be compatible anyhow.

[walbourn]

Index Buffers

I can use SDKMESH_INDEX_BUFFER_HEADER.IndexType values other than IT_16BIT = 0 and IT_32BIT = 1.

enum SDKMESH_INDEX_TYPE : uint32_t
{
    IT_16BIT = 0,
    IT_32BIT,
    IT_COMPRESSED_16BIT,
    IT_COMPRESSED_32BIT
};

// For compressed IB, SizeInBytes is compressed and uncompressed size is NumIndices * 2 or NumIndices * 4

Vertex Buffers

Currently the VB header has uint64_t StrideBytes; which is pretty extreme (the maximum supported stride in Direct3D is 2048 bytes).

I'll replace this with two uint32_t fields

enum SDKMESH_VB_COMPRESSION : uint32_t
{
    VC_NONE = 0,
    VC_COMPRESSED = 1,
};

struct SDKMESH_VERTEX_BUFFER_HEADER
{
    uint64_t NumVertices;
    uint64_t SizeBytes;
    uint32_t StrideBytes;
    uint32_t Compression;
    D3DVERTEXELEMENT9 Decl[MAX_VERTEX_ELEMENTS];
    uint64_t DataOffset;
};

// For compressed VB, SizeInBytes is compressed and uncompressed size is NumVertices * StrideBytes

This scheme for SDKMESH_VERTEX_BUFFER_HEADER would break BigEndian support in existing sdkmesh files, but I don't support it currently anyhow In theory if I had to, I could just define an alternate version of this header as SDKMESH_VERTEX_BUFFER_HEADER_BE to swap the order to uint32_t Compression; uint32_t StrideBytes; which would work for all valid "BigEndian" SDKMESH files anyhow.

[walbourn]

For the DDS format, if the magic number is "DDSZ", then the data after the header (or extended header) is 'zip' compressed as a single buffer. The information in the header describes the uncompressed resource, and the uncompressed size is stored in DDS_HEADER.reserved2.

[walbourn]

One option is to update texconv, meshconvert, and the SDK Samples Content Exporter create these compressed data types, which might be a bit tricky.

Another option is to just provide a command-line tool with the DirectX Tool Kit for compressing/uncompressing sdkmesh and dds files: meshcompress. While this makes it a two-step process, it avoids having to add a bunch of features to the other toolchains, or making DirectXTex and other tooling support direct viewing of "DDSZ" files. This tool can operate just on the bytes without having to have a detailed understanding of the formats themselves, and the tool itself would only run on Windows 8 or later versions of Windows.

meshcompress would support decompressing back to sdkmesh and dds so you could use other tooling.

I would want to update ddsdump and sdkmeshdump to handle the compressed versions for viewing the metadata since neither of these tools actually care about the data area proper.

[walbourn]

I think I'll adopt a convention of .zdkmesh and .ddz to indicate these compressed variants. None of the existing tooling would support them anyhow. They would just be a feature specific to the DirectX Tool Kit loading code.

[walbourn]

I have a solution like this in some ATG samples that will be published soon. I will add a doc page on this topic when it's available.

[walbourn]

https://github.com/microsoft/Xbox-GDK-Samples/blob/main/Kits/ATGTK/CompressedTextureFactory.h

https://github.com/microsoft/Xbox-GDK-Samples/blob/main/Kits/ATGTK/ReadCompressedData.h https://github.com/microsoft/Xbox-GDK-Samples/blob/main/Kits/ATGTK/ReadCompressedData.cpp

https://github.com/microsoft/Xbox-GDK-Samples/tree/main/Samples/Tools/xbcompress

[walbourn]

https://github.com/microsoft/DirectXTK12/wiki/Compressing-assets