toktx: Zstd deflation failed; KTX error: Out of memory.

[c2matrix]

when I try toktx --uastc 4 --zcmp 3 N.ktx2 N.png, it fail with message "toktx: Zstd deflation failed; KTX error: Out of memory." if i remove --zcmp 3 option it will be ok. I tried many images all with the same format (1024*1024 rgba png), just one image has this problem. I'm use OSX Monterey 12.0.1, toktx is 4.0.0~6 the image is attached below. .

[MarkCallow]

This is happening because the "deflated" image is larger than the UASTC image. Investigation into why is necessary.

[MarkCallow]

I plan to consult the zstd experts but it looks to me as though the nature of the data in this image means it does not compress well with lossless techniques. Running zstd, with default compression level of 3, on the .png file results in a larger file. Of course that already has internal Deflate compression which makes it tougher for any subsequent compression steps.

You can try increasing the zstd compression level, --zcmp <n> or enabling RDO --uastc_rdo_l <n>. I haven't tried increasing the zmp level. An RDO value of 5 or greater results in the compressed being a little smaller. Even with the maximum RDO value of 10, the size reduction is minimal so you may as well skip the zstd step.

libktx should probably handle this case by aborting the zstd compression and not changing the input ktxTexture2. If it does so, I have to figure out how to notify the caller.

[MarkCallow]

The fix in PR #527 ensures there is enough space for the output so there will no longer be an out of memory error. Using the default compression level, 3, and no RDO the result is a handful of bytes larger than the original. To make it smaller you can use RDO, as suggested in an earlier comment or you can raise the zstd compression level. --zcmp 20 is needed in this case. Using both RDO and an increased compression level will be better.

I think this image is a struggle for zstd due to the random nature of the pattern.