embg
commented
5 months ago There is some discussion here: https://github.com/facebook/zstd/issues/3970
Open skyskyhu opened 5 months ago
embg
commented
5 months ago There is some discussion here: https://github.com/facebook/zstd/issues/3970
Rectently I integrated the zstd plugin with intel QAT hardware acceleration. The hardware acceleration speed is siginificantly faster than that of software compression. However, my primary objective is to reducue total storage size and cost using zstd compression. For all testing files, I observed that the compression result of hardware acceleration is lower than software compression as shown below. I suspect this is caused by the limitation of zstd hardware acceleration doucmented at the link and link. History buffers are not currently supported, resulting in a lower hardware compression ratio compared to software compression. I am wondering if there is any plan to support history buffers in zstd hardware acceleration in the near future. Thanks!
hardware compression with level 12
./benchmark -m1 -l10 -c64K -t5 -E2 -L12 1.txt Thread 1: Compression: 446020002 -> 130257784, Throughput: Comp: 508 MB/s, Decomp: 1273 MB/s, Compression Ratio: 29.20%, PASS
software compression with level 12
./benchmark -m0 -l10 -c64K -t5 -E2 -L12 1.txt Thread 1: Compression: 446020002 -> 120037796, Throughput: Comp: 39 MB/s, Decomp: 1350 MB/s, Compression Ratio: 26.91%, PASS
software compression with level 9
./benchmark -m0 -l10 -c64K -t5 -E2 -L9 1.txt Thread 1: Compression: 446020002 -> 120318010, Throughput: Comp: 99 MB/s, Decomp: 1341 MB/s, Compression Ratio: 26.98%, PASS