Closed liusheng closed 3 years ago
环境信息：
zstd版本为1.4.5（目前lzbench支持的最高版本），参照lzbench的README介绍，使用常用作各种测试的开源数据包“silesia.tar”作为测试数据源，其中包含了各种类型的数据，具体见这里，这里也记录一下Hadoop TestDFSIO测试生成的5GB数据作为对比。
1.4.5
TestDFSIO生成5GB数据测试结果
hadoop@hadoop-benchmark:/opt/lzbench$ ll -h /tmp/test_io_0 -rw-r--r-- 1 hadoop hadoop 4.9G Aug 31 08:26 /tmp/test_io_0
ARM 测试结果:
hadoop@hadoop-benchmark:/opt/lzbench$ ./lzbench -ezstd /tmp/test_io_0 lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski Compressor name Compress. Decompress. Compr. size Ratio Filename memcpy 1875 MB/s 2700 MB/s 5242880000 100.00 /tmp/test_io_0 zstd 1.4.5 -1 613 MB/s 11449 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -2 661 MB/s 11723 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -3 628 MB/s 12009 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -4 683 MB/s 11269 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -5 657 MB/s 11832 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -6 630 MB/s 11354 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -7 509 MB/s 11193 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -8 426 MB/s 10908 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -9 445 MB/s 11785 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -10 446 MB/s 11216 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -11 413 MB/s 8779 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -12 417 MB/s 10344 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -13 278 MB/s 11933 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -14 291 MB/s 9509 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -15 272 MB/s 11373 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -16 617 MB/s 11645 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -17 615 MB/s 11601 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -18 601 MB/s 7265 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -19 557 MB/s 11795 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -20 540 MB/s 10552 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -21 564 MB/s 12386 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -22 549 MB/s 13067 MB/s 160062 0.00 /tmp/test_io_0 done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
x86 测试结果:
hadoop@hadoop-benchmark:/opt/lzbench$ ./lzbench -ezstd /tmp/test_io_0 lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski Compressor name Compress. Decompress. Compr. size Ratio Filename memcpy 4543 MB/s 4534 MB/s 5242880000 100.00 /tmp/test_io_0 zstd 1.4.5 -1 1318 MB/s 6542 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -2 1311 MB/s 6523 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -3 1319 MB/s 6533 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -4 1317 MB/s 6520 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -5 1315 MB/s 6496 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -6 1327 MB/s 6578 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -7 684 MB/s 6603 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -8 540 MB/s 6590 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -9 539 MB/s 6576 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -10 540 MB/s 6569 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -11 539 MB/s 6607 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -12 538 MB/s 6575 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -13 302 MB/s 6568 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -14 303 MB/s 6586 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -15 302 MB/s 6605 MB/s 160065 0.00 /tmp/test_io_0 zstd 1.4.5 -16 1054 MB/s 6601 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -17 1061 MB/s 6582 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -18 1058 MB/s 6589 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -19 1044 MB/s 6593 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -20 1030 MB/s 6597 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -21 1006 MB/s 6607 MB/s 160062 0.00 /tmp/test_io_0 zstd 1.4.5 -22 954 MB/s 6603 MB/s 160062 0.00 /tmp/test_io_0 done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
silesia.tar数据测试结果
silesia.tar
hadoop@hadoop-benchmark:/opt/lzbench$ ll -h /opt/silesia.tar -rw-r--r-- 1 root root 203M Apr 16 2016 /opt/silesia.tar
hadoop@hadoop-benchmark:/opt/lzbench$ ./lzbench -ezstd /opt/silesia.tar lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski Compressor name Compress. Decompress. Compr. size Ratio Filename memcpy 3046 MB/s 3061 MB/s 211947520 100.00 /opt/silesia.tar zstd 1.4.5 -1 223 MB/s 590 MB/s 73509593 34.68 /opt/silesia.tar zstd 1.4.5 -2 158 MB/s 532 MB/s 69595425 32.84 /opt/silesia.tar zstd 1.4.5 -3 103 MB/s 509 MB/s 66682002 31.46 /opt/silesia.tar zstd 1.4.5 -4 86 MB/s 486 MB/s 65501237 30.90 /opt/silesia.tar zstd 1.4.5 -5 49 MB/s 474 MB/s 63994615 30.19 /opt/silesia.tar zstd 1.4.5 -6 37 MB/s 481 MB/s 62895995 29.68 /opt/silesia.tar zstd 1.4.5 -7 26 MB/s 514 MB/s 61337745 28.94 /opt/silesia.tar zstd 1.4.5 -8 23 MB/s 530 MB/s 60758732 28.67 /opt/silesia.tar zstd 1.4.5 -9 16 MB/s 534 MB/s 60186629 28.40 /opt/silesia.tar zstd 1.4.5 -10 12 MB/s 512 MB/s 59533835 28.09 /opt/silesia.tar zstd 1.4.5 -11 9.43 MB/s 503 MB/s 59240318 27.95 /opt/silesia.tar zstd 1.4.5 -12 6.34 MB/s 508 MB/s 58774564 27.73 /opt/silesia.tar zstd 1.4.5 -13 5.47 MB/s 514 MB/s 58088029 27.41 /opt/silesia.tar zstd 1.4.5 -14 4.67 MB/s 505 MB/s 57575731 27.17 /opt/silesia.tar zstd 1.4.5 -15 3.61 MB/s 488 MB/s 57168845 26.97 /opt/silesia.tar zstd 1.4.5 -16 3.09 MB/s 514 MB/s 55706489 26.28 /opt/silesia.tar zstd 1.4.5 -17 2.45 MB/s 488 MB/s 54603591 25.76 /opt/silesia.tar zstd 1.4.5 -18 2.03 MB/s 457 MB/s 53691954 25.33 /opt/silesia.tar zstd 1.4.5 -19 1.69 MB/s 433 MB/s 53255720 25.13 /opt/silesia.tar zstd 1.4.5 -20 1.57 MB/s 387 MB/s 52840403 24.93 /opt/silesia.tar zstd 1.4.5 -21 1.42 MB/s 387 MB/s 52741466 24.88 /opt/silesia.tar zstd 1.4.5 -22 1.31 MB/s 384 MB/s 52739759 24.88 /opt/silesia.tar done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
hadoop@hadoop-benchmark:/opt/lzbench$ ./lzbench -ezstd /opt/silesia.tar lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski Compressor name Compress. Decompress. Compr. size Ratio Filename memcpy 4347 MB/s 4359 MB/s 211947520 100.00 /opt/silesia.tar zstd 1.4.5 -1 177 MB/s 461 MB/s 73509593 34.68 /opt/silesia.tar zstd 1.4.5 -2 147 MB/s 389 MB/s 69595425 32.84 /opt/silesia.tar zstd 1.4.5 -3 102 MB/s 359 MB/s 66682002 31.46 /opt/silesia.tar zstd 1.4.5 -4 93 MB/s 342 MB/s 65501237 30.90 /opt/silesia.tar zstd 1.4.5 -5 45 MB/s 338 MB/s 63994615 30.19 /opt/silesia.tar zstd 1.4.5 -6 32 MB/s 341 MB/s 62895995 29.68 /opt/silesia.tar zstd 1.4.5 -7 23 MB/s 364 MB/s 61337745 28.94 /opt/silesia.tar zstd 1.4.5 -8 20 MB/s 379 MB/s 60758732 28.67 /opt/silesia.tar zstd 1.4.5 -9 15 MB/s 381 MB/s 60186629 28.40 /opt/silesia.tar zstd 1.4.5 -10 12 MB/s 369 MB/s 59533835 28.09 /opt/silesia.tar zstd 1.4.5 -11 10 MB/s 371 MB/s 59240318 27.95 /opt/silesia.tar zstd 1.4.5 -12 7.66 MB/s 374 MB/s 58774564 27.73 /opt/silesia.tar zstd 1.4.5 -13 5.70 MB/s 383 MB/s 58088029 27.41 /opt/silesia.tar zstd 1.4.5 -14 4.89 MB/s 381 MB/s 57575731 27.17 /opt/silesia.tar zstd 1.4.5 -15 3.89 MB/s 380 MB/s 57168845 26.97 /opt/silesia.tar zstd 1.4.5 -16 2.99 MB/s 378 MB/s 55706489 26.28 /opt/silesia.tar zstd 1.4.5 -17 2.44 MB/s 363 MB/s 54603591 25.76 /opt/silesia.tar zstd 1.4.5 -18 1.83 MB/s 342 MB/s 53691954 25.33 /opt/silesia.tar zstd 1.4.5 -19 1.47 MB/s 341 MB/s 53255720 25.13 /opt/silesia.tar zstd 1.4.5 -20 1.40 MB/s 329 MB/s 52840403 24.93 /opt/silesia.tar zstd 1.4.5 -21 1.26 MB/s 333 MB/s 52741466 24.88 /opt/silesia.tar zstd 1.4.5 -22 1.14 MB/s 327 MB/s 52739759 24.88 /opt/silesia.tar done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
测试条件
环境信息：
TestDFSIO 测试对比
Lzbench 直接测试zstd性能
zstd版本为
1.4.5（目前lzbench支持的最高版本），参照lzbench的README介绍，使用常用作各种测试的开源数据包“silesia.tar”作为测试数据源，其中包含了各种类型的数据，具体见这里，这里也记录一下Hadoop TestDFSIO测试生成的5GB数据作为对比。TestDFSIO生成5GB数据测试结果
ARM 测试结果:
x86 测试结果:
silesia.tar数据测试结果ARM 测试结果:
x86 测试结果: