search

Cyan4973 / xxHash

Extremely fast non-cryptographic hash algorithm
http://www.xxhash.com/
Other
8.97k stars 769 forks source link

rsync 3.2.3 + xxh128/xxh3 support #481

Closed centminmod closed 3 years ago

centminmod commented 3 years ago

No sure if this belongs on xxHash issues tracker or rsync's. I tried building 2 rsync 3.2.3 RPMs on CentOS 7. One on a system with avx2 support and one without avx2 support.

The resulting rsync 3.2.3 binary's help output lists that xxh128 and xxh3 support is only on the non-avx2 built version and not the avx2 one? So what on the system determines whether rsync with xxHash will support xxh128 and xxh3 ? Trying to determine if I missed a build step/flag/configuration somewhere.

Both systems had xxhash-devel installed from EPEL repo

yum -q info xxhash-devel
Installed Packages
Name        : xxhash-devel
Arch        : x86_64
Version     : 0.8.0
Release     : 1.el7
Size        : 183 k
Repo        : installed
From repo   : epel
Summary     : Extremely fast hash algorithm - development files
URL         : http://www.xxhash.com/
License     : BSD
Description : Development files for the xxhash library

avx2 support system

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Stepping:              3
CPU MHz:               4199.707
CPU max MHz:           4400.0000
CPU min MHz:           800.0000
BogoMIPS:              7982.22
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
free -mlt
              total        used        free      shared  buff/cache   available
Mem:          31973        7163       13891         165       10918       24252
Low:          31973       18081       13891
High:             0           0           0
Swap:          2045           0        2045
Total:        34019        7163       15937

no avx2 support system

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Model name:            Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz
Stepping:              7
CPU MHz:               3299.998
BogoMIPS:              6599.99
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm tsc_adjust xsaveopt arat
rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
free -mlt
              total        used        free      shared  buff/cache   available
Mem:           3788        1228         334         192        2226        2093
Low:           3788        3454         334
High:             0           0           0
Swap:          2047         142        1905
Total:         5836        1370        2239
Cyan4973 commented 3 years ago

It's a good question, but I suspect it should better be directed at rsync. xxhash has basically no control over how it's integrated into rsync, there are (too) many scenarios that could lead to such a side effect.

centminmod commented 3 years ago

cheers @Cyan4973 will ask rsync folks https://github.com/WayneD/rsync/issues/122 :)