search

dnbaker / dashing

Fast and accurate genomic distances using HyperLogLog
GNU General Public License v3.0
160 stars 11 forks source link

bioconda install: Illegal instruction #41

Open wwood opened 4 years ago

wwood commented 4 years ago

Hi,

Trying to use dashing as it seems very useful, thanks.

Not sure if this issue belongs here but on my system the bioconda install gives an illegal instruction error:

$ dashing cmp -M tests/data/abisko4/*fna
Dashing version: v0.4.0
Illegal instruction (core dumped)

Specifically I have

  dashing            bioconda/linux-64::dashing-0.4.0-hfc8b89e_0

installed. I presume this is an issue with dashing compilation itself, but not 100% sure on that.

Any suggestions? On my other computer it works without issue. If it helps this is my cpu specs

$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               42
Model name:          Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
Stepping:            7
CPU MHz:             1714.000
CPU max MHz:         3800.0000
CPU min MHz:         1600.0000
BogoMIPS:            6784.26
Virtualisation:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            8192K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

Thanks, ben.

dnbaker commented 4 years ago

Hi Ben,

Thanks for reporting the issue, and lscpu is really helpful. It looks like you have SSE instruction sets, AVX, and pclmul but not AVX2.

Could you try a couple things to help me figure out what's going on?

gcc -march=native -dM -E - < /dev/null | grep -i "SSE\|AVX" | sort

It could also help if you tried the statically-linked prebuilt binaries.

My guess (from the above) is that it will run using the 128-bit version but crash on the 256-bit version.

The other thing you could try is git clone --recursive https://github.com/dnbaker/dashing && cd dashing && make dashing -j8, and then see if it fails. (bioconda should, I think, be trying that, but it could help to try it outside of the condaverse.)

Thanks!

Daniel

wwood commented 4 years ago

Thanks for the quick reply.

$ gcc -march=native -dM -E - < /dev/null | grep -i "SSE\|AVX" | sort
#define __AVX__ 1
#define __corei7_avx 1
#define __corei7_avx__ 1
#define __SSE__ 1
#define __SSE2__ 1
#define __SSE2_MATH__ 1
#define __SSE3__ 1
#define __SSE4_1__ 1
#define __SSE4_2__ 1
#define __SSE_MATH__ 1
#define __SSSE3__ 1
#define __tune_corei7_avx__ 1
$ ./dashing_s128 cmp -M ~/git/galah-ben/tests/data/abisko4/*fna
Dashing version: v0.4.2
Illegal instruction (core dumped)
$ ./dashing_s256 cmp -M ~/git/galah-ben/tests/data/abisko4/*fna
Dashing version: v0.4.2
Illegal instruction (core dumped)

The make seemed to complete successfully, but produce unusable binaries:

ben@ben:/tmp$ git clone --recursive https://github.com/dnbaker/dashing && cd dashing && make dashing -j8
Cloning into 'dashing'...
remote: Enumerating objects: 139, done.
remote: Counting objects: 100% (139/139), done.
remote: Compressing objects: 100% (97/97), done.
remote: Total 1941 (delta 80), reused 96 (delta 42), pack-reused 1802
Receiving objects: 100% (1941/1941), 164.03 MiB | 15.47 MiB/s, done.
Resolving deltas: 100% (1070/1070), done.
Submodule 'bonsai' (https://github.com/dnbaker/bonsai) registered for path 'bonsai'
Submodule 'distmat' (https://github.com/dnbaker/distmat) registered for path 'distmat'
Submodule 'khset' (https://github.com/dnbaker/khset) registered for path 'khset'
Submodule 'sketch' (https://github.com/dnbaker/sketch) registered for path 'sketch'
Cloning into '/tmp/dashing/bonsai'...
remote: Enumerating objects: 27, done.        
remote: Counting objects: 100% (27/27), done.        
remote: Compressing objects: 100% (16/16), done.        
remote: Total 5967 (delta 11), reused 21 (delta 8), pack-reused 5940        
Receiving objects: 100% (5967/5967), 75.57 MiB | 13.39 MiB/s, done.
Resolving deltas: 100% (4395/4395), done.
Cloning into '/tmp/dashing/khset'...
remote: Enumerating objects: 8, done.        
remote: Counting objects: 100% (8/8), done.        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 58 (delta 2), reused 7 (delta 2), pack-reused 50        
Cloning into '/tmp/dashing/distmat'...
remote: Enumerating objects: 15, done.        
remote: Counting objects: 100% (15/15), done.        
remote: Compressing objects: 100% (12/12), done.        
remote: Total 169 (delta 7), reused 11 (delta 3), pack-reused 154        
Receiving objects: 100% (169/169), 35.86 KiB | 233.00 KiB/s, done.
Resolving deltas: 100% (73/73), done.
Cloning into '/tmp/dashing/sketch'...
remote: Enumerating objects: 161, done.        
remote: Counting objects: 100% (161/161), done.        
remote: Compressing objects: 100% (64/64), done.        
remote: Total 4662 (delta 80), reused 131 (delta 59), pack-reused 4501        
Receiving objects: 100% (4662/4662), 1.16 MiB | 1.17 MiB/s, done.
Resolving deltas: 100% (3281/3281), done.
Submodule path 'bonsai': checked out '55e3c6918c1fb7f7fd294b15bfbe5f6587406856'
Submodule 'circularqueue' (https://github.com/dnbaker/circularqueue) registered for path 'bonsai/circularqueue'
Submodule 'clhash' (https://github.com/lemire/clhash) registered for path 'bonsai/clhash'
Submodule 'flat_hash_map' (https://github.com/skarupke/flat_hash_map) registered for path 'bonsai/flat_hash_map'
Submodule 'hll' (https://github.com/dnbaker/sketch) registered for path 'bonsai/hll'
Submodule 'klib' (https://github.com/attractivechaos/klib) registered for path 'bonsai/klib'
Submodule 'kspp' (https://github.com/dnbaker/kspp) registered for path 'bonsai/kspp'
Submodule 'lazy' (https://github.com/dnbaker/lazy) registered for path 'bonsai/lazy'
Submodule 'libpopcnt' (https://github.com/kimwalisch/libpopcnt) registered for path 'bonsai/libpopcnt'
Submodule 'linear' (https://github.com/dnbaker/linear) registered for path 'bonsai/linear'
Submodule 'ntHash' (https://github.com/bcgsc/ntHash) registered for path 'bonsai/ntHash'
Submodule 'pdqsort' (https://github.com/orlp/pdqsort) registered for path 'bonsai/pdqsort'
Submodule 'rollinghashcpp' (https://github.com/dnbaker/rollinghashcpp) registered for path 'bonsai/rollinghashcpp'
Submodule 'zlib' (https://github.com/madler/zlib) registered for path 'bonsai/zlib'
Cloning into '/tmp/dashing/bonsai/circularqueue'...
remote: Enumerating objects: 1, done.        
remote: Counting objects: 100% (1/1), done.        
remote: Total 92 (delta 0), reused 1 (delta 0), pack-reused 91        
Cloning into '/tmp/dashing/bonsai/clhash'...
remote: Enumerating objects: 85, done.        
remote: Total 85 (delta 0), reused 0 (delta 0), pack-reused 85        
Cloning into '/tmp/dashing/bonsai/lazy'...
remote: Enumerating objects: 7, done.        
remote: Counting objects: 100% (7/7), done.        
remote: Compressing objects: 100% (5/5), done.        
remote: Total 44 (delta 2), reused 7 (delta 2), pack-reused 37        
Cloning into '/tmp/dashing/bonsai/flat_hash_map'...
remote: Enumerating objects: 31, done.        
remote: Total 31 (delta 0), reused 0 (delta 0), pack-reused 31        
Cloning into '/tmp/dashing/bonsai/hll'...
remote: Enumerating objects: 161, done.        
remote: Counting objects: 100% (161/161), done.        
remote: Compressing objects: 100% (64/64), done.        
remote: Total 4662 (delta 80), reused 131 (delta 59), pack-reused 4501        
Receiving objects: 100% (4662/4662), 1.16 MiB | 1.15 MiB/s, done.
Resolving deltas: 100% (3281/3281), done.
Cloning into '/tmp/dashing/bonsai/kspp'...
remote: Enumerating objects: 12, done.        
remote: Counting objects: 100% (12/12), done.        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 312 (delta 4), reused 11 (delta 3), pack-reused 300        
Receiving objects: 100% (312/312), 61.63 KiB | 307.00 KiB/s, done.
Resolving deltas: 100% (191/191), done.
Cloning into '/tmp/dashing/bonsai/libpopcnt'...
remote: Enumerating objects: 21, done.        
remote: Counting objects: 100% (21/21), done.        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 825 (delta 9), reused 17 (delta 6), pack-reused 804        
Receiving objects: 100% (825/825), 133.35 KiB | 332.00 KiB/s, done.
Resolving deltas: 100% (521/521), done.
Cloning into '/tmp/dashing/bonsai/klib'...
remote: Enumerating objects: 44, done.        
remote: Counting objects: 100% (44/44), done.        
remote: Compressing objects: 100% (31/31), done.        
remote: Total 1340 (delta 27), reused 30 (delta 13), pack-reused 1296        
Receiving objects: 100% (1340/1340), 1.20 MiB | 1.19 MiB/s, done.
Resolving deltas: 100% (771/771), done.
Cloning into '/tmp/dashing/bonsai/zlib'...
remote: Enumerating objects: 5144, done.        
remote: Total 5144 (delta 0), reused 0 (delta 0), pack-reused 5144        
Receiving objects: 100% (5144/5144), 3.61 MiB | 2.66 MiB/s, done.
Resolving deltas: 100% (3615/3615), done.
Cloning into '/tmp/dashing/bonsai/linear'...
remote: Enumerating objects: 42, done.        
remote: Total 42 (delta 0), reused 0 (delta 0), pack-reused 42        
Cloning into '/tmp/dashing/bonsai/pdqsort'...
remote: Enumerating objects: 292, done.        
remote: Total 292 (delta 0), reused 0 (delta 0), pack-reused 292        
Receiving objects: 100% (292/292), 86.99 KiB | 220.00 KiB/s, done.
Resolving deltas: 100% (162/162), done.
Cloning into '/tmp/dashing/bonsai/rollinghashcpp'...
remote: Enumerating objects: 6, done.        
remote: Counting objects: 100% (6/6), done.        
remote: Compressing objects: 100% (4/4), done.        
remote: Total 279 (delta 2), reused 6 (delta 2), pack-reused 273        
Receiving objects: 100% (279/279), 77.30 KiB | 314.00 KiB/s, done.
Resolving deltas: 100% (172/172), done.
Cloning into '/tmp/dashing/bonsai/ntHash'...
remote: Enumerating objects: 59, done.        
remote: Counting objects: 100% (59/59), done.        
remote: Compressing objects: 100% (50/50), done.        
remote: Total 496 (delta 22), reused 27 (delta 8), pack-reused 437        
Receiving objects: 100% (496/496), 11.87 MiB | 5.28 MiB/s, done.
Resolving deltas: 100% (266/266), done.
Submodule path 'bonsai/circularqueue': checked out 'cbe02d8e4792cf0f3eaf24a5f69f217762dc076d'
Submodule path 'bonsai/clhash': checked out '742f81a66c8e2ae7889d1bc4c4b4d8734bdcd5af'
Submodule path 'bonsai/flat_hash_map': checked out '2c4687431f978f02a3780e24b8b701d22aa32d9c'
Submodule path 'bonsai/hll': checked out '404dad2c6949a27121f8a2e9b4cd79dde1817647'
Submodule 'aesctr' (https://github.com/dnbaker/aesctr) registered for path 'bonsai/hll/aesctr'
Submodule 'circularqueue' (https://github.com/dnbaker/circularqueue) registered for path 'bonsai/hll/circularqueue'
Submodule 'compact_vector' (https://github.com/gmarcais/compact_vector) registered for path 'bonsai/hll/compact_vector'
Submodule 'flat_hash_map' (https://github.com/skarupke/flat_hash_map) registered for path 'bonsai/hll/flat_hash_map'
Submodule 'libpopcnt' (https://github.com/kimwalisch/libpopcnt) registered for path 'bonsai/hll/libpopcnt'
Submodule 'pybind11' (https://github.com/pybind/pybind11) registered for path 'bonsai/hll/pybind11'
Submodule 'vec' (https://github.com/dnbaker/vec) registered for path 'bonsai/hll/vec'
Submodule 'xxHash' (https://github.com/Cyan4973/xxHash) registered for path 'bonsai/hll/xxHash'
Cloning into '/tmp/dashing/bonsai/hll/aesctr'...
remote: Enumerating objects: 13, done.        
remote: Counting objects: 100% (13/13), done.        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 78 (delta 6), reused 11 (delta 4), pack-reused 65
Cloning into '/tmp/dashing/bonsai/hll/circularqueue'...
remote: Enumerating objects: 1, done.        
remote: Counting objects: 100% (1/1), done.        
remote: Total 92 (delta 0), reused 1 (delta 0), pack-reused 91        
Cloning into '/tmp/dashing/bonsai/hll/compact_vector'...
remote: Enumerating objects: 296, done.        
remote: Total 296 (delta 0), reused 0 (delta 0), pack-reused 296
Receiving objects: 100% (296/296), 414.44 KiB | 488.00 KiB/s, done.
Resolving deltas: 100% (195/195), done.
Cloning into '/tmp/dashing/bonsai/hll/flat_hash_map'...
remote: Enumerating objects: 31, done.        
remote: Total 31 (delta 0), reused 0 (delta 0), pack-reused 31        
Cloning into '/tmp/dashing/bonsai/hll/vec'...
remote: Enumerating objects: 74, done.        
remote: Counting objects: 100% (74/74), done.        
remote: Compressing objects: 100% (56/56), done.        
remote: Total 255 (delta 37), reused 52 (delta 18), pack-reused 181        
Receiving objects: 100% (255/255), 66.30 KiB | 337.00 KiB/s, done.
Resolving deltas: 100% (148/148), done.
Cloning into '/tmp/dashing/bonsai/hll/xxHash'...
remote: Enumerating objects: 2672, done.        
remote: Total 2672 (delta 0), reused 0 (delta 0), pack-reused 2672        
Receiving objects: 100% (2672/2672), 1.33 MiB | 1.33 MiB/s, done.
Resolving deltas: 100% (1581/1581), done.
Cloning into '/tmp/dashing/bonsai/hll/libpopcnt'...
remote: Enumerating objects: 21, done.        
remote: Counting objects: 100% (21/21), done.        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 825 (delta 9), reused 17 (delta 6), pack-reused 804        
Receiving objects: 100% (825/825), 133.35 KiB | 156.00 KiB/s, done.
Resolving deltas: 100% (521/521), done.
Cloning into '/tmp/dashing/bonsai/hll/pybind11'...
remote: Enumerating objects: 11332, done.        
remote: Total 11332 (delta 0), reused 0 (delta 0), pack-reused 11332        
Receiving objects: 100% (11332/11332), 4.16 MiB | 4.39 MiB/s, done.
Resolving deltas: 100% (7647/7647), done.
Submodule path 'bonsai/hll/aesctr': checked out 'dbe690e8e903b873e6f2eb76d453da38e9fcba0b'
Submodule path 'bonsai/hll/circularqueue': checked out 'cbe02d8e4792cf0f3eaf24a5f69f217762dc076d'
Submodule path 'bonsai/hll/compact_vector': checked out 'e3bbd15290ac76b9732fab7b2d2a5c83ae01595e'
Submodule path 'bonsai/hll/flat_hash_map': checked out '2c4687431f978f02a3780e24b8b701d22aa32d9c'
Submodule path 'bonsai/hll/libpopcnt': checked out '0de541e96ee4c88e5bd28a0e941ac91e99c3b62f'
Submodule path 'bonsai/hll/pybind11': checked out '34c2281e315c51f5270321101dc733c1cf26214f'
Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'bonsai/hll/pybind11/tools/clang'
Cloning into '/tmp/dashing/bonsai/hll/pybind11/tools/clang'...
remote: Enumerating objects: 8, done.        
remote: Counting objects: 100% (8/8), done.        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 361 (delta 2), reused 2 (delta 0), pack-reused 353        
Receiving objects: 100% (361/361), 147.50 KiB | 92.00 KiB/s, done.
Resolving deltas: 100% (151/151), done.
Submodule path 'bonsai/hll/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
Submodule path 'bonsai/hll/vec': checked out 'd26da8847b4e225e3d6ccad85d1589a4345284f8'
Submodule 'blaze' (https://bitbucket.org/blaze-lib/blaze.git) registered for path 'bonsai/hll/vec/blaze'
Submodule 'sleef' (https://github.com/shibatch/sleef) registered for path 'bonsai/hll/vec/sleef'
Cloning into '/tmp/dashing/bonsai/hll/vec/blaze'...
remote: Counting objects: 176514, done.        
remote: Compressing objects: 100% (21456/21456), done.        
remote: Total 176514 (delta 153992), reused 176322 (delta 153863)        
Receiving objects: 100% (176514/176514), 70.56 MiB | 9.34 MiB/s, done.
Resolving deltas: 100% (153992/153992), done.
Cloning into '/tmp/dashing/bonsai/hll/vec/sleef'...
remote: Enumerating objects: 42, done.        
remote: Counting objects: 100% (42/42), done.        
remote: Compressing objects: 100% (36/36), done.        
remote: Total 3215 (delta 20), reused 17 (delta 6), pack-reused 3173        
Receiving objects: 100% (3215/3215), 3.00 MiB | 2.37 MiB/s, done.
Resolving deltas: 100% (2436/2436), done.
Submodule path 'bonsai/hll/vec/blaze': checked out '620923485f3211cc30f63782017a8603c53f1ceb'
Submodule path 'bonsai/hll/vec/sleef': checked out '0b3c2edb5a0b53d2ffa1062c0fcb189ec20cedba'
Submodule path 'bonsai/hll/xxHash': checked out 'e22268617bb287e5b407b3c6cc29c1963c8d7c2d'
Submodule path 'bonsai/klib': checked out 'f719aad5fa273424fab4b0d884d68375b7cc2520'
Submodule path 'bonsai/kspp': checked out 'c567606ac21f8fabbdc4e4cbda4481b1452091d0'
Submodule 'klib' (https://github.com/attractivechaos/klib) registered for path 'bonsai/kspp/klib'
Cloning into '/tmp/dashing/bonsai/kspp/klib'...
remote: Enumerating objects: 44, done.        
remote: Counting objects: 100% (44/44), done.        
remote: Compressing objects: 100% (31/31), done.        
remote: Total 1340 (delta 27), reused 30 (delta 13), pack-reused 1296        
Receiving objects: 100% (1340/1340), 1.20 MiB | 1.18 MiB/s, done.
Resolving deltas: 100% (771/771), done.
Submodule path 'bonsai/kspp/klib': checked out 'f719aad5fa273424fab4b0d884d68375b7cc2520'
Submodule path 'bonsai/lazy': checked out 'c20343cb972a73bff676473a60c3bcb4818e52e6'
Submodule path 'bonsai/libpopcnt': checked out '0de541e96ee4c88e5bd28a0e941ac91e99c3b62f'
Submodule path 'bonsai/linear': checked out '0aa224367a0f09b5c3e5c90c715e3ff8fb2ffad3'
Submodule path 'bonsai/ntHash': checked out '65325bac67a840d2488908cfa266737297c159ff'
Submodule path 'bonsai/pdqsort': checked out '08879029ab8dcb80a70142acb709e3df02de5d37'
Submodule path 'bonsai/rollinghashcpp': checked out 'e68381300c074cbc6609a32367b9b7d93c65702e'
Submodule path 'bonsai/zlib': checked out 'cacf7f1d4e3d44d871b605da3b647f07d718623f'
Submodule path 'distmat': checked out '36f341c0231ef59ac546f5a6a37af74569884546'
Submodule 'pybind11' (https://github.com/pybind/pybind11) registered for path 'distmat/pybind11'
Cloning into '/tmp/dashing/distmat/pybind11'...
remote: Enumerating objects: 11332, done.        
remote: Total 11332 (delta 0), reused 0 (delta 0), pack-reused 11332
Receiving objects: 100% (11332/11332), 4.16 MiB | 2.50 MiB/s, done.
Resolving deltas: 100% (7647/7647), done.
Submodule path 'distmat/pybind11': checked out '12e8774bc9aa4603136f2979088619b495850ca2'
Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'distmat/pybind11/tools/clang'
Cloning into '/tmp/dashing/distmat/pybind11/tools/clang'...
remote: Enumerating objects: 8, done.        
remote: Counting objects: 100% (8/8), done.        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 361 (delta 2), reused 2 (delta 0), pack-reused 353        
Receiving objects: 100% (361/361), 147.50 KiB | 367.00 KiB/s, done.
Resolving deltas: 100% (151/151), done.
Submodule path 'distmat/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
Submodule path 'khset': checked out 'e33b306a5260c5a1b8d1f5174385060bb98bcf90'
Submodule 'klib' (https://github.com/attractivechaos/klib) registered for path 'khset/klib'
Cloning into '/tmp/dashing/khset/klib'...
remote: Enumerating objects: 44, done.        
remote: Counting objects: 100% (44/44), done.        
remote: Compressing objects: 100% (31/31), done.        
remote: Total 1340 (delta 27), reused 30 (delta 13), pack-reused 1296        
Receiving objects: 100% (1340/1340), 1.20 MiB | 1.17 MiB/s, done.
Resolving deltas: 100% (771/771), done.
Submodule path 'khset/klib': checked out 'f719aad5fa273424fab4b0d884d68375b7cc2520'
Submodule path 'sketch': checked out '404dad2c6949a27121f8a2e9b4cd79dde1817647'
Submodule 'aesctr' (https://github.com/dnbaker/aesctr) registered for path 'sketch/aesctr'
Submodule 'circularqueue' (https://github.com/dnbaker/circularqueue) registered for path 'sketch/circularqueue'
Submodule 'compact_vector' (https://github.com/gmarcais/compact_vector) registered for path 'sketch/compact_vector'
Submodule 'flat_hash_map' (https://github.com/skarupke/flat_hash_map) registered for path 'sketch/flat_hash_map'
Submodule 'libpopcnt' (https://github.com/kimwalisch/libpopcnt) registered for path 'sketch/libpopcnt'
Submodule 'pybind11' (https://github.com/pybind/pybind11) registered for path 'sketch/pybind11'
Submodule 'vec' (https://github.com/dnbaker/vec) registered for path 'sketch/vec'
Submodule 'xxHash' (https://github.com/Cyan4973/xxHash) registered for path 'sketch/xxHash'
Cloning into '/tmp/dashing/sketch/aesctr'...
remote: Enumerating objects: 13, done.        
remote: Counting objects: 100% (13/13), done.        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 78 (delta 6), reused 11 (delta 4), pack-reused 65        
Cloning into '/tmp/dashing/sketch/circularqueue'...
remote: Enumerating objects: 1, done.        
remote: Counting objects: 100% (1/1), done.        
remote: Total 92 (delta 0), reused 1 (delta 0), pack-reused 91        
Cloning into '/tmp/dashing/sketch/compact_vector'...
remote: Enumerating objects: 296, done.        
remote: Total 296 (delta 0), reused 0 (delta 0), pack-reused 296        
Receiving objects: 100% (296/296), 414.44 KiB | 625.00 KiB/s, done.
Resolving deltas: 100% (195/195), done.
Cloning into '/tmp/dashing/sketch/flat_hash_map'...
remote: Enumerating objects: 31, done.        
remote: Total 31 (delta 0), reused 0 (delta 0), pack-reused 31        
Cloning into '/tmp/dashing/sketch/vec'...
remote: Enumerating objects: 74, done.        
remote: Counting objects: 100% (74/74), done.        
remote: Compressing objects: 100% (56/56), done.        
remote: Total 255 (delta 37), reused 52 (delta 18), pack-reused 181        
Receiving objects: 100% (255/255), 66.30 KiB | 336.00 KiB/s, done.
Resolving deltas: 100% (148/148), done.
Cloning into '/tmp/dashing/sketch/libpopcnt'...
remote: Enumerating objects: 21, done.        
remote: Counting objects: 100% (21/21), done.        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 825 (delta 9), reused 17 (delta 6), pack-reused 804        
Receiving objects: 100% (825/825), 133.35 KiB | 86.00 KiB/s, done.
Resolving deltas: 100% (521/521), done.
Cloning into '/tmp/dashing/sketch/xxHash'...
remote: Enumerating objects: 2672, done.        
remote: Total 2672 (delta 0), reused 0 (delta 0), pack-reused 2672        
Receiving objects: 100% (2672/2672), 1.33 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (1581/1581), done.
Cloning into '/tmp/dashing/sketch/pybind11'...
remote: Enumerating objects: 11332, done.        
remote: Total 11332 (delta 0), reused 0 (delta 0), pack-reused 11332        
Receiving objects: 100% (11332/11332), 4.16 MiB | 2.95 MiB/s, done.
Resolving deltas: 100% (7649/7649), done.
Submodule path 'sketch/aesctr': checked out 'dbe690e8e903b873e6f2eb76d453da38e9fcba0b'
Submodule path 'sketch/circularqueue': checked out 'cbe02d8e4792cf0f3eaf24a5f69f217762dc076d'
Submodule path 'sketch/compact_vector': checked out 'e3bbd15290ac76b9732fab7b2d2a5c83ae01595e'
Submodule path 'sketch/flat_hash_map': checked out '2c4687431f978f02a3780e24b8b701d22aa32d9c'
Submodule path 'sketch/libpopcnt': checked out '0de541e96ee4c88e5bd28a0e941ac91e99c3b62f'
Submodule path 'sketch/pybind11': checked out '34c2281e315c51f5270321101dc733c1cf26214f'
Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'sketch/pybind11/tools/clang'
Cloning into '/tmp/dashing/sketch/pybind11/tools/clang'...
remote: Enumerating objects: 8, done.        
remote: Counting objects: 100% (8/8), done.        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 361 (delta 2), reused 2 (delta 0), pack-reused 353        
Receiving objects: 100% (361/361), 147.50 KiB | 367.00 KiB/s, done.
Resolving deltas: 100% (151/151), done.
Submodule path 'sketch/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
Submodule path 'sketch/vec': checked out 'd26da8847b4e225e3d6ccad85d1589a4345284f8'
Submodule 'blaze' (https://bitbucket.org/blaze-lib/blaze.git) registered for path 'sketch/vec/blaze'
Submodule 'sleef' (https://github.com/shibatch/sleef) registered for path 'sketch/vec/sleef'
Cloning into '/tmp/dashing/sketch/vec/blaze'...
remote: Counting objects: 176514, done.        
remote: Compressing objects: 100% (21456/21456), done.        
remote: Total 176514 (delta 153994), reused 176322 (delta 153863)        
Receiving objects: 100% (176514/176514), 70.56 MiB | 9.53 MiB/s, done.
Resolving deltas: 100% (153994/153994), done.
Cloning into '/tmp/dashing/sketch/vec/sleef'...
remote: Enumerating objects: 42, done.        
remote: Counting objects: 100% (42/42), done.        
remote: Compressing objects: 100% (36/36), done.        
remote: Total 3215 (delta 20), reused 17 (delta 6), pack-reused 3173        
Receiving objects: 100% (3215/3215), 3.00 MiB | 2.21 MiB/s, done.
Resolving deltas: 100% (2436/2436), done.
Submodule path 'sketch/vec/blaze': checked out '620923485f3211cc30f63782017a8603c53f1ceb'
Submodule path 'sketch/vec/sleef': checked out '0b3c2edb5a0b53d2ffa1062c0fcb189ec20cedba'
Submodule path 'sketch/xxHash': checked out 'e22268617bb287e5b407b3c6cc29c1963c8d7c2d'
cp bonsai/zlib/libz.a libz.a || (cd bonsai/zlib && ./configure && make && cd ../.. && cp bonsai/zlib/libz.a libz.a)
ls libzstd.a || (cd bonsai && make libzstd.a && cp libzstd.a ../)
cd bonsai && make libzstd.a
cd bonsai && make libzstd.a
cd bonsai && make libzstd.a
cd bonsai && make libzstd.a
cd bonsai && make libzstd.a
cd bonsai && make clhash.o && cd ..
cp: cannot stat 'bonsai/zlib/libz.a': No such file or directory
make[1]: Entering directory '/tmp/dashing/bonsai'
make[1]: Entering directory '/tmp/dashing/bonsai'
make[1]: Entering directory '/tmp/dashing/bonsai'
make[1]: Entering directory '/tmp/dashing/bonsai'
make[1]: Entering directory '/tmp/dashing/bonsai'
make[1]: Entering directory '/tmp/dashing/bonsai'
ls: cannot access 'libzstd.a': No such file or directory
make[1]: Entering directory '/tmp/dashing/bonsai'
git clone --recursive --single-branch --branch dev https://github.com/facebook/zstd
git clone --recursive --single-branch --branch dev https://github.com/facebook/zstd
git clone --recursive --single-branch --branch dev https://github.com/facebook/zstd
git clone --recursive --single-branch --branch dev https://github.com/facebook/zstd
git clone --recursive --single-branch --branch dev https://github.com/facebook/zstd
Cloning into 'zstd'...
fatal: destination path 'zstd' already exists and is not an empty directory.
make[1]: *** [Makefile:83: zstd/lib] Error 128
make[1]: Leaving directory '/tmp/dashing/bonsai'
fatal: destination path 'zstd' already exists and is not an empty directory.
git clone --recursive --single-branch --branch dev https://github.com/facebook/zstd
make[1]: *** [Makefile:83: zstd/lib] Error 128
make[1]: Leaving directory '/tmp/dashing/bonsai'
fatal: destination path 'zstd' already exists and is not an empty directory.
make[1]: *** [Makefile:83: zstd/lib] Error 128
make[1]: Leaving directory '/tmp/dashing/bonsai'
fatal: destination path 'zstd' already exists and is not an empty directory.
make[1]: *** [Makefile:83: zstd/lib] Error 128
make[1]: Leaving directory '/tmp/dashing/bonsai'
fatal: destination path 'zstd' already exists and is not an empty directory.
make[1]: *** [Makefile:83: zstd/lib] Error 128
make[1]: Leaving directory '/tmp/dashing/bonsai'
ls clhash.o 2>/dev/null || mv clhash/clhash.o . 2>/dev/null || (cd clhash && git checkout master && make && cd .. && ln -s clhash/clhash.o .)
make: *** [Makefile:123: bonsai/zstd/zlibWrapper/gzclose.c] Error 2
make: *** Waiting for unfinished jobs....
make: *** [Makefile:123: bonsai/zstd/zlibWrapper/gzlib.c] Error 2
make: *** [Makefile:123: bonsai/zstd/zlibWrapper/gzread.c] Error 2
make: *** [Makefile:123: bonsai/zstd/zlibWrapper/gzwrite.c] Error 2
make: *** [Makefile:81: libzstd.a] Error 2
Checking for gcc...
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
make[2]: Entering directory '/tmp/dashing/bonsai/clhash'
cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -c ./src/clhash.c -Iinclude
Checking for shared library support...
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (4/4), done.
Building shared library libz.so.1.2.11 with gcc.
Checking for size_t... Yes.36/40755), 1.45 MiB | 1.38 MiB/s   s   
Checking for off64_t... Yes.4/40755), 1.45 MiB | 1.38 MiB/s   
Checking for fseeko... Yes.
cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o unit ./tests/unit.c -Iinclude  clhash.o
g++ -fPIC -std=c++11 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o cppunit ./tests/cppunit.cpp -Iinclude  clhash.o
cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o benchmark ./benchmarks/benchmark.c -Iinclude  clhash.o
cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o example example.c -Iinclude  clhash.o
g++ -fPIC -std=c++11 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o cppexample cppexample.cpp -Iinclude  clhash.o
Checking for strerror... Yes.0/40755), 5.81 MiB | 3.75 MiB/s   
Checking for unistd.h... Yes.9/40755), 5.81 MiB | 3.75 MiB/s   
Checking for stdarg.h... Yes.0/40755), 12.21 MiB | 5.96 MiB/s   
remote: Total 40755 (delta 23), reused 23 (delta 23), pack-reused 40728
Receiving objects: 100% (40755/40755), 20.97 MiB | 7.66 MiB/s, done.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
make[1]: Entering directory '/tmp/dashing/bonsai/zlib'
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o example.o test/example.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o adler32.o adler32.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o crc32.o crc32.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o deflate.o deflate.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o infback.o infback.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inffast.o inffast.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inflate.o inflate.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inftrees.o inftrees.c
Resolving deltas: 100% (30063/30063), done.
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o trees.o trees.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o zutil.o zutil.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o compress.o compress.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o uncompr.o uncompr.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzclose.o gzclose.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzlib.o gzlib.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzread.o gzread.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzwrite.o gzwrite.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o minigzip.o test/minigzip.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/adler32.o adler32.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/crc32.o crc32.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/infback.o infback.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/deflate.o deflate.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/inffast.o inffast.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/inflate.o inflate.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/inftrees.o inftrees.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/trees.o trees.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/zutil.o zutil.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/compress.o compress.c
make[2]: Leaving directory '/tmp/dashing/bonsai/clhash'
make[1]: Leaving directory '/tmp/dashing/bonsai'
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/uncompr.o uncompr.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzclose.o gzclose.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzlib.o gzlib.c
cd zstd && make lib && mv lib/libzstd.a .. && cd ..
make[2]: Entering directory '/tmp/dashing/bonsai/zstd'
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o example64.o test/example.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzread.o gzread.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzwrite.o gzwrite.c
make[3]: Entering directory '/tmp/dashing/bonsai/zstd/lib'
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o minigzip64.o test/minigzip.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o 
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example example.o -L. libz.a
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip minigzip.o -L. libz.a
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example64 example64.o -L. libz.a
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip64 minigzip64.o -L. libz.a
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/debug.o common/debug.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/entropy_common.o common/entropy_common.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/error_private.o common/error_private.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/fse_decompress.o common/fse_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/pool.o common/pool.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/threading.o common/threading.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/xxhash.o common/xxhash.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/zstd_common.o common/zstd_common.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/fse_compress.o compress/fse_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/hist.o compress/hist.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/huf_compress.o compress/huf_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress.o compress/zstd_compress.c
gcc -shared -Wl,-soname,libz.so.1,--version-script,zlib.map -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o libz.so.1.2.11 adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo  -lc 
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress_literals.o compress/zstd_compress_literals.c
rm -f libz.so libz.so.1
ln -s libz.so.1.2.11 libz.so
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress_sequences.o compress/zstd_compress_sequences.c
ln -s libz.so.1.2.11 libz.so.1
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress_superblock.o compress/zstd_compress_superblock.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o examplesh example.o -L. libz.so.1.2.11
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_double_fast.o compress/zstd_double_fast.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzipsh minigzip.o -L. libz.so.1.2.11
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_fast.o compress/zstd_fast.c
make[1]: Leaving directory '/tmp/dashing/bonsai/zlib'
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_lazy.o compress/zstd_lazy.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_ldm.o compress/zstd_ldm.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_opt.o compress/zstd_opt.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstdmt_compress.o compress/zstdmt_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/huf_decompress.o decompress/huf_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/zstd_ddict.o decompress/zstd_ddict.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/zstd_decompress.o decompress/zstd_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/zstd_decompress_block.o decompress/zstd_decompress_block.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o deprecated/zbuff_common.o deprecated/zbuff_common.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o deprecated/zbuff_compress.o deprecated/zbuff_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o deprecated/zbuff_decompress.o deprecated/zbuff_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/cover.o dictBuilder/cover.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/divsufsort.o dictBuilder/divsufsort.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/fastcover.o dictBuilder/fastcover.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/zdict.o dictBuilder/zdict.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o legacy/zstd_v05.o legacy/zstd_v05.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o legacy/zstd_v06.o legacy/zstd_v06.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o legacy/zstd_v07.o legacy/zstd_v07.c
compiling dynamic library 1.4.5
compiling static library
creating versioned links
make[3]: Leaving directory '/tmp/dashing/bonsai/zstd/lib'
make[2]: Leaving directory '/tmp/dashing/bonsai/zstd'
make[1]: Leaving directory '/tmp/dashing/bonsai'
rm bonsai/zstd/zlibWrapper/gzlib.c bonsai/zstd/zlibWrapper/gzclose.c bonsai/zstd/zlibWrapper/gzread.c bonsai/zstd/zlibWrapper/gzwrite.c bonsai/zstd/zlibWrapper/zstd_zlibwrapper.c
ben@ben:/tmp/dashing/release/linux$ ./dashing_s128 cmp -M ~/git/galah-ben/tests/data/abisko4/*fna
Dashing version: v0.4.2
Illegal instruction (core dumped)
ben@ben:/tmp/dashing/release/linux$ ./dashing_s256 cmp -M ~/git/galah-ben/tests/data/abisko4/*fna
Dashing version: v0.4.2
Illegal instruction (core dumped)
ben@ben:/tmp/dashing/release/linux$ ./dashing_s512 cmp -M ~/git/galah-ben/tests/data/abisko4/*fna
Illegal instruction (core dumped)
dnbaker commented 4 years ago

Try make again for the git repo, sometimes a second invocation helps. (I think it's a race condition in the Makefile due to the zlib.)

What's your OS? We've tried under Arch, Ubuntu and CentOS.

wwood commented 4 years ago

I'm on Ubuntu 18.04. I have another 18.04 install on a newer laptop and that works fine, I guess because it has the required instructions.

I seem to be having problems with zlib, as you say. But not sure it is a race condition since it still occurs without -j:

ben@ben:/tmp/dashing$ make clean
rm -f sketchcorekhs sketchcorecbbmh sketchcmpbbmh readfilt finalizers sketchcmpbf sketchcmphll sketchcmprmh sketchcmpkhs hllmain sketchcorebbmh sketchcorebf sketchcorermh sketchcorehll sketchcmpcrmh construct mkdistmain union sketchcmpsmh unionsz sketchcoresmh cardests sketchcorecrmh main distmain dashing sketchcorekhs_d sketchcorecbbmh_d sketchcmpbbmh_d readfilt_d finalizers_d sketchcmpbf_d sketchcmphll_d sketchcmprmh_d sketchcmpkhs_d hllmain_d sketchcorebbmh_d sketchcorebf_d sketchcorermh_d sketchcorehll_d sketchcmpcrmh_d construct_d mkdistmain_d union_d sketchcmpsmh_d unionsz_d sketchcoresmh_d cardests_d sketchcorecrmh_d main_d distmain_d dashing_d libzstd.a bonsai/clhash.o clhash.o \
bonsai/klib/kthread.o bonsai/klib/kstring.o libgomp.a \
&& cd bonsai/zstd && make clean && cd ../zlib && make clean && cd ../.. \
&& rm -f libz.* && rm -f dashing.a
make[1]: Entering directory '/tmp/dashing/bonsai/zstd'
Cleaning completed
make[1]: Leaving directory '/tmp/dashing/bonsai/zstd'
make[1]: Entering directory '/tmp/dashing/bonsai/zlib'
rm -f *.o *.lo *~ \
   example minigzip examplesh minigzipsh \
   example64 minigzip64 \
   infcover \
   libz.* foo.gz so_locations \
   _match.s maketree contrib/infback9/*.o
rm -rf objs
rm -f *.gcda *.gcno *.gcov
rm -f contrib/infback9/*.gcda contrib/infback9/*.gcno contrib/infback9/*.gcov
make[1]: Leaving directory '/tmp/dashing/bonsai/zlib'
ben@ben:/tmp/dashing$ make
cp bonsai/zlib/libz.a libz.a || (cd bonsai/zlib && ./configure && make && cd ../.. && cp bonsai/zlib/libz.a libz.a)
cp: cannot stat 'bonsai/zlib/libz.a': No such file or directory
Checking for gcc...
Checking for shared library support...
Building shared library libz.so.1.2.11 with gcc.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
make[1]: Entering directory '/tmp/dashing/bonsai/zlib'
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o example.o test/example.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o adler32.o adler32.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o crc32.o crc32.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o deflate.o deflate.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o infback.o infback.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inffast.o inffast.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inflate.o inflate.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inftrees.o inftrees.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o trees.o trees.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o zutil.o zutil.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o compress.o compress.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o uncompr.o uncompr.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzclose.o gzclose.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzlib.o gzlib.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzread.o gzread.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzwrite.o gzwrite.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o 
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example example.o -L. libz.a
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o minigzip.o test/minigzip.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip minigzip.o -L. libz.a
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/adler32.o adler32.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/crc32.o crc32.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/deflate.o deflate.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/infback.o infback.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/inffast.o inffast.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/inflate.o inflate.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/inftrees.o inftrees.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/trees.o trees.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/zutil.o zutil.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/compress.o compress.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/uncompr.o uncompr.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzclose.o gzclose.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzlib.o gzlib.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzread.o gzread.c
gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -DPIC -c -o objs/gzwrite.o gzwrite.c
gcc -shared -Wl,-soname,libz.so.1,--version-script,zlib.map -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o libz.so.1.2.11 adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo  -lc 
rm -f libz.so libz.so.1
ln -s libz.so.1.2.11 libz.so
ln -s libz.so.1.2.11 libz.so.1
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o examplesh example.o -L. libz.so.1.2.11
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzipsh minigzip.o -L. libz.so.1.2.11
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o example64.o test/example.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example64 example64.o -L. libz.a
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o minigzip64.o test/minigzip.c
gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip64 minigzip64.o -L. libz.a
make[1]: Leaving directory '/tmp/dashing/bonsai/zlib'
ls libzstd.a || (cd bonsai && make libzstd.a && cp libzstd.a ../)
ls: cannot access 'libzstd.a': No such file or directory
make[1]: Entering directory '/tmp/dashing/bonsai'
cd zstd && make lib && mv lib/libzstd.a .. && cd ..
make[2]: Entering directory '/tmp/dashing/bonsai/zstd'
make[3]: Entering directory '/tmp/dashing/bonsai/zstd/lib'
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/debug.o common/debug.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/entropy_common.o common/entropy_common.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/error_private.o common/error_private.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/fse_decompress.o common/fse_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/pool.o common/pool.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/threading.o common/threading.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/xxhash.o common/xxhash.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o common/zstd_common.o common/zstd_common.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/fse_compress.o compress/fse_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/hist.o compress/hist.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/huf_compress.o compress/huf_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress.o compress/zstd_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress_literals.o compress/zstd_compress_literals.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress_sequences.o compress/zstd_compress_sequences.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_compress_superblock.o compress/zstd_compress_superblock.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_double_fast.o compress/zstd_double_fast.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_fast.o compress/zstd_fast.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_lazy.o compress/zstd_lazy.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_ldm.o compress/zstd_ldm.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstd_opt.o compress/zstd_opt.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o compress/zstdmt_compress.o compress/zstdmt_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/huf_decompress.o decompress/huf_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/zstd_ddict.o decompress/zstd_ddict.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/zstd_decompress.o decompress/zstd_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o decompress/zstd_decompress_block.o decompress/zstd_decompress_block.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o deprecated/zbuff_common.o deprecated/zbuff_common.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o deprecated/zbuff_compress.o deprecated/zbuff_compress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o deprecated/zbuff_decompress.o deprecated/zbuff_decompress.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/cover.o dictBuilder/cover.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/divsufsort.o dictBuilder/divsufsort.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/fastcover.o dictBuilder/fastcover.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o dictBuilder/zdict.o dictBuilder/zdict.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o legacy/zstd_v05.o legacy/zstd_v05.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o legacy/zstd_v06.o legacy/zstd_v06.c
cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -I. -I./common -DXXH_NAMESPACE=ZSTD_ -I./legacy -DZSTD_LEGACY_SUPPORT=5  -c -o legacy/zstd_v07.o legacy/zstd_v07.c
compiling static library
compiling dynamic library 1.4.5
creating versioned links
make[3]: Leaving directory '/tmp/dashing/bonsai/zstd/lib'
make[2]: Leaving directory '/tmp/dashing/bonsai/zstd'
make[1]: Leaving directory '/tmp/dashing/bonsai'
g++ -O3 -funroll-loops -pipe -fno-strict-aliasing -march=native -mpclmul -DUSE_PDQSORT -DNOT_THREADSAFE   -flto  -fopenmp -fno-rtti -std=c++14 -Wall -Wextra -Wno-char-subscripts -Wpointer-arith -Wwrite-strings -Wdisabled-optimization -Wformat -Wcast-align -Wno-unused-function -Wno-unused-parameter -pedantic -Wunused-variable -Wno-attributes -Wno-pedantic  -Wno-ignored-attributes -Wno-missing-braces -Wno-unknown-pragmas -DDASHING_VERSION=\"v0.4.3\"  -Ibonsai/clhash/include -I.  -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib  -Isketch/vec -Ibonsai -Ibonsai/include/ -Isketch/include -Isketch -Isketch/include/sketch -Isketch/vec -L.  -Lbonsai/zlib -c -O3 src/dashing.cpp libz.a -o src/dashing.o -DZWRAP_USE_ZSTD=1 -lzstd -lz -DNDEBUG
g++: warning: libz.a: linker input file unused because linking not done
cd bonsai && make libzstd.a
make[1]: Entering directory '/tmp/dashing/bonsai'
cd zstd && make lib && mv lib/libzstd.a .. && cd ..
make[2]: Entering directory '/tmp/dashing/bonsai/zstd'
make[3]: Entering directory '/tmp/dashing/bonsai/zstd/lib'
compiling static library
make[3]: Leaving directory '/tmp/dashing/bonsai/zstd/lib'
make[2]: Leaving directory '/tmp/dashing/bonsai/zstd'
make[1]: Leaving directory '/tmp/dashing/bonsai'
cd bonsai && make libzstd.a && cd zstd && make lib  && cd zlibWrapper && make gzclose.o
make[1]: Entering directory '/tmp/dashing/bonsai'
cd zstd && make lib && mv lib/libzstd.a .. && cd ..
make[2]: Entering directory '/tmp/dashing/bonsai/zstd'
make[3]: Entering directory '/tmp/dashing/bonsai/zstd/lib'
compiling static library
make[3]: Leaving directory '/tmp/dashing/bonsai/zstd/lib'
make[2]: Leaving directory '/tmp/dashing/bonsai/zstd'
make[1]: Leaving directory '/tmp/dashing/bonsai'
make[1]: Entering directory '/tmp/dashing/bonsai/zstd'
make[2]: Entering directory '/tmp/dashing/bonsai/zstd/lib'
compiling static library
make[2]: Leaving directory '/tmp/dashing/bonsai/zstd/lib'
make[1]: Leaving directory '/tmp/dashing/bonsai/zstd'
make[1]: Entering directory '/tmp/dashing/bonsai/zstd/zlibWrapper'
make[1]: *** No rule to make target 'gzclose.o'.  Stop.
make[1]: Leaving directory '/tmp/dashing/bonsai/zstd/zlibWrapper'
make: *** [Makefile:135: bonsai/zstd/zlibWrapper/gzclose.o] Error 2
dnbaker commented 4 years ago

Hi,

It seems that dashing_128 was using AVX registers, even though avx2 was disabled. This modified version might work now. And I've modified how zstd is built, so if that doesn't work, I'd recommend cloning from scratch and trying to recompile.

Sorry for the hassle, and thanks again for the feedback.

wwood commented 4 years ago

Hi,

That modified version threw an illegal instruction.

The make -j 8 dashing after a fresh clone failed, but then running make dashing after that did work, as you suggested before. Also, the ./dashing it created worked without throwing an illegal instruction.

dnbaker commented 4 years ago

Glad to hear it! I was also able to find what was causing the problem in parallel build. (-j20 was fine, at least.) I've rebuilt the static builds as well. I don't know if it would fix your issue, but there are at least no uses of ymm registers.

wwood commented 4 years ago

Thanks, afraid I still see dashing_s128 throwing an illegal instruction error though.

wwood commented 4 years ago

Hi,

Unfortunately it seems like dashing 0.4.0 from bioconda still fails with illegal instruction. I wonder if simply updating the bioconda package to the newest version might help things? Thanks.

dnbaker commented 4 years ago

Hi ben,

Sure, let me try to get some updates into conda. Last I was working with them, the build was a bit more complex than they tend to prefer (due to simd detection), and I'll update the builtin binaries as well.

Thanks for letting me know.

Daniel

wwood commented 4 years ago

Thanks for the efforts. Ideally, would it be possible to compile dashing in such a way that there is only a single binary, but one that calls different compiled code based on what instruction set it finds at runtime? I believe https://github.com/bbuchfink/diamond does this (both in distributed binary and through conda), for instance, but could be wrong.

dnbaker commented 4 years ago

I think it's Diamond is compiling from source by specifying architecture, e.g., https://github.com/bbuchfink/diamond/blob/master/Dockerfile#L11, which is simply using cmake and building for bare x86-64 (https://github.com/bioconda/bioconda-recipes/blob/master/recipes/diamond/build.sh).

I'll work with bioconda again for an update; it should be able to just clone recursively and make.

And I've updated the statically-linked binaries here, which I'll merge into main assuming that CI tests pass.

I've seen some tools that use fat binaries and select at runtime, but some OS's don't like running executables with any unsupported operations. And the way that Dashing is written, it would be difficult to separate every function using SIMD that way. I think the easiest way would be to install a wrapper script and all statically-linked binaries as a Python package; I'll think about that.

tl;dr:

Binaries are updated as of this commit, and I'll work with bioconda for updated releases as well as consider a the wrapper script in the next couple of weeks.

wwood commented 4 years ago

re diamond, I think that you are right - without bothering to dig through the source code, at http://www.diamondsearch.org/index.php?pages/installation/ it says

DIAMOND compiles as generic C++ code and has no particular requirements on the hardware architecture, but it makes use of the SSE instruction set of the Intel/AMD x86-64 platform if available and will run considerably faster on that platform.

which suggests it looks for architecture at runtime.

At the bioconda build file https://github.com/bioconda/bioconda-recipes/blob/master/recipes/diamond/build.sh it removes the -march=native to maximise compatibility - not sure what the docker file has to do with anything.

Anyway, that is all of little help to here, if choosing at runtime is hard to code. I had the impression that anaconda could build for multiple architectures, but am not sure on the specifics - python wrapper sounds fine though sounds like you'd be re-inventing the wheel somewhat (though more useful for non-bioconda peoples).

Instead of asking for a recursive git clone you might also just release a proper source archive, not just the one github auto-generates, and then bioconda can grab that.

Anyway, I'll stop typing half-random thoughts now. Thanks.

alienzj commented 3 years ago

Hi, I test dashing v0.4.0 installed from bioconda, still have the problem: Illegal instruction. But when I try dashing_128 and dashing_256 from the git repo release, it work, dashing_512 can't work.

Server info:

~|___(0.0) lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             48
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 61
Model name:            Intel Core Processor (Broadwell)
Stepping:              2
CPU MHz:               1995.191
BogoMIPS:              3990.38
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0-47

and:

~|___(0.0) gcc -march=native -dM -E - < /dev/null | grep -i "SSE\|AVX" | sort
#define __AVX__ 1
#define __AVX2__ 1
#define __core_avx2 1
#define __core_avx2__ 1
#define __SSE__ 1
#define __SSE2__ 1
#define __SSE2_MATH__ 1
#define __SSE3__ 1
#define __SSE4_1__ 1
#define __SSE4_2__ 1
#define __SSE_MATH__ 1
#define __SSSE3__ 1
#define __tune_core_avx2__ 1

Thanks.

dnbaker commented 3 years ago

Hi alienzj --

Thanks for the report! I'll give it a look. The flags should help. I'm not surprised that 512 failed, since your machine doesn't have avx512 support.

In the meantime, can you try git clone --recursive https://github.com/dnbaker/dashing && cd dashing && make dashing for a manual installation?

Also, on this matter, I'm planning on using a wrapper like MMSeqs2's script which compiles every possible version and uses a script to select the correct hardware version, which I'll be working on in the next week or so.

wwood commented 2 years ago

Hi again,

Sorry to bring this old thing up again. Unfortunately dashing in conda (v1.0) is a regression - I'm getting illegal instruction on EPYC servers, and dashing v2 doesn't appear to have a bioconda package at all. v0.4.0 on bioconda works though, hence why I say regression.

It would be great if the package could choose the correct architecture at runtime e.g. like DIAMOND does.

Thanks, ben

dnbaker commented 2 years ago

Hi Ben,

I agree - it would be nice. It would be possible to put together a wrapper that selects the right version.

And thanks for letting me know about illegal instruction on EPYC; the 256-bit vector version should work on it, but if it's not there's a bug. I'll see if I can figure out what the problem is. If I can't find it, I'll comb through the binary for a wrong instruction.

When I get time, I'll try to set up bioconda for Dashing2. Hopefully it's a bit more flexible than it was the last time I tried.

Thanks,

Daniel

dnbaker commented 2 years ago

If you have time, could you try the s256 binary from this folder on your EPYC server? I tried something that might have fixed the problem, but I don't have one to test on.

wwood commented 2 years ago

HI, thanks for the quick reply and fix. That s256 exe seems good to me. I guess the idea is to make a new release?