search

dnbaker / dashing

Fast and accurate genomic distances using HyperLogLog
GNU General Public License v3.0
160 stars 11 forks source link

compile error: ./bonsai/bonsai/include/util.h:32:25: fatal error: lazy/vector.h: No such file or directory #23

Closed yhg926 closed 5 years ago

yhg926 commented 5 years ago

when I do "make update dashing", it report errors:

cc -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib -DNDEBUG -c bonsai/zstd/zlibWrapper/gzclose.c -o bonsai/zstd/zlibWrapper/gzclose.o -lz cc -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib -DNDEBUG -c bonsai/zstd/zlibWrapper/gzlib.c -o bonsai/zstd/zlibWrapper/gzlib.o -lz cc -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib -DNDEBUG -c bonsai/zstd/zlibWrapper/zstd_zlibwrapper.c -o bonsai/zstd/zlibWrapper/zstd_zlibwrapper.o -lz cc -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib -DNDEBUG -c bonsai/zstd/zlibWrapper/gzread.c -o bonsai/zstd/zlibWrapper/gzread.o -lz cc -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib -DNDEBUG -c bonsai/zstd/zlibWrapper/gzwrite.c -o bonsai/zstd/zlibWrapper/gzwrite.o -lz g++ -O3 -funroll-loops -pipe -fno-strict-aliasing -march=native -mpclmul -DUSE_PDQSORT -DNOT_THREADSAFE -DENABLE_COMPUTED_GOTO -fopenmp -fno-rtti -std=c++14 -Wall -Wextra -Wno-char-subscripts -Wpointer-arith -Wwrite-strings -Wdisabled-optimization - Wformat -Wcast-align -Wno-unused-function -Wno-unused-parameter -pedantic -Wunused-variable -Wno-attributes -Wno-pedantic -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib bonsai/zstd/zlibWrapper/gzclose.o bonsai/zstd/zlibWrapper/gzlib.o bonsai/zstd/zlibWrapper/zstd_zlibwrapper.o bonsai/zstd/zlibWrapper/gzread.o bonsai/zstd/zlibWrapper/gzwrite.o libzstd.a bonsai/bonsai/clhash.o bonsai/klib/kthread.o -DNDEBUG src/dashing.cpp -o dashing -DZWRAP_USE_ZSTD=1 -lzstd -lz In file included from src/dashing.cpp:4:0: ./bonsai/bonsai/include/util.h:32:25: fatal error: lazy/vector.h: No such file or directory compilation terminated. Makefile:103: recipe for target 'dashing' failed make: *** [dashing] Error 1

dnbaker commented 5 years ago

Hi,

Thanks for your report!

lazy/vector.h should be in dashing/bonsai/lazy, which make update should populate vector.h, but I wonder if this might be a git version issue.

Could you try cd bonsai/lazy && git checkout master && git pull origin master && (ls vector.h || echo "Failed to check out vector.h")?

If that doesn't work, would you try deleting your copy of the repository and cloning from scratch with git clone --recursive https://github.com/dnbaker/dashing? If submodule update isn't working as expected, the recursive clone would probably fix it.

Please let me know how these work and if you need any further help. Thanks!

yhg926 commented 5 years ago

  1. I have run " cd bonsai/lazy && git checkout master && git pull origin master && (ls vector.h || echo "Failed to check out vector.h")", no error reported but there is no executable program generated in dashing folder.

  2. Delete old repository && git clone --recursive https://github.com/dnbaker/dashing && make : git submodule update --init --remote --recursive . && cd bonsai && git checkout master && git pull && make update && \ cd linear && git checkout master && git pull && cd .. && cd .. && cd distmat && git checkout master && git pull && cd .. remote: Enumerating objects: 21, done. remote: Counting objects: 100% (21/21), done. remote: Compressing objects: 100% (2/2), done. remote: Total 13 (delta 11), reused 13 (delta 11), pack-reused 0 Unpacking objects: 100% (13/13), done. From https://github.com/dnbaker/sketch c2f32d4..58d230b master -> origin/master Submodule path 'bonsai/hll': checked out '58d230b98d2c424dc03dd33be094aed3909c3bf3' Submodule path 'bonsai/kspp/klib': checked out 'f719aad5fa273424fab4b0d884d68375b7cc2520' Submodule path 'khset/klib': checked out 'f719aad5fa273424fab4b0d884d68375b7cc2520' M hll M kspp Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. make[1]: Entering directory '/home/hgyi/tools/dashing/bonsai' make update -C bonsai make[2]: Entering directory '/home/hgyi/tools/dashing/bonsai/bonsai' for i in circularqueue clhash cppitertools hll klib kspp libpopcnt linear pdqsort tinythreadpp; \ do cd ../$i && git checkout master && git pull && cd ../bonsai; done; \ cd ../hll/vec && git checkout master && git pull && cd ../../bonsai Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. /bin/sh: 2: cd: can't cd to ../cppitertools Previous HEAD position was 58d230b Simplify. Switched to branch 'master' Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. (use "git pull" to update your local branch) Updating c2f32d4..58d230b Fast-forward bf.h | 1 + ccm.h | 3 +- common.h | 604 ++------------------------------------------------------------------------------------------------------------------------------- hash.h | 624 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ hll.h | 6 ++ mult.h | 1 - src/test_revhash.cpp | 10 ++- 7 files changed, 645 insertions(+), 604 deletions(-) create mode 100644 hash.h Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. M klib Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. M hll M kspp Already on 'master' Your branch is up to date with 'origin/master'. Already up to date. Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. make[2]: Leaving directory '/home/hgyi/tools/dashing/bonsai/bonsai' make[1]: Leaving directory '/home/hgyi/tools/dashing/bonsai' Already on 'master' Your branch is up to date with 'origin/master'. Already up to date. Switched to branch 'master' Your branch is up to date with 'origin/master'. Already up to date. cd bonsai/bonsai && make libzstd.a && cp libzstd.a ../../ make[1]: Entering directory '/home/hgyi/tools/dashing/bonsai/bonsai' cd .. && git clone --recursive https://github.com/facebook/zstd Cloning into 'zstd'... remote: Enumerating objects: 9, done. remote: Counting objects: 100% (9/9), done. remote: Compressing objects: 100% (9/9), done. remote: Total 36652 (delta 1), reused 4 (delta 0), pack-reused 36643 Receiving objects: 100% (36652/36652), 18.99 MiB | 389.00 KiB/s, done. Resolving deltas: 100% (26825/26825), done. cd ../zstd && make lib && mv lib/libzstd.a ../bonsai && cd ../bonsai make[2]: Entering directory '/home/hgyi/tools/dashing/bonsai/zstd' make[3]: Entering directory '/home/hgyi/tools/dashing/bonsai/zstd/lib' cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/debug.o common/debug.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/entropy_common.o common/entropy_common.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/error_private.o common/error_private.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/fse_decompress.o common/fse_decompress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/pool.o common/pool.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/threading.o common/threading.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/xxhash.o common/xxhash.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o common/zstd_common.o common/zstd_common.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/fse_compress.o compress/fse_compress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/hist.o compress/hist.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/huf_compress.o compress/huf_compress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstd_compress.o compress/zstd_compress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstd_double_fast.o compress/zstd_double_fast.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstd_fast.o compress/zstd_fast.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstd_lazy.o compress/zstd_lazy.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstd_ldm.o compress/zstd_ldm.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstd_opt.o compress/zstd_opt.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o compress/zstdmt_compress.o compress/zstdmt_compress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o decompress/huf_decompress.o decompress/huf_decompress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o decompress/zstd_ddict.o decompress/zstd_ddict.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o decompress/zstd_decompress.o decompress/zstd_decompress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o decompress/zstd_decompress_block.o decompress/zstd_decompress_block.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o deprecated/zbuff_common.o deprecated/zbuff_common.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o deprecated/zbuff_compress.o deprecated/zbuff_compress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o deprecated/zbuff_decompress.o deprecated/zbuff_decompress.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o dictBuilder/cover.o dictBuilder/cover.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o dictBuilder/divsufsort.o dictBuilder/divsufsort.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o dictBuilder/fastcover.o dictBuilder/fastcover.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o dictBuilder/zdict.o dictBuilder/zdict.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o legacy/zstd_v05.o legacy/zstd_v05.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o legacy/zstd_v06.o legacy/zstd_v06.c cc -O3 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -I. -I./common -DXXHNAMESPACE=ZSTD -I./legacy -DZSTD_LEGACY_SUPPORT=5 -c -o legacy/zstd_v07.o legacy/zstd_v07.c compiling static library compiling dynamic library 1.4.0 creating versioned links make[3]: Leaving directory '/home/hgyi/tools/dashing/bonsai/zstd/lib' make[2]: Leaving directory '/home/hgyi/tools/dashing/bonsai/zstd' make[1]: Leaving directory '/home/hgyi/tools/dashing/bonsai/bonsai' cd bonsai/bonsai && make clhash.o && cd ../.. make[1]: Entering directory '/home/hgyi/tools/dashing/bonsai/bonsai' ls clhash.o 2>/dev/null || mv ../clhash/clhash.o . 2>/dev/null || (cd ../clhash && git checkout master && make && cd ../bonsai && ln -s ../clhash/clhash.o .) Already on 'master' Your branch is up to date with 'origin/master'. make[2]: Entering directory '/home/hgyi/tools/dashing/bonsai/clhash' cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -c ./src/clhash.c -Iinclude cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o unit ./tests/unit.c -Iinclude clhash.o g++ -fPIC -std=c++11 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o cppunit ./tests/cppunit.cpp -Iinclude clhash.o cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o benchmark ./benchmarks/benchmark.c -Iinclude clhash.o cc -fPIC -std=c99 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o example example.c -Iinclude clhash.o g++ -fPIC -std=c++11 -O3 -msse4.2 -mpclmul -march=native -funroll-loops -Wstrict-overflow -Wstrict-aliasing -Wall -Wextra -pedantic -Wshadow -o cppexample cppexample.cpp -Iinclude clhash.o make[2]: Leaving directory '/home/hgyi/tools/dashing/bonsai/clhash' make[1]: Leaving directory '/home/hgyi/tools/dashing/bonsai/bonsai' cc -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib -DNDEBUG -c bonsai/klib/kthread.c -o bonsai/klib/kthread.o -lz cd bonsai/zlib && ./configure && make && touch libz.so Checking for gcc... Checking for shared library support... Building shared library libz.so.1.2.11 with gcc. Checking for size_t... Yes. Checking for off64_t... Yes. Checking for fseeko... Yes. Checking for strerror... Yes. Checking for unistd.h... Yes. Checking for stdarg.h... Yes. Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf(). Checking for vsnprintf() in stdio.h... Yes. Checking for return value of vsnprintf()... Yes. Checking for attribute(visibility) support... Yes. make[1]: Entering directory '/home/hgyi/tools/dashing/bonsai/zlib' gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o example.o test/example.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o adler32.o adler32.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o crc32.o crc32.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o deflate.o deflate.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o infback.o infback.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inffast.o inffast.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inflate.o inflate.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inftrees.o inftrees.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o trees.o trees.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o zutil.o zutil.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o compress.o compress.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o uncompr.o uncompr.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzclose.o gzclose.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzlib.o gzlib.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzread.o gzread.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzwrite.o gzwrite.c ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example example.o -L. libz.a gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o minigzip.o test/minigzip.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip minigzip.o -L. libz.a gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/adler32.o adler32.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/crc32.o crc32.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/deflate.o deflate.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/infback.o infback.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/inffast.o inffast.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/inflate.o inflate.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/inftrees.o inftrees.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/trees.o trees.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/zutil.o zutil.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/compress.o compress.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/uncompr.o uncompr.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/gzclose.o gzclose.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/gzlib.o gzlib.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/gzread.o gzread.c gcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -DPIC -c -o objs/gzwrite.o gzwrite.c gcc -shared -Wl,-soname,libz.so.1,--version-script,zlib.map -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o libz.so.1.2.11 adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo -lc rm -f libz.so libz.so.1 ln -s libz.so.1.2.11 libz.so ln -s libz.so.1.2.11 libz.so.1 gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o examplesh example.o -L. libz.so.1.2.11 gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzipsh minigzip.o -L. libz.so.1.2.11 gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o example64.o test/example.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example64 example64.o -L. libz.a gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o minigzip64.o test/minigzip.c gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip64 minigzip64.o -L. libz.a make[1]: Leaving directory '/home/hgyi/tools/dashing/bonsai/zlib' g++ -O3 -funroll-loops -pipe -fno-strict-aliasing -march=native -mpclmul -DUSE_PDQSORT -DNOT_THREADSAFE -DENABLE_COMPUTED_GOTO -fopenmp -fno-rtti -std=c++14 -Wall -Wextra -Wno-char-subscripts -Wpointer-arith -Wwrite-strings -Wdisabled-optimization -Wformat -Wcast-align -Wno-unused-function -Wno-unused-parameter -pedantic -Wunused-variable -Wno-attributes -Wno-pedantic -DDASHING_VERSION=\"v0.3.1-7-g1692\" -Ibonsai/clhash/include -I. -Ibonsai/zlib -Ibonsai/libpopcnt -Iinclude -Ibonsai/circularqueue -Ibonsai/zstd/zlibWrapper -Ibonsai/zstd/lib/common -Ibonsai/zstd/lib -Ibonsai/hll -Ibonsai/hll/vec -Ibonsai -Ibonsai/bonsai/include/ -L. -Lbonsai/zlib libzstd.a bonsai/bonsai/clhash.o bonsai/klib/kthread.o src/dashing.cpp -o dashing -DZWRAP_USE_ZSTD=1 -lzstd -lz -DNDEBUG In file included from bonsai/hll/bbmh.h:5:0, from src/dashing.cpp:7: bonsai/hll/aesctr/wy.h:26:0: warning: ignoring #pragma messsage [-Wunknown-pragmas]

    pragma messsage("no unique address not supported")

    ^ In file included from bonsai/hll/hll.h:3:0, from ./bonsai/bonsai/include/popcnt.h:8, from ./bonsai/bonsai/include/util.h:35, from src/dashing.cpp:3: bonsai/hll/common.h: In function 'uint64_t sketch::common::sum_of_u64s(T) [with T = vector(8) long long int; uint64_t = long unsigned int]': bonsai/hll/common.h:337:39: error: '_mm512_reduce_add_epi64' was not declared in this scope return _mm512_reduce_add_epi64(val); ^ In file included from ./bonsai/bonsai/include/entropy.h:2:0, from ./bonsai/bonsai/include/encoder.h:12, from ./bonsai/bonsai/include/database.h:4, from src/dashing.cpp:4: bonsai/circularqueue/circular_buffer.h: In instantiation of 'T circ::roundup(T) [with T = unsigned int]': bonsai/circularqueue/circular_buffer.h:147:26: required from 'circ::FastCircularQueue<T, SizeType>::FastCircularQueue(SizeType) [with T = bns::ElScore<long unsigned int, long unsigned int>; SizeType = unsigned int]' ./bonsai/bonsai/include/qmap.h:46:47: required from 'bns::QueueMap<T, ScoreType>::QueueMap(std::size_t) [with T = long unsigned int; ScoreType = long unsigned int; std::size_t = long unsigned int]' ./bonsai/bonsai/include/encoder.h:153:33: required from 'bns::Encoder::Encoder(char, bns::u64, const bns::Spacer&, void, bool) [with ScoreType = bns::score::Entropy; bns::u64 = long unsigned int]' src/dashing.cpp:527:9: required from 'void bns::sketch_core(uint32_t, uint32_t, uint32_t, uint32_t, const bns::Spacer&, const std::vector<std::cxx11::basic_string >&, const string&, const string&, std::vector<sketch::cm::ccmbase_t<> >&, sketch::hll::EstimationMethod, sketch::hll::JointEstimationMethod, bns::KSeqBufferHolder&, const std::vector&, const string&, bool, bool, uint32_t, bool, bns::EncodingType) [with SketchType = sketch::hll::hllbase_t<>; uint32_t = unsigned int; std::cxx11::string = std::cxx11::basic_string]' src/dashing.cpp:649:19: required from here bonsai/circularqueue/circular_buffer.h:30:37: warning: right shift count >= width of type [-Wshift-count-overflow] CIRC_CONSTIF(sizeof(T) > 4) x|=x>>32; // Throws a warning on shift count. This is avoided in C++17, but is unavoidable in C++14 because of the lack of `if con ^ In file included from ./bonsai/bonsai/include/popcnt.h:8:0, from ./bonsai/bonsai/include/util.h:35, from src/dashing.cpp:3: bonsai/hll/hll.h: At global scope: bonsai/hll/hll.h:1649:30: warning: 'sketch::whll::WH_LOG_EXPL' defined but not used [-Wunused-variable] static const long double WH_LOG_EXPL = std::log(WH_EXPL); ^ Makefile:109: recipe for target 'dashing' failed make: *** [dashing] Error 1

dnbaker commented 5 years ago

Perfect! That's exactly what I needed to know.

You can either clone from scratch again like before or cd bonsai && git checkout master && git pull origin master.

yhg926 commented 5 years ago

i do "cd bonsai && git checkout master && git pull origin master && make ", but still report error

hgyi@sustc-HG:~/tools/dashing/bonsai$ git checkout master && git pull origin master
M   clhash
M   kspp
M   zlib
Already on 'master'
Your branch is up to date with 'origin/master'.
From https://github.com/dnbaker/bonsai
 * branch            master     -> FETCH_HEAD
Already up to date.
hgyi@sustc-HG:~/tools/dashing/bonsai$ make
make -C bonsai
make[1]: Entering directory '/home/hgyi/tools/dashing/bonsai/bonsai'
for i in circularqueue clhash hll klib kspp libpopcnt linear pdqsort tinythreadpp; \
        do cd ../$i && git checkout master && git pull && cd ../bonsai; done; \
cd ../hll/vec && git checkout master && git pull && cd ../../bonsai
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
M   klib
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
M   clhash
M   kspp
M   zlib
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
g++ -O3 -funroll-loops -pipe -fno-strict-aliasing -march=native -mpclmul   -fopenmp -fno-rtti -std=c++14 -Wall -Wextra -Wno-char-subscripts -Wpointer-arith -Wwrite-strings -Wdisabled-optimization -Wformat -Wcast-align -Wno-unused-function -Wno-unused-parameter -pedantic -DUSE_PDQSORT -Wunused-variable -Wno-ignored-attributes -Wno-cast-align -Wno-gnu-zero-variadic-macro-arguments -DBONSAI_VERSION=\"v0.2.1-146-ga002\"  -Iclhash/include -I. -I.. -I../libpopcnt -I.. -Iinclude -I../circularqueue -I../zstd/zlibWrapper -I../zstd/lib/common -I../zstd/lib  -I../hll/vec -I../hll -I../pdqsort -L.  clhash.o ../klib/kthread.o -DNDEBUG bin/kmeridx.cpp -o kmeridx -lz
In file included from ../hll/hll.h:3:0,
                 from ../bonsai/include/popcnt.h:8,
                 from ../bonsai/include/util.h:35,
                 from ../bonsai/include/hash.h:3,
                 from ../bonsai/include/encoder.h:9,
                 from ../bonsai/include/kmeridx.h:3,
                 from bin/kmeridx.cpp:1:
../hll/common.h: In function 'uint64_t sketch::common::sum_of_u64s(T) [with T = __vector(8) long long int; uint64_t = long unsigned int]':
../hll/common.h:337:39: error: '_mm512_reduce_add_epi64' was not declared in this scope
     return _mm512_reduce_add_epi64(val);
                                       ^
In file included from ../hll/common.h:36:0,
                 from ../hll/hll.h:3,
                 from ../bonsai/include/popcnt.h:8,
                 from ../bonsai/include/util.h:35,
                 from ../bonsai/include/hash.h:3,
                 from ../bonsai/include/encoder.h:9,
                 from ../bonsai/include/kmeridx.h:3,
                 from bin/kmeridx.cpp:1:
../bonsai/include/popcnt.h: In function 'unsigned int pop::detail::unrolled_bitdiff(const uint64_t*, const uint64_t*, size_t)':
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:779:27: note: in expansion of macro 'VEC_FALLTHROUGH'
                     ITER; VEC_FALLTHROUGH\
                           ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:35: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                   ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:65: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                                                 ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:95: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                                                                               ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:35: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                   ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:65: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                                                 ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:95: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                                                                               ^
../bonsai/include/popcnt.h:125:5: note: in expansion of macro 'DO_DUFF'
     DO_DUFF(len, ITER);
     ^
../bonsai/include/hash.h: In function 'unsigned int bns::dbm_hash(const char*, std::size_t)':
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:779:27: note: in expansion of macro 'VEC_FALLTHROUGH'
                     ITER; VEC_FALLTHROUGH\
                           ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:35: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                   ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:65: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                                                 ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:95: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                                                                               ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:35: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                   ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:65: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                                                 ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:95: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                                                                               ^
../bonsai/include/hash.h:107:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, HASHC);
         ^
../bonsai/include/encoder.h: In member function 'bns::u64 bns::Encoder<ScoreType>::kmer(unsigned int)':
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:779:27: note: in expansion of macro 'VEC_FALLTHROUGH'
                     ITER; VEC_FALLTHROUGH\
                           ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:35: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                   ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:65: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                                                 ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:780:95: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 7: ITER; VEC_FALLTHROUGH case 6: ITER; VEC_FALLTHROUGH case 5: ITER; VEC_FALLTHROUGH\
                                                                                               ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:35: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                   ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:65: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                                                 ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
../hll/vec/vec.h:22:29: warning: attributes at the beginning of statement are ignored [-Wattributes]
 #    define VEC_FALLTHROUGH [[gnu::fallthrough]];
                             ^
../hll/vec/vec.h:781:95: note: in expansion of macro 'VEC_FALLTHROUGH'
                     case 4: ITER; VEC_FALLTHROUGH case 3: ITER; VEC_FALLTHROUGH case 2: ITER; VEC_FALLTHROUGH case 1: ITER;\
                                                                                               ^
../bonsai/include/encoder.h:451:9: note: in expansion of macro 'DO_DUFF'
         DO_DUFF(len, ITER);
         ^
In file included from ../bonsai/include/popcnt.h:8:0,
                 from ../bonsai/include/util.h:35,
                 from ../bonsai/include/hash.h:3,
                 from ../bonsai/include/encoder.h:9,
                 from ../bonsai/include/kmeridx.h:3,
                 from bin/kmeridx.cpp:1:
../hll/hll.h: At global scope:
../hll/hll.h:1649:30: warning: 'sketch::whll::WH_LOG_EXPL' defined but not used [-Wunused-variable]
     static const long double WH_LOG_EXPL = std::log(WH_EXPL);
                              ^
cc1plus: warning: unrecognized command line option '-Wno-gnu-zero-variadic-macro-arguments'
cc1plus: warning: unrecognized command line option '-Wno-ignored-attributes'
Makefile:124: recipe for target 'kmeridx' failed
make[1]: *** [kmeridx] Error 1
make[1]: Leaving directory '/home/hgyi/tools/dashing/bonsai/bonsai'
Makefile:4: recipe for target 'all' failed
make: *** [all] Error 2
dnbaker commented 5 years ago

Thanks!

Apparently, _mm512_reduce_add_epi64 isn't supported by gcc < 7 or clang < 4, so I've corrected this in the sketch library and you should be able to build dashing now. (And you can skip trying to build bonsai; just git clone --recursive https://github.com/dnbaker/dashing/ && cd dashing && make dashing -j4.)

To save the hassle in the future, I've also starting making binary releases in case you'd rather not continue working on compilation. You can find it in release/linux/dashing_s512.gz, which is a compressed executable that should run on your platform once decompressed. (128, 256, and 512, in that folder, refer to vector widths.) If you get an 'illegal instruction' error, fall back to dashing_s256.

yhg926 commented 5 years ago

Thank you very much, it finally works!

yhg926 commented 5 years ago

Hi Daniel : I am not sure if the arguments I used is optimal, but It showed low CPU% usage on my test dataset of ~100k bacteria genomes. , any suggestion?

/usr/bin/time -v dashing sketch -p 12 -F dashing.test2.list Dashing version: v0.3.2-2-gbc95 [int bns::sketch_main(int, char)] Sketching genomes with sketch: 0/HLL/HyperLogLog [int bns::sketch_main(int, char)] Successfully finished sketching from 99750 files Command being timed: "dashing sketch -p 12 -F dashing.test2.list" User time (seconds): 7974.64 System time (seconds): 75.23 Percent of CPU this job got: 134% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:40:01 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 191220 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1 Minor (reclaiming a frame) page faults: 49065 Voluntary context switches: 419993 Involuntary context switches: 1075469 Swaps: 0 File system inputs: 282608328 File system outputs: 798000 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

dnbaker commented 5 years ago

That's strange, we typically get nearly perfect threadscaling.

I just tried the following:

[dbaker49@jhu.edu@langmead-bigmem dnb]$ time -v dashing sketch -F fnames.txt -p 100
Dashing version: v0.3.0-30-gcf25
[int bns::sketch_main(int, char**)] Sketching genomes with sketch: 0/HLL/HyperLogLog
[int bns::sketch_main(int, char**)] Successfully finished sketching from 87113 files
    Command being timed: "dashing sketch -F fnames.txt -p 100"
    User time (seconds): 26875.30
    System time (seconds): 244.56
    Percent of CPU this job got: 9690%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 4:39.85
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 13326180
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 2
    Minor (reclaiming a frame) page faults: 19129892
    Voluntary context switches: 59642
    Involuntary context switches: 175867
    Swaps: 0
    File system inputs: 24256
    File system outputs: 696904
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

What's the output of nproc on your machine? Additionally, was this using a binary release or one you compiled?

Thanks for checking!

yhg926 commented 5 years ago

My dashing was generated by the command you suggested “git clone --recursive https://github.com/dnbaker/dashing/ && cd dashing && make dashing -j4” I can run nproc tomorrow. But now my best guessing is I use a spinning disk while you are using ssd or your disk run much faster than mine. (I have test my disk IO limit is 150M/s, what is yours?)

Sent from my iPhone

On Apr 15, 2019, at 10:10 PM, Daniel Baker notifications@github.com wrote:

git clone --recursive https://github.com/dnbaker/dashing/ && cd dashing && make dashing -j4

yhg926 commented 5 years ago

hgyi@sustc-HG:/data/hgyi/work$ nproc 12

dnbaker commented 5 years ago

I'm not sure I understand why. I'm using a storage array, which is a set of RAIDed spinning disks.

The only thing which I think might help could be adding the flag --avoid-sorting. We expect a lot of variety in set sizes, so we process the largest genomes first as a form of load-balancing. Adding this could cut down on some spurious disk accesses.

dnbaker commented 5 years ago

I think I'll close this for now, since the primary issue is solved, but I'm open to future discussion with the threadscaling or any other issues as needed.

Thanks!