search

raja-appuswamy / accel-align-release

A fast seed-embed-extend based sequence mapper and aligner
MIT License
22 stars 6 forks source link

Installation v1.1 #21

Closed ksahlin closed 2 years ago

ksahlin commented 2 years ago

Hi @raja-appuswamy,

Any idea on the below?

I removed the accel-align-release repository and did:

git clone --recursive https://github.com/raja-appuswamy/accel-align-release
cd accel-align-release/
make

The commands and output were:

(base) [kris@rackham2 source]$ git clone --recursive https://github.com/raja-appuswamy/accel-align-release
Cloning into 'accel-align-release'...
remote: Enumerating objects: 310, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 310 (delta 19), reused 29 (delta 14), pack-reused 270
Receiving objects: 100% (310/310), 9.76 MiB | 2.13 MiB/s, done.
Resolving deltas: 100% (187/187), done.
Submodule 'WFA' (https://github.com/smarco/WFA) registered for path 'WFA'
Cloning into 'WFA'...
remote: Enumerating objects: 5855, done.
remote: Counting objects: 100% (217/217), done.
remote: Compressing objects: 100% (136/136), done.
remote: Total 5855 (delta 129), reused 136 (delta 81), pack-reused 5638
Receiving objects: 100% (5855/5855), 99.68 MiB | 25.84 MiB/s, done.
Resolving deltas: 100% (2071/2071), done.
Submodule path 'WFA': checked out '385cf1c261a3513cd39e6db1faccf13d947056ca'
(base) [kris@rackham2 source]$ cd accel-align-release/
(base) [kris@rackham2 accel-align-release]$ make
make -C WFA all
make[1]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA'
make --directory=benchmark all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/benchmark'
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_edit.c -o ../build/benchmark_edit.o
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_gap_affine.c -o ../build/benchmark_gap_affine.o
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_gap_lineal.c -o ../build/benchmark_gap_lineal.o
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_utils.c -o ../build/benchmark_utils.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/benchmark'
make --directory=edit all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/edit'
gcc -Wall -g -O3 -fPIC -I.. -c edit_cigar.c -o ../build/edit_cigar.o
gcc -Wall -g -O3 -fPIC -I.. -c edit_dp.c -o ../build/edit_dp.o
gcc -Wall -g -O3 -fPIC -I.. -c edit_table.c -o ../build/edit_table.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/edit'
make --directory=gap_affine all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/gap_affine'
gcc -Wall -g -O3 -fPIC -I.. -c affine_penalties.c -o ../build/affine_penalties.o
gcc -Wall -g -O3 -fPIC -march=native  -I.. -c affine_wavefront.c -o ../build/affine_wavefront.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_table.c -o ../build/affine_table.o
gcc -Wall -g -O3 -fPIC -march=native  -I.. -c affine_wavefront_align.c -o ../build/affine_wavefront_align.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_backtrace.c -o ../build/affine_wavefront_backtrace.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_display.c -o ../build/affine_wavefront_display.o
gcc -Wall -g -O3 -fPIC -march=native  -I.. -c affine_wavefront_extend.c -o ../build/affine_wavefront_extend.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_penalties.c -o ../build/affine_wavefront_penalties.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_reduction.c -o ../build/affine_wavefront_reduction.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_utils.c -o ../build/affine_wavefront_utils.o
gcc -Wall -g -O3 -fPIC -I.. -c swg.c -o ../build/swg.o
gcc -Wall -g -O3 -fPIC -I.. -c wavefront_stats.c -o ../build/wavefront_stats.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/gap_affine'
make --directory=gap_lineal all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/gap_lineal'
gcc -Wall -g -O3 -fPIC -I.. -c nw.c -o ../build/nw.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/gap_lineal'
make --directory=system all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/system'
gcc -Wall -g -O3 -fPIC -I.. -c mm_allocator.c -o ../build/mm_allocator.o
gcc -Wall -g -O3 -fPIC -I.. -c profiler_counter.c -o ../build/profiler_counter.o
gcc -Wall -g -O3 -fPIC -I.. -c profiler_timer.c -o ../build/profiler_timer.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/system'
make --directory=utils all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/utils'
gcc -Wall -g -O3 -fPIC -I.. -c commons.c -o ../build/commons.o
gcc -Wall -g -O3 -fPIC -I.. -c dna_text.c -o ../build/dna_text.o
gcc -Wall -g -O3 -fPIC -I.. -c string_padded.c -o ../build/string_padded.o
gcc -Wall -g -O3 -fPIC -I.. -c vector.c -o ../build/vector.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/utils'
ar -rsc build/libwfa.a build/*.o 2> /dev/null
gcc ./WFA/build/libwfa.a -lz -ltbb  -shared -L.  build/*.o -o build/libwfa.so 
gcc: error: ./WFA/build/libwfa.a: No such file or directory
make[1]: *** [lib_wfa_so] Error 1
make[1]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA'
make: *** [WFA] Error 2
raja-appuswamy commented 2 years ago

Hello Kristoffer, Looks like there was a problem with the linker flags. Strange that it doesn't create an issue on our side. I have pushed a fix. Please try it and let me know if it works.

ksahlin commented 2 years ago

Thanks! That took care of the first issue. Let me first say that I'm a noob on make before going on about remaining issues.

After your fix I reached further, but got:

g++ -o accindex index.cpp embedding.cpp ./WFA/build/libwfa.a -lz -ltbb  -g -Wall -pthread -O3 -isystem./WFA -mavx2 -L./WFA/build -std=c++14 
In file included from index.cpp:1:0:
header.h:17:10: fatal error: zlib.h: No such file or directory
 #include <zlib.h>
          ^~~~~~~~
compilation terminated.
In file included from embedding.cpp:1:0:
header.h:17:10: fatal error: zlib.h: No such file or directory
 #include <zlib.h>
          ^~~~~~~~
compilation terminated.

As I have encountered this before, I modified the makefile with the path to zlib so that the fist lines of accelaligns makefile read:

ifneq ($(DEBUG),)
        CFLAGS=-g -Wall -pthread -O0 -DDBGPRINT -isystem./WFA -L$./WFA/build -L/proj/snic2020-15-201/anaconda3/lib -I/proj/snic2020-15-201/anaconda3/include  -std=c++14
else
        CFLAGS=-g -Wall -pthread -O3 -isystem./WFA -mavx2 -L./WFA/build -L/proj/snic2020-15-201/anaconda3/lib -I/proj/snic2020-15-201/anaconda3/include -std=c++14
endif

Then I get the following error. Could you advice me on how to solve the below error with tbb? Or are there precompiled binaries? I may look into the docker solution later.

(base) [kris@rackham2 accel-align-release]$ make
make -C WFA clean all
make[1]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA'
rm -rf bin build
make --directory=benchmark all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/benchmark'
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_edit.c -o ../build/benchmark_edit.o
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_gap_affine.c -o ../build/benchmark_gap_affine.o
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_gap_lineal.c -o ../build/benchmark_gap_lineal.o
gcc -Wall -g -O3 -fPIC -I.. -c benchmark_utils.c -o ../build/benchmark_utils.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/benchmark'
make --directory=edit all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/edit'
gcc -Wall -g -O3 -fPIC -I.. -c edit_cigar.c -o ../build/edit_cigar.o
gcc -Wall -g -O3 -fPIC -I.. -c edit_dp.c -o ../build/edit_dp.o
gcc -Wall -g -O3 -fPIC -I.. -c edit_table.c -o ../build/edit_table.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/edit'
make --directory=gap_affine all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/gap_affine'
gcc -Wall -g -O3 -fPIC -I.. -c affine_penalties.c -o ../build/affine_penalties.o
gcc -Wall -g -O3 -fPIC -march=native  -I.. -c affine_wavefront.c -o ../build/affine_wavefront.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_table.c -o ../build/affine_table.o
gcc -Wall -g -O3 -fPIC -march=native  -I.. -c affine_wavefront_align.c -o ../build/affine_wavefront_align.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_backtrace.c -o ../build/affine_wavefront_backtrace.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_display.c -o ../build/affine_wavefront_display.o
gcc -Wall -g -O3 -fPIC -march=native  -I.. -c affine_wavefront_extend.c -o ../build/affine_wavefront_extend.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_penalties.c -o ../build/affine_wavefront_penalties.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_reduction.c -o ../build/affine_wavefront_reduction.o
gcc -Wall -g -O3 -fPIC -I.. -c affine_wavefront_utils.c -o ../build/affine_wavefront_utils.o
gcc -Wall -g -O3 -fPIC -I.. -c swg.c -o ../build/swg.o
gcc -Wall -g -O3 -fPIC -I.. -c wavefront_stats.c -o ../build/wavefront_stats.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/gap_affine'
make --directory=gap_lineal all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/gap_lineal'
gcc -Wall -g -O3 -fPIC -I.. -c nw.c -o ../build/nw.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/gap_lineal'
make --directory=system all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/system'
gcc -Wall -g -O3 -fPIC -I.. -c mm_allocator.c -o ../build/mm_allocator.o
gcc -Wall -g -O3 -fPIC -I.. -c profiler_counter.c -o ../build/profiler_counter.o
gcc -Wall -g -O3 -fPIC -I.. -c profiler_timer.c -o ../build/profiler_timer.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/system'
make --directory=utils all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/utils'
gcc -Wall -g -O3 -fPIC -I.. -c commons.c -o ../build/commons.o
gcc -Wall -g -O3 -fPIC -I.. -c dna_text.c -o ../build/dna_text.o
gcc -Wall -g -O3 -fPIC -I.. -c string_padded.c -o ../build/string_padded.o
gcc -Wall -g -O3 -fPIC -I.. -c vector.c -o ../build/vector.o
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/utils'
ar -rsc build/libwfa.a build/*.o 2> /dev/null
gcc -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/proj/snic2020-15-201/anaconda3/lib -Wl,-rpath-link,/proj/snic2020-15-201/anaconda3/lib -L/proj/snic2020-15-201/anaconda3/lib -shared -L.  build/*.o -o build/libwfa.so 
make --directory=tools all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/tools'
gcc -Wall -g -O3 -fPIC generate_dataset.c -o ../bin/generate_dataset -lm -lrt 
gcc -Wall -g -O3 -fPIC -I.. align_benchmark.c ../build/*.o -o ../bin/align_benchmark -lm -lrt 
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/tools'
make[1]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA'
g++ -o accindex index.cpp embedding.cpp ./WFA/build/libwfa.a -lz -ltbb  -g -Wall -pthread -O3 -isystem./WFA -mavx2 -L./WFA/build -L/proj/snic2020-15-201/anaconda3/lib -I/proj/snic2020-15-201/anaconda3/include -std=c++14 
In file included from index.cpp:1:0:
header.h:29:10: fatal error: tbb/tbb.h: No such file or directory
 #include "tbb/tbb.h"
          ^~~~~~~~~~~
compilation terminated.
In file included from embedding.cpp:1:0:
header.h:29:10: fatal error: tbb/tbb.h: No such file or directory
 #include "tbb/tbb.h"
          ^~~~~~~~~~~
compilation terminated.
make: *** [accindex] Error 1
ksahlin commented 2 years ago

of course it signals that I don't have tbb, but I assume I installed accelalign v1.0 on the same system. I should have looked into what I did with the make file last time before removing the directory... :)

ksahlin commented 2 years ago

FWIW, I believe that I installed tbb successfully (i.e., downloading the source in the link you provide and running make finished without errors)

yanlolo commented 2 years ago

Hey Kristoffer,

The precompiled binary files: accindex-x86-64 and accalign-x86-64 are for index and align.

For the tbb errors, after download the tbb code, maybe you could try following 2 steps before make.

  1. set 'TBBROOT' in path/tbb/tbb2019_20190320oss/bin/tbbvars.sh as the path to tbb folder
  2. source path/tbb/tbb2019_20190320oss/bin/tbbvars.sh intel64

Please try and let us know if it works.

Regards, Yiqing

ksahlin commented 2 years ago

Hi @yanlolo ,

It seems the binaries are not statically built. Would it be possible to provide static binaries? After downloading the binaries:

(base) [kris@rackham2 bin]$ wget https://github.com/raja-appuswamy/accel-align-release/raw/master/accalign-x86-64
--2022-02-26 17:41:54--  https://github.com/raja-appuswamy/accel-align-release/raw/master/accalign-x86-64
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/raja-appuswamy/accel-align-release/master/accalign-x86-64 [following]
--2022-02-26 17:41:54--  https://raw.githubusercontent.com/raja-appuswamy/accel-align-release/master/accalign-x86-64
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2406248 (2.3M) [application/octet-stream]
Saving to: ‘accalign-x86-64’

100%[===============================================================================================================================================================================================================================================================================>] 2,406,248   --.-K/s   in 0.01s   

2022-02-26 17:41:55 (199 MB/s) - ‘accalign-x86-64’ saved [2406248/2406248]

(base) [kris@rackham2 bin]$ chmod +x accalign-x86-64 
(base) [kris@rackham2 bin]$ ./accalign-x86-64 
./accalign-x86-64: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./accalign-x86-64)
./accalign-x86-64: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./accalign-x86-64)
./accalign-x86-64: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by ./accalign-x86-64)
ksahlin commented 2 years ago

As for the second solution, I'm not entirely sure that I performed the steps correctly, but edited line 75 in tbbvars.sh to

TBBROOT="/home/kris/source/tbb2019_20190320oss"

then

(base) [kris@rackham2 accel-align-release]$ source /home/kris/source/tbb2019_20190320oss/bin/tbbvars.sh intel64

Then I get (some last lines of make in accel-align-release repo:

make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/utils'
ar -rsc build/libwfa.a build/*.o 2> /dev/null
gcc -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/proj/snic2020-15-201/anaconda3/lib -Wl,-rpath-link,/proj/snic2020-15-201/anaconda3/lib -L/proj/snic2020-15-201/anaconda3/lib -shared -L.  build/*.o -o build/libwfa.so 
make --directory=tools all
make[2]: Entering directory `/domus/h1/kris/source/accel-align-release/WFA/tools'
gcc -Wall -g -O3 -fPIC generate_dataset.c -o ../bin/generate_dataset -lm -lrt 
gcc -Wall -g -O3 -fPIC -I.. align_benchmark.c ../build/*.o -o ../bin/align_benchmark -lm -lrt 
make[2]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA/tools'
make[1]: Leaving directory `/domus/h1/kris/source/accel-align-release/WFA'
g++ -o accindex index.cpp embedding.cpp ./WFA/build/libwfa.a -lz -ltbb  -g -Wall -pthread -O3 -isystem./WFA -mavx2 -L./WFA/build -L/proj/snic2020-15-201/anaconda3/lib -I/proj/snic2020-15-201/anaconda3/include -std=c++14 
/crex/proj/snic2020-15-201/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -ltbb
collect2: error: ld returned 1 exit status
make: *** [accindex] Error 1
yanlolo commented 2 years ago

For the tbb error: could try sudo apt-get install libz-dev to install zlib, and git stash your changes in makefile and make again?

ksahlin commented 2 years ago

I don't have sudo access on our server, unfortunately.

yanlolo commented 2 years ago

maybe could you please try echo $LIBRARY_PATH?

e.g

$ echo $LIBRARY_PATH
/media/ssd/yan/code/tbb/tbb2019_20190320oss/lib/intel64/gcc4.7:/media/ssd/yan/code/tbb/tbb2019_20190320oss/lib/intel64/gcc4.7:/media/ssd/yan/code/tbb/tbb2019_20190320oss/lib/intel64/gcc4.7

And then check the path accordingly, and will find the tbb lib. 

$ ls /media/ssd/yan/code/tbb/tbb2019_20190320oss/lib/intel64/gcc4.7
libtbb.so    libtbb_debug.so    libtbb_preview.so    libtbb_preview_debug.so    libtbbmalloc.so    libtbbmalloc_debug.so    libtbbmalloc_proxy.so    libtbbmalloc_proxy_debug.so
libtbb.so.2  libtbb_debug.so.2  libtbb_preview.so.2  libtbb_preview_debug.so.2  libtbbmalloc.so.2  libtbbmalloc_debug.so.2  libtbbmalloc_proxy.so.2  libtbbmalloc_proxy_debug.so.2

If libtbb is in that folder, gcc should be able to find libtbb by '-ltbb'

raja-appuswamy commented 2 years ago

Hey kristoffer, While you debug your tbb issue, we have pushed a new binary with statically linked tbb. Let us know if this works. Also, regarding tbb installation, the easiest way we have found is "apt install libtbb-dev". You need dev package for headers.

ksahlin commented 2 years ago

Thanks for the binaries! I've now gotten it to work and will abandon the alternative installation plans.

Feel free to close this issue and I might come to revisit this if needed.

raja-appuswamy commented 2 years ago

Alright. Its great that you shared these compilation issues with us. Thanks for testing it out. We really appreciate the feedback. We'll make sure we upload static binaries here on out until we switch out of TBB, which is planned in the future.