Compile error for multi-threading mode

[ryao-mdanderson]

Hello Delly author:

When I tried to compile multi-threading mode on our HPC cluster, RHEL 7.9, I see these output and hit the errors: we use gcc/9.3.0 and openmpi/4.1.1

$ make PARALLEL=1 -B src/delly if [ -r src/htslib/Makefile ]; then cd src/htslib && autoreconf -i && ./configure --disable-s3 --disable-gcs --disable-libcurl --disable-plugins && make && make lib-static && cd ../../ && touch .htslib; fi ....... htscodecs_bundled.mk:2: htscodecs_bundled.mk: Too many open files in system ...... libhts.a(hts.o): In function hts_feature_string': /rsrch3/home/itops/ryao/delly/delly/src/htslib/hts.c:227: undefined reference tohtscodecs_version' libhts.a(hts.o): In function hts_test_feature': /rsrch3/home/itops/ryao/delly/delly/src/htslib/hts.c:149: undefined reference tohtscodecs_version' libhts.a(cram_io.o): In function cram_compress_by_method': /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1866: undefined reference toarith_compress_to' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1846: undefined reference to rans_compress_4x16' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1878: undefined reference toencode_names' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1809: undefined reference to fqz_compress' libhts.a(cram_io.o): In functioncram_uncompress_block': /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1685: undefined reference to rans_uncompress_4x16' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1705: undefined reference toarith_uncompress_to' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1672: undefined reference to fqz_decompress' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1725: undefined reference todecode_names' libhts.a(hfile.o): In function load_hfile_plugins': /rsrch3/home/itops/ryao/delly/delly/src/htslib/hfile.c:1084: undefined reference tohfile_plugin_init_libcurl' collect2: error: ld returned 1 exit status make[1]: [htsfile] Error 1 make[1]: Leaving directory `/rsrch3/home/itops/ryao/delly/delly/src/htslib' make: [.htslib] Error 2

Do you have any suggestion? Need any modification in the Makefile?

Thank you very much for your help.

Rong Yao

[tobiasrausch]

Looks like a problem while compiling htslib (one of the dependencies of delly). Have you done a recursive checkout? Try recompiling using:

make clean && make distclean && make all

[ryao-mdanderson]

Hello @tobiasrausch I tried your command, the problem is still from htslib, I saw the followings

In function cram_compress_by_method': /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1878: undefined reference toencode_names' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1846: undefined reference to rans_compress_4x16' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1866: undefined reference toarith_compress_to' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1809: undefined reference to fqz_compress' libhts.a(cram_io.o): In functioncram_uncompress_block': /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1725: undefined reference to decode_names' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1685: undefined reference torans_uncompress_4x16' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1705: undefined reference to arith_uncompress_to' /rsrch3/home/itops/ryao/delly/delly/src/htslib/cram/cram_io.c:1672: undefined reference tofqz_decompress' libhts.a(hts.o): In function hts_feature_string': /rsrch3/home/itops/ryao/delly/delly/src/htslib/hts.c:227: undefined reference tohtscodecs_version' libhts.a(hts.o): In function hts_test_feature': /rsrch3/home/itops/ryao/delly/delly/src/htslib/hts.c:149: undefined reference tohtscodecs_version' collect2: error: ld returned 1 exit status make[1]: [htsfile] Error 1 make[1]: Leaving directory `/rsrch3/home/itops/ryao/delly/delly/src/htslib' make: [.htslib] Error 2

Any suggestion?

Regards, Rong

[ryao-mdanderson]

the goal is to build a multi threading mode, should it have to be build with the command: make PARALLEL=1 -B src/delly ?

Does bioconda package, a sif file or a docker container have a multi threading mode built?

Thanks!

[tobiasrausch]

There is a parallel docker container: https://hub.docker.com/u/dellytools

[ryao-mdanderson]

Hello @tobiasrausch 👍

I pulled a delly-parallel container from docker: singularity pull docker://dellytools/delly-parallel this generates a singularity file delly-parallel_latest.sif.

I am able to invoke the delly helper message: singularity exec delly-parallel_latest.sif delly -h

---

Program: Delly This is free software, and you are welcome to redistribute it under certain conditions (BSD License); for license details use '-l'. This program comes with ABSOLUTELY NO WARRANTY; for details use '-w'.

Delly (Version: 1.1.5) Contact: Tobias Rausch (rausch@embl.de)

---

Usage: delly

Short-read SV calling: call discover and genotype structural variants merge merge structural variants across VCF/BCF files and within a single VCF/BCF file filter filter somatic or germline structural variants

Long-read SV calling: lr long-read SV discovery

Copy-number variant calling: cnv discover and genotype copy-number variants classify classify somatic or germline copy-number variants

Please advise how do I use parallel mode? I seem not to find an example in the readme.

Thank you! Rong

[lmanchon]

same issue, compilation failed, some references are missing in htslib:

git clone --recursive https://github.com/dellytools/delly.git cd delly make PARALLEL=1 -B src/delly

gcc -shared -Wl,-soname,libhts.so.3 -fvisibility=hidden -o libhts.so kfunc.pico kstring.pico bcf_sr_sort.pico bgzf.pico errmod.pico faidx.pico header.pico hfile.pico hts.pico hts_expr.pico hts_os.pico md5.pico multipart.pico probaln.pico realn.pico regidx.pico region.pico sam.pico synced_bcf_reader.pico vcf_sweep.pico tbx.pico textutils.pico thread_pool.pico vcf.pico vcfutils.pico cram/cram_codecs.pico cram/cram_decode.pico cram/cram_encode.pico cram/cram_external.pico cram/cram_index.pico cram/cram_io.pico cram/cram_stats.pico cram/mFILE.pico cram/open_trace_file.pico cram/pooled_alloc.pico cram/string_alloc.pico -llzma -lbz2 -lz -lm -lpthread ln -sf libhts.so libhts.so.3 gcc -Wall -g -O2 -fvisibility=hidden -I. -c -o bgzip.o bgzip.c gcc -fvisibility=hidden -o bgzip bgzip.o libhts.a -llzma -lbz2 -lz -lm -lpthread libhts.a(hts.o): In function hts_feature_string': delly/src/htslib/hts.c:227: undefined reference tohtscodecs_version' libhts.a(hts.o): In function hts_test_feature': delly/src/htslib/hts.c:149: undefined reference tohtscodecs_version' libhts.a(cram_io.o): In function cram_compress_by_method': delly/src/htslib/cram/cram_io.c:1866: undefined reference toarith_compress_to' delly/src/htslib/cram/cram_io.c:1846: undefined reference to rans_compress_4x16' delly/src/htslib/cram/cram_io.c:1826: undefined reference torans_compress' delly/src/htslib/cram/cram_io.c:1878: undefined reference to encode_names' delly/src/htslib/cram/cram_io.c:1809: undefined reference tofqz_compress' libhts.a(cram_io.o): In function cram_uncompress_block': delly/src/htslib/cram/cram_io.c:1654: undefined reference torans_uncompress' delly/src/htslib/cram/cram_io.c:1685: undefined reference to rans_uncompress_4x16' delly/src/htslib/cram/cram_io.c:1705: undefined reference toarith_uncompress_to' delly/src/htslib/cram/cram_io.c:1672: undefined reference to fqz_decompress' delly/src/htslib/cram/cram_io.c:1725: undefined reference todecode_names' libhts.a(cram_codecs.o): In function cram_xpack_encode_flush': delly/src/htslib/cram/cram_codecs.c:1448: undefined reference tohts_pack' libhts.a(cram_codecs.o): In function cram_xrle_decode_expand_char': delly/src/htslib/cram/cram_codecs.c:2033: undefined reference torle_decode' libhts.a(cram_codecs.o): In function cram_xrle_encode_flush': delly/src/htslib/cram/cram_codecs.c:2203: undefined reference torle_encode' libhts.a(cram_codecs.o): In function cram_xpack_decode_expand_char': delly/src/htslib/cram/cram_codecs.c:1328: undefined reference tohts_unpack' collect2: error: ld returned 1 exit status make[1]: [bgzip] Error 1 make[1]: Leaving directory `delly/src/htslib' make: [.htslib] Error 2

[tobiasrausch]

Probably some issue on your host system, like an outdated lzma library. We build the parallel version on every commit using CI/CD: https://github.com/dellytools/delly/blob/main/parallel/Dockerfile

[lmanchon]

yes probably, docker is a solution, other solution: buid a static version of the parallel version, right ?