Can't install it

[olferievm]

Hi Mark Andre,

I tried installing Katcher on my Ubuntu computer and got the following errors after executing a make command. I completely lost what is missing. I have samtools 1.13 installed using htslib 1.19.1. Any ideas?

mkdir -p obj gcc -c -o obj/functions.o src/functions.c -I ~/.local/include -L ~/.local/lib64 -I include mkdir -p obj gcc -c -o obj/main.o src/main.c -I ~/.local/include -L ~/.local/lib64 -I include mkdir -p bin gcc -o bin/katcher obj/functions.o obj/main.o -I ~/.local/include -L ~/.local/lib64 -I include -lhts -lpthread mkdir -p bin gcc src/list_kmers.c -o bin/list_kmers mkdir -p bin gcc -I ~/.local/include -L ~/.local/lib64 -I include -lhts -o bin/add_pvalues src/add_pvalues.c src/functions.c /usr/bin/ld: /tmp/ccQT80Go.o: in function sam_format_aux1': add_pvalues.c:(.text+0x9e6): undefined reference toksprintf' /usr/bin/ld: add_pvalues.c:(.text+0xa39): undefined reference to ksprintf' /usr/bin/ld: add_pvalues.c:(.text+0xff0): undefined reference tokputd' /usr/bin/ld: add_pvalues.c:(.text+0x1085): undefined reference to hts_log' /usr/bin/ld: /tmp/ccQT80Go.o: in functionbam_aux_get_str': add_pvalues.c:(.text+0x10be): undefined reference to bam_aux_get' /usr/bin/ld: /tmp/ccQT80Go.o: in functionmain': add_pvalues.c:(.text+0x1cb7): undefined reference to hts_open' /usr/bin/ld: add_pvalues.c:(.text+0x1cd4): undefined reference tohts_open' /usr/bin/ld: add_pvalues.c:(.text+0x1cdd): undefined reference to bam_init1' /usr/bin/ld: add_pvalues.c:(.text+0x1ce6): undefined reference tosam_hdr_init' /usr/bin/ld: add_pvalues.c:(.text+0x1cf6): undefined reference to sam_hdr_read' /usr/bin/ld: add_pvalues.c:(.text+0x1d4e): undefined reference tostringify_argv' /usr/bin/ld: add_pvalues.c:(.text+0x1d7e): undefined reference to sam_hdr_add_pg' /usr/bin/ld: add_pvalues.c:(.text+0x1db8): undefined reference tosam_hdr_write' /usr/bin/ld: add_pvalues.c:(.text+0x1fbe): undefined reference to bam_aux_append' /usr/bin/ld: add_pvalues.c:(.text+0x1ffc): undefined reference tosam_write1' /usr/bin/ld: add_pvalues.c:(.text+0x205c): undefined reference to sam_read1' /usr/bin/ld: add_pvalues.c:(.text+0x2070): undefined reference tohts_close' /usr/bin/ld: add_pvalues.c:(.text+0x207c): undefined reference to hts_close' /usr/bin/ld: /tmp/ccdyOD5m.o: in functionmatch_kmers': functions.c:(.text+0x505): undefined reference to bam_aux_append' /usr/bin/ld: functions.c:(.text+0x576): undefined reference tosam_write1' /usr/bin/ld: /tmp/ccdyOD5m.o: in function fillbuf': functions.c:(.text+0x628): undefined reference tosam_read1' /usr/bin/ld: /tmp/ccdyOD5m.o: in function copy_buf': functions.c:(.text+0x68d): undefined reference tobam_copy1' /usr/bin/ld: /tmp/ccdyOD5m.o: in function init_buffers': functions.c:(.text+0xc6b): undefined reference tobam_init1' /usr/bin/ld: functions.c:(.text+0xc88): undefined reference to `bam_init1' collect2: error: ld returned 1 exit status make: *** [Makefile:18: bin/add_pvalues] Error 1

[DavideGue]

Hi, I'm having exactly the same issue.

[malemay]

Hi Mikhail and Davide,

Thank you for your interest in running katcher! From the error messages you get, I am getting the impression that the linker fails to find the htslib shared library. Did you install htslib through your distro's package manager, or is it installed in some local directory? You should be looking for a file that looks like "libhts.so", which needs to be accessible by the linker.

Depending on your answer I can direct you to the proper way to install katcher.

Marc-André

[DavideGue]

Hi Marc-André, thank you for the answer. I thought htslib was installed with bcftools and samtools, but it’s not. Can I install the latest htslib release without affecting bcftools and samtools? Thank you Davide

[malemay]

Hi Davide,

Depending on how you installed bcf/samtools, it could be that these programs linked to a local copy of htslib, which was not installed globally on your machine.

If this is the case, I do not think there would be a problem installing htslib globally. Since the links to the htslib library are determined when compiling bcf/samtools, these programs should still link to the original libraries they were compiled with.

The easiest option is probably to install htslib through your distro's package manager. If your distro does not provide htslib, then you can install htslib globally on your machine by following the instructions on htslib's GitHub page and run "make install".

If you do not have root access to your machine, then you will need to install library locally.

[olferievm]

Hi Marc-Andre,

I updated to htslib1.19.1 and added LD flag in .bashrc export LD_LIBRARY_PATH=/usr/local/include/htslib:$LD_LIBRARY_PATH. Not sure if it works for Samtools.

samtools --version samtools 1.13 Using htslib 1.19.1 Copyright (C) 2021 Genome Research Ltd.

Samtools compilation details: Features: build=configure curses=yes CC: gcc CPPFLAGS: -frelease -Wdate-time -D_FORTIFY_SOURCE=2 CFLAGS: -g -O2 -ffile-prefix-map=�BUILDPATH�=. -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security LDFLAGS: -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -Wl,-z,now HTSDIR: LIBS: CURSES_LIB: -lcurses

HTSlib compilation details: Features: build=configure libcurl=yes S3=yes GCS=yes libdeflate=yes lzma=yes bzip2=yes plugins=no htscodecs=1.6.0 CC: gcc CPPFLAGS: CFLAGS: -Wall -g -O2 -fvisibility=hidden LDFLAGS: -fvisibility=hidden

HTSlib URL scheme handlers present: built-in: preload, data, file S3 Multipart Upload: s3w, s3w+https, s3w+http Amazon S3: s3+https, s3+http, s3 Google Cloud Storage: gs+http, gs+https, gs libcurl: imaps, pop3, gophers, http, smb, gopher, sftp, ftps, imap, smtp, smtps, rtsp, scp, ftp, telnet, mqtt, rtmp, ldap, https, ldaps, smbs, tftp, pop3s, dict crypt4gh-needed: crypt4gh mem: mem

On Mon, Mar 4, 2024 at 9:01 AM Marc-André Lemay @.***> wrote:

Hi Mikhail and Davide,

From the error messages you get, I am getting the impression that the linker fails to find the htslib shared library. Did you install htslib through your distro's package manager, or is it installed in some local directory? You should be looking for a file that looks like "libhts.so", which needs to be accessible by the linker.

Depending on your answer I can direct you to the proper way to install katcher.

Marc-André

— Reply to this email directly, view it on GitHub https://github.com/malemay/katcher/issues/1#issuecomment-1976645443, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIHTQDG6R4Z6U6QOVTYBYHDYWR5CLAVCNFSM6AAAAABECLACE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZWGY2DKNBUGM . You are receiving this because you authored the thread.Message ID: @.***>

[DavideGue]

Hi Marc-André, you hit the mark! The two programs are linked to a local copy of htslib! I will try to install htslib. Thank you Davide

[malemay]

Hi Mikhail,

I suspect your problem is the same as Davide. Try the solution I suggested for him and in case it does not work, I will try helping you further.

Marc-André

[DavideGue]

Hi Marc-André, I installed htslib ls -l /usr/local/lib/libhts.a -rw-r--r-- 1 root root 8692406 mar 5 07:44 /usr/local/lib/libhts.a After doing this I followed the instructions to install katcher and got exactly the same error message as before. Could make file in CFLAGS=-I /.local/include -L **/.local/lib64** -I include be the problem? Davide

[malemay]

Hi Davide,

Hard-coding library paths in the Makefile surely isn't good coding, however I have just tested compiling on a machine that does not have a ~/.local/lib or ~/.local/lib64 path and compilation worked just fine, both against my distro's globally installed hts library and against a locally compiled hts library. So I doubt this is the issue, but feel free to just modify the Makefile and see if that solves the problem for you.

It could be that the linker does not look into /usr/local/lib/ when searching for installed libraries. First, list the locations that your linker looks for using the instructions on the following StackOverflow page:

https://stackoverflow.com/questions/9922949/how-to-print-the-ldlinker-search-path

And look for /usr/local/lib in the output.

If it does not include it, then there are various ways to solve this:

https://stackoverflow.com/questions/2726993/how-to-specify-preference-of-library-path

Adding /usr/local/lib to the environment variable LD_LIBRARY_PATH could be an option. Altering the CFLAGS used for compilation is another option.

Let me know if that solves your issue.

Hi Marc-André, I installed htslib ls -l /usr/local/lib/libhts.a -rw-r--r-- 1 root root 8692406 mar 5 07:44 /usr/local/lib/libhts.a After doing this I followed the instructions to install katcher and got exactly the same error message as before. Could make file in CFLAGS=-I /.local/include -L /.local/lib64 -I include be the problem? Davide

[DavideGue]

Hi Marc-André, I verified where the linker and compiler look for shared library as follows:

gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,;  ,g' | tr \; \\012
libraries:
  /usr/lib/gcc/x86_64-linux-gnu/11/:/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/11/:/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/x86_64-linux-gnu/lib/:/usr/lib/x86_64-linux-gnu/11/:/usr/lib/x86_64-linux-gnu/:/usr/lib/:/lib/x86_64-linux-gnu/11/:/lib/x86_64-linux-gnu/:/lib/:/usr/lib/x86_64-linux-gnu/11/:/usr/lib/x86_64-linux-gnu/:/usr/lib/:/usr/x86_64-linux-gnu/lib/:/usr/lib/:/lib/:/usr/lib/

and

ld --verbose | grep SEARCH_DIR
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu64"); SEARCH_DIR("=/usr/local/lib64"); SEARCH_DIR("=/lib64"); SEARCH_DIR("=/usr/lib64"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib");

I also tried to add /usr/local/lib using LD_LIBRARY_PATH env variable, or adding the path directly to the Makefile. However all my attempts were unsuccesful Thank yoou Davide

[malemay]

Hi Davide,

I have been trying to create a static binary so you would not have to build the program yourself, however these attempts have been unsuccessful and I still do not understand why.

BUT: the issues I have been running into trying to link to the static library are exactly the same as yours. Do you happen to have a dynamic library version of libhts (libhts.so instead of libhts.a) in the location where htslib installed? Perhaps the linker is trying to link to libhts.a, which fails, whereas you should be linking to libhts.so.

As a matter of fact, my distro deletes libhts.a when installing libhts so that might be why I am not having the issue on my machine.

[DavideGue]

The content of the folder /usr/local/lib is

ls
libhts.a  libhts.so  libhts.so.1.19.1  libhts.so.3  pkgconfig  python3.10  R

WHile the content of /usr/local/include/htslib is the following:

ls
bgzf.h   hts_defs.h    hts_log.h  khash.h          kroundup.h  regidx.h             thread_pool.h
cram.h   hts_endian.h  hts_os.h   khash_str2int.h  kseq.h      sam.h                vcf.h
faidx.h  hts_expr.h    kbitset.h  klist.h          ksort.h     synced_bcf_reader.h  vcf_sweep.h
hfile.h  hts.h         kfunc.h    knetfile.h       kstring.h   tbx.h                vcfutils.h

Is there something wrong? Davide

[malemay]

Hi Davide,

This is a bit of a hack, but try renaming the file libhts.a to something else (or make a backup of it in a different folder and then delete it) and then try compiling katcher. The idea is that I want to make sure the linker does not see libhts.a. We want to link to the shared library (libhts.so).

Marc-André

[DavideGue]

Hi Marc-André, I deleted libhts.a from /usr/local/lib; however, I'm still getting the same error! Davide

[malemay]

Hi Davide,

Try deleting all files that may have been created in the katcher/bin/ and katcher/obj/ directories, and then run make again. Then please copy-and-paste the contents of the error log you get here.

[DavideGue]

Here the output: davide@administrator:~/Programmi/katcher$

mkdir -p obj
gcc -c -o obj/functions.o src/functions.c -I ~/.local/include -L ~/.local/lib64 -I include
mkdir -p obj
gcc -c -o obj/main.o src/main.c -I ~/.local/include -L ~/.local/lib64 -I include
mkdir -p bin
gcc -o bin/katcher obj/functions.o obj/main.o -I ~/.local/include -L ~/.local/lib64 -I include -lhts -lpthread
mkdir -p bin
gcc src/list_kmers.c -o bin/list_kmers
mkdir -p bin
gcc -I ~/.local/include -L ~/.local/lib64 -I include -lhts -o bin/add_pvalues src/add_pvalues.c src/functions.c
/usr/bin/ld: /tmp/ccUKoiHN.o: in function `sam_format_aux1':
add_pvalues.c:(.text+0x9e6): undefined reference to `ksprintf'
/usr/bin/ld: add_pvalues.c:(.text+0xa39): undefined reference to `ksprintf'
/usr/bin/ld: add_pvalues.c:(.text+0xff0): undefined reference to `kputd'
/usr/bin/ld: add_pvalues.c:(.text+0x1085): undefined reference to `hts_log'
/usr/bin/ld: /tmp/ccUKoiHN.o: in function `bam_aux_get_str':
add_pvalues.c:(.text+0x10be): undefined reference to `bam_aux_get'
/usr/bin/ld: /tmp/ccUKoiHN.o: in function `main':
add_pvalues.c:(.text+0x1cb7): undefined reference to `hts_open'
/usr/bin/ld: add_pvalues.c:(.text+0x1cd4): undefined reference to `hts_open'
/usr/bin/ld: add_pvalues.c:(.text+0x1cdd): undefined reference to `bam_init1'
/usr/bin/ld: add_pvalues.c:(.text+0x1ce6): undefined reference to `sam_hdr_init'
/usr/bin/ld: add_pvalues.c:(.text+0x1cf6): undefined reference to `sam_hdr_read'
/usr/bin/ld: add_pvalues.c:(.text+0x1d4e): undefined reference to `stringify_argv'
/usr/bin/ld: add_pvalues.c:(.text+0x1d7e): undefined reference to `sam_hdr_add_pg'
/usr/bin/ld: add_pvalues.c:(.text+0x1db8): undefined reference to `sam_hdr_write'
/usr/bin/ld: add_pvalues.c:(.text+0x1fbe): undefined reference to `bam_aux_append'
/usr/bin/ld: add_pvalues.c:(.text+0x1ffc): undefined reference to `sam_write1'
/usr/bin/ld: add_pvalues.c:(.text+0x205c): undefined reference to `sam_read1'
/usr/bin/ld: add_pvalues.c:(.text+0x2070): undefined reference to `hts_close'
/usr/bin/ld: add_pvalues.c:(.text+0x207c): undefined reference to `hts_close'
/usr/bin/ld: /tmp/ccoS6rLt.o: in function `match_kmers':
functions.c:(.text+0x505): undefined reference to `bam_aux_append'
/usr/bin/ld: functions.c:(.text+0x576): undefined reference to `sam_write1'
/usr/bin/ld: /tmp/ccoS6rLt.o: in function `fillbuf':
functions.c:(.text+0x628): undefined reference to `sam_read1'
/usr/bin/ld: /tmp/ccoS6rLt.o: in function `copy_buf':
functions.c:(.text+0x68d): undefined reference to `bam_copy1'
/usr/bin/ld: /tmp/ccoS6rLt.o: in function `init_buffers':
functions.c:(.text+0xc6b): undefined reference to `bam_init1'
/usr/bin/ld: functions.c:(.text+0xc88): undefined reference to `bam_init1'
collect2: error: ld returned 1 exit status
make: *** [Makefile:18: bin/add_pvalues] Errore 1

Davide

[malemay]

Is there anything in the directory "~/.local/lib64" ?

And more generally, are you sure that htslib is not installed elsewhere than /usr/local among the paths that the linker may try to look in? (The SEARCH paths that you posted above)

[DavideGue]

It looks like there is no "~/.local/lib64, and also ~/.local/include is missing. However there is /usr/lib64 containing ld-linux-x86-64.so.2 In addition to /usr/local/lib, these are the directories containg htslib files:

./miniconda3/envs/katcher_env/lib/libhts.so.3 # I also tried to use conda to install libhts and make it run, however it didn't work
./miniconda3/envs/katcher_env/lib/libhts.so
./miniconda3/envs/katcher_env/lib/libhts.a
./miniconda3/envs/katcher_env/lib/libhts.so.1.19.1
./miniconda3/pkgs/htslib-1.19.1-h81da01d_1/lib/libhts.so.3
./miniconda3/pkgs/htslib-1.19.1-h81da01d_1/lib/libhts.so
./miniconda3/pkgs/htslib-1.19.1-h81da01d_1/lib/libhts.a
./miniconda3/pkgs/htslib-1.19.1-h81da01d_1/lib/libhts.so.1.19.1
./Scrivania/libhts.a
./Scaricati/htslib-1.19.1/libhts.so.3
./Scaricati/htslib-1.19.1/libhts.so
./Scaricati/htslib-1.19.1/libhts.a
./Programmi/bedtools2/src/utils/htslib/libhts.a

and

./miniconda3/envs/katcher_env/include/htslib ./miniconda3/envs/katcher_env/libexec/htslib ./miniconda3/pkgs/htslib-1.19.1-h81da01d_1/include/htslib ./miniconda3/pkgs/htslib-1.19.1-h81da01d_1/libexec/htslib ./Scaricati/htslib-1.19.1/htslib ./Programmi/bedtools2/src/utils/htslib ./Programmi/bedtools2/src/utils/htslib/htslib

Finally there are also the following directories: ldconfig -p | grep libhts libhtscodecs.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libhtscodecs.so.2 libhts.so.3 (libc6,x86-64) => /usr/local/lib/libhts.so.3 libhts.so.3 (libc6,x86-64) => /lib/x86_64-linux-gnu/libhts.so.3 libhts.so (libc6,x86-64) => /usr/local/lib/libhts.so libhts.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libhts.so

can you see something wrong? Davide

[malemay]

Hi Davide and Mikhail,

I have published a static release of all four katcher executables. Can you try them? I have tested them on three different systems and they worked fine.

[DavideGue]

Hi Marc-André, I tried katcher and add_pvalue and they work fine, thank you. However when I try to run bedtools bamtobed on the output of add_pvalue, it does not recongnize the value in the PV tag and rounds them all to 0. I know this is probably due to beddtools, however I wondered how I can generate a datframe to plot a Manhattan plot for the associated kmers. Davide

[malemay]

Hi Davide,

I have opened a new issue regarding the processing of the output of add_pvalues, to make it easier to find for users who may have the same problem.

Regarding the executables, I am glad that they work!

Before I close this issue as solved, I would like to ask if you could try compiling katcher again from the most recent commit of the master branch? I have made some changes to the Makefile which I believe may have solved the problem.

Marc-André

[DavideGue]

Hi Marc-André, I tried to compile katcher again (from git clone https://github.com/malemay/katcher), howerver It fails again. Davide