./configure with or without libcurl htslib-1.17

[grenaud]

I am on Ubuntu 20.04. If I follow:

autoreconf -i
./configure
 make

I get:

hfile_s3.c:126:2: error: #error No HMAC() routine found by configure
  126 | #error No HMAC() routine found by configure
      |  ^~~~~
hfile_s3.c: In function ‘auth_header_callback’:
hfile_s3.c:404:26: error: ‘DIGEST_BUFSIZ’ undeclared (first use in this function)
  404 |     unsigned char digest[DIGEST_BUFSIZ];

If I try with:

autoreconf -i
./configure --disable-libcurl
make

Then I get:

-htslib1.17-/1.17hfile.c/:hfile.c1084::1084 :undefined  undefinedreference  referenceto  to` hfile_plugin_init_libcurl`'
hfile_plugin_init_libcurl'

[jkbonfield]

It's possible that reconfiguring doesn't adequately get spotted as a dependency. Did you do a "make clean"? It may be best to try that first before re-making to see if it's something simple.

I just tested --disable-libcurl here and it built fine.

[grenaud]

Hi James, it actually built fine on my desktop but not on our cluster, I have libhts-dev installed, do you think that is the problem?

[whitwham]

I made myself an Ubuntu 20.04 VM, installed libhts-dev, then git cloned the latest version of htslib from the repository. I installed enough libraries to get it to compile but not curl.

When I do ./configure I get this included:

checking for curl/curl.h... no checking for curl_easy_pause in -lcurl... no checking for curl_easy_init in -lcurl... no configure: WARNING: libcurl not enabled: library not found configure: WARNING: GCS support not enabled: requires libcurl support configure: WARNING: S3 support not enabled: requires libcurl support

Then make worked without an error.

What does for configure say about curl?

[grenaud]

So i tried: ` git clone --recurse-submodules https://github.com/samtools/htslib.git cd htslib/ autoreconf -i ./configure --disable-libcurl make

on our server+my local desktop. on my desktop, it builds fine. On the server, it fine until it says: gcc -fvisibility=hidden -o bgzip bgzip.o libhts.a -llzma -lbz2 -lz -lm -lpthread /usr/bin/ld: libhts.a(hfile.o): in function load_hfile_plugins': /tmp/htslib/hfile.c:1094: undefined reference tohfile_plugin_init_libcurl' collect2: error: ld returned 1 exit status make: *** [Makefile:512: bgzip] Error 1 `

any pointers?

[whitwham]

What did the config log file say about curl?

[grenaud]

same on both machines:

$ ./configure --disable-libcurl configure:5874: WARNING: GCS support not enabled: requires libcurl support configure:5898: WARNING: S3 support not enabled: requires libcurl support libcurl='disabled'

[whitwham]

There could be something strange in the server's set up that means it is looking at the wrong libhts.a when compling.

On the server, if you do: gcc -print-search-dirs -fvisibility=hidden -o bgzip bgzip.o libhts.a -llzma -lbz2 -lz -lm -lpthread

Does it come up with anything suspicious? You could also check you environment variables.

[grenaud]

they are identical between the server and desktop: install: /usr/lib/gcc/x86_64-linux-gnu/9/ programs: =/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/bin/ libraries: =/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib/:/lib/x86_64-linux-gnu/9/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/9/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../:/lib/:/usr/lib/

[whitwham]

Pretty much identical to mine too. Though the machine I'm currently on has an older version of gcc.

[daviesrob]

If you could zip up and attach the contents of your config.h, config.mk and config.log files it might help work out what's going on. And also the output of make print-config.

[grenaud]

Hello! My apologies, I was really sick last week. The output of make print-config is:

HTS_CFLAGS_AVX2 = -mavx2
HTS_CFLAGS_AVX512 = -mavx512f
HTS_CFLAGS_SSE4 = -msse4.1 -mpopcnt -mssse3
HTS_HAVE_NEON = @hts_have_neon@
LDFLAGS = -fvisibility=hidden
LIBHTS_OBJS = kfunc.o kstring.o bcf_sr_sort.o bgzf.o errmod.o faidx.o header.o hfile.o hts.o hts_expr.o hts_os.o md5.o multipart.o probaln.o realn.o regidx.o region.o sam.o sam_mods.o synced_bcf_reader.o vcf_sweep.o tbx.o textutils.o thread_pool.o vcf.o vcfutils.o cram/cram_codecs.o cram/cram_decode.o cram/cram_encode.o cram/cram_external.o cram/cram_index.o cram/cram_io.o cram/cram_stats.o cram/mFILE.o cram/open_trace_file.o cram/pooled_alloc.o cram/string_alloc.o htscodecs/htscodecs/arith_dynamic.o htscodecs/htscodecs/fqzcomp_qual.o htscodecs/htscodecs/htscodecs.o htscodecs/htscodecs/pack.o htscodecs/htscodecs/rANS_static4x16pr.o htscodecs/htscodecs/rANS_static32x16pr_avx2.o htscodecs/htscodecs/rANS_static32x16pr_avx512.o htscodecs/htscodecs/rANS_static32x16pr_sse4.o htscodecs/htscodecs/rANS_static32x16pr_neon.o htscodecs/htscodecs/rANS_static32x16pr.o htscodecs/htscodecs/rANS_static.o htscodecs/htscodecs/rle.o htscodecs/htscodecs/tokenise_name3.o htscodecs/htscodecs/utils.o
LIBS = -llzma -lbz2 -lz -lm
PLATFORM = default

config_files.tar.gz

[daviesrob]

Well, this is a complete mystery. As far as I can tell, the configuration files are correct for building with --disable-libcurl yet you're managing to build a line of code that should only be there if you define HAVE_LIBCURL. The only way I can see that happening is if you're either getting the wrong config.h file, or picking up the wrong libhts.a.

As a somewhat desperate measure, you could try this, after running configure --disable-libcurl:

gcc -H -Wall -g -O2 -fvisibility=hidden  -I.  -c -o hfile.o hfile.c

which will dump out all the include files being searched when building hfile.o. Look in the output to see if it tries to include anything called config.h that isn't ./config.h.

Then on the linker side run make libhts.a to ensure you've got the static libhts library and then run:

gcc -fvisibility=hidden -Wl,--verbose -o bgzip bgzip.o libhts.a -llzma -lbz2 -lz -lm -lpthread

That will produce a load of logging amongst which you should be able to find a line like:

attempt to open libhts.a succeeded

and a list of its contents. If it gives a path for libhts.a instead then it'll be picking up the wrong one.

Apart from that, if you can find an ubuntu 20.04 machine that works, you could try building HTSlib there and copy over the results - which might be easier.

[grenaud]

This is interesting, I used: gcc -H -Wall -g -O2 -fvisibility=hidden -I. -c -o hfile.o hfile.c saw another location in the config.h. Then I realized than in my .bashrc I had CPLUS_INCLUDE_PATH C_INCLUDE_PATH set to this different location of htslib.

I did unset CPLUS_INCLUDE_PATH unset C_INCLUDE_PATH

and it builds fine! I am closing, thanks for the help!

[jkbonfield]

Glad this was fixed. Out of interest though, where had that explicit C_INCLUDE_PATH come from? We've seen a number of compilation errors caused by this, but don't know the culprit. Is it some package framework adding it on your behalf (if so which?) or simply some poor set of instructions somewhere for another tool? (Also if so, which?) It's something really that needs fixing as it's a spectacularly bad idea and will very likely break more than just our code as it's hijacking the compiler's ability to include local files.

Edit: one other thing - out of curiosity what's in that other config.h that it was picking up? Autoconf ones often have the package name in them so you know where it came from. Generally make install shouldn't be installing config.h files as they're meant to be used for compiling and linking, and not for third parties to be using. It sounds like a badly written package has erroneously copied their config.h file over on install.

[jmarshall]

Then I realized than in my .bashrc I had CPLUS_INCLUDE_PATH C_INCLUDE_PATH set to this different location of htslib.

So if this was indeed the config.h from another htslib source directory, then this says something about the general wisdom of setting C_INCLUDE_PATH to a source directory as opposed to a …/include installation directory. At least with an installation directory you can expect that C_INCLUDE_PATH will be giving access to headers that the other package considers to be public rather than internal.

[jkbonfield]

Oops yes I forgot the bit about "different location of htslib". It would indeed have to be a source dir as we don't install config.h.

[grenaud]

@jkbonfield It was added as part of my bashrc to compile certain programs without having to rebuild htslib at each time :-)