2.11.0 doesn't find ncbi-vdb: configure: error: required ncbi-vdb package not found

yurivict commented 3 years ago

I call configure with these arguments:

--with-ngs-sdk-prefix=/usr/local --with-ncbi-vdb-sources=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0 --with-xml2-prefix=/usr/local --with-hdf5-prefix=/usr/local CC=cc CXX=c++ TOOLS= --without-debug --prefix=/usr/local --build=amd64-portbld-freebsd12.2

It breaks after saying that it found it:

checking for ngs-sdk package...
    includes... /usr/local
    libraries... /usr/local/lib
includes: /usr/local/include
libraries: /usr/local/lib
checking for ncbi-vdb package source files and build results...
    includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
    src... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
configure: error: required ncbi-vdb package not found.
===>  Script "configure" failed unexpectedly.
Please report the problem to mzaki@niid.go.jp [maintainer] and attach the
"/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
(e.g. a /usr/local/sbin/pkg-static info -g -Ea).

klymenko commented 3 years ago

How did you configure ncbi-vdb and what did it print?

yurivict commented 3 years ago

configuration command is:

MAKE=gmake ac_cv_path_PERL=/usr/local/bin/perl ac_cv_path_PERL_PATH=/usr/local/bin/perl  PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work PATH=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin SHELL=/bin/sh CONFIG_SHELL=/bin/sh CMAKE_PREFIX_PATH="/usr/local" CONFIG_SITE=/disk-samsung/freebsd-ports/Templates/config.site lt_cv_sys_max_cmd_len=524288 \
./configure \
--with-xml2-prefix=/usr/local  --with-hdf5-prefix=/usr/local  --with-mbedtls-prefix=/usr/local  CC="cc" CXX="c++" TOOLS="clang" --prefix=/usr/local

It prints:

===>  Configuring for ncbi-vdb-2.11.0_1
Configuring NCBI-VDB package
checking system type... FreeBSD
checking machine architecture... amd64
checking for supported architecture... amd64 (64 bits) is supported
checking for supported OS... FreeBSD (bsd) is supported
checking for supported tool chain... clang tool chain is supported
checking for c++... yes
checking for Python 3... skipped
checking for fuse library... no
checking for hdf5 library... yes
checking for magic library... yes
checking for mbedtls library... yes
checking for xml2 library... yes
checking for ngs-sdk package...
    includes... ../ngs/ngs-sdk
    includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ngs/ngs-sdk
    libraries... no
configure: ngs-sdk package: found interface files but not libraries.
includes: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ngs-2.11.0/ngs-sdk
checking for ngs-java package...
    jar... configure: optional ngs-java package not found: skipped.

configure: creating 'build/Makefile.config.bsd.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb
build output path: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel
includedir: /usr/local/include
bindir: /usr/local/bin
libdir: /usr/local/lib
CC = cc -c
CPP = c++
configured with: "'--with-xml2-prefix=/usr/local' '--with-hdf5-prefix=/usr/local' '--with-mbedtls-prefix=/usr/local' 'CC=cc' 'CXX=c++' 'TOOLS=clang' '--prefix=/usr/local' '--build=amd64-portbld-freebsd12.2'"

klymenko commented 3 years ago

Try to add --with-ncbi-vdb-build=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2

yurivict commented 3 years ago

Try to add --with-ncbi-vdb-build=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2

No, this didn't help.

checking for ncbi-vdb package source files and build results...
    includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
    src... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
    libraries... no
    src...  libraries... no
    src... configure: error: required ncbi-vdb package not found.
===>  Script "configure" failed unexpectedly.

What exactly does the build need?

I have ncbi-vdb installed:

$ pkg info -l ncbi-vdb
ncbi-vdb-2.11.0_1:
    /usr/local/include/ncbi-vdb/NGS.hpp
    /usr/local/lib/libkdf5-static.a
    /usr/local/lib/libkdf5.a
    /usr/local/lib/libkdf5.a.2
    /usr/local/lib/libkdf5.a.2.11.0
    /usr/local/lib/libkdf5.so
    /usr/local/lib/libkdf5.so.2
    /usr/local/lib/libkdf5.so.2.11.0
    /usr/local/lib/libncbi-ngs-c++-static.a
    /usr/local/lib/libncbi-ngs-c++.a
    /usr/local/lib/libncbi-ngs-c++.a.2
    /usr/local/lib/libncbi-ngs-c++.a.2.11.0
    /usr/local/lib/libncbi-ngs-static.a
    /usr/local/lib/libncbi-ngs.a
    /usr/local/lib/libncbi-ngs.a.2
    /usr/local/lib/libncbi-ngs.a.2.11.0
    /usr/local/lib/libncbi-vdb-static.a
    /usr/local/lib/libncbi-vdb.a
    /usr/local/lib/libncbi-vdb.a.2
    /usr/local/lib/libncbi-vdb.a.2.11.0
    /usr/local/lib/libncbi-vdb.so
    /usr/local/lib/libncbi-vdb.so.2
    /usr/local/lib/libncbi-vdb.so.2.11.0
    /usr/local/lib/libncbi-wvdb-static.a
    /usr/local/lib/libncbi-wvdb.a
    /usr/local/lib/libncbi-wvdb.a.2
    /usr/local/lib/libncbi-wvdb.a.2.11.0
    /usr/local/lib/libncbi-wvdb.so
    /usr/local/lib/libncbi-wvdb.so.2
    /usr/local/lib/libncbi-wvdb.so.2.11.0
    /usr/local/lib/ncbi/certs.kfg
    /usr/local/lib/ncbi/default.kfg
    /usr/local/share/licenses/ncbi-vdb-2.11.0_1/LICENSE
    /usr/local/share/licenses/ncbi-vdb-2.11.0_1/PD
    /usr/local/share/licenses/ncbi-vdb-2.11.0_1/catalog.mk

Is anything missing (was forgotten to be installed) in the above file list that sra-tools needs so that it also needs a full build directory?

klymenko commented 3 years ago

Yes, sra-tools needs the full build of ncbi-vdb.

Do you have lib/libncbi-vdb.a and ilib/libkapp.a in /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel?

yurivict commented 3 years ago

Do you have lib/libncbi-vdb.a and ilib/libkapp.a in /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel?

Both files are present.

klymenko commented 3 years ago

Please add -d option to configure and post the output.

yurivict commented 3 years ago

===>  Configuring for sra-tools-2.11.0
Configuring SRA-TOOLS package
checking system type... FreeBSD
checking machine architecture... amd64
checking for supported architecture... amd64 (64 bits) is supported
checking for supported OS... FreeBSD (FreeBSD) is supported
checking for supported tool chain... clang tool chain is supported
checking for c++... 
        running c++ --help
    yes
checking for Python 3... no
checking for QMake... 
        running qmake... QMake version 3.1
 qmake
checking for fuse library... 
        running echo -e '#include <fuse.h>  \n int main() { fuse_get_context(); }\n' | clang -xc -D_FILE_OFFSET_BITS=64 -I/usr/include  - -lfuse
<stdin>:1:10: fatal error: 'fuse.h' file not found
#include <fuse.h>  
         ^~~~~~~~
1 error generated.
    no
checking for hdf5 library... '/usr/local/lib': 
        running echo -e '#include <hdf5.h>  \n int main() { H5close         (); }\n' | clang -xc  -I/usr/local/include -L/usr/local/lib - -Wl,-Bstatic -lhdf5 -Wl,-Bdynamic -ldl -lm -lz
ld: error: undefined symbol: SZ_encoder_enabled
>>> referenced by H5Z.o:(H5Z__init_package) in archive /usr/local/lib/libhdf5.a

ld: error: undefined symbol: SZ_BufftoBuffCompress
>>> referenced by H5Zszip.o:(H5Z_filter_szip) in archive /usr/local/lib/libhdf5.a

ld: error: undefined symbol: SZ_BufftoBuffDecompress
>>> referenced by H5Zszip.o:(H5Z_filter_szip) in archive /usr/local/lib/libhdf5.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
    no
checking for magic library... 
        running echo -e '#include <magic.h> \n int main() { magic_open     (0); }\n' | clang -xc  -I/usr/include  - -lmagic
    yes
        HAVE_MAGIC = 1
        MAGIC_INCDIR = /usr/include
checking for xml2 library... '/usr/local/lib': 
        running echo -e '#include <libxml/xmlreader.h>\nint main() { xmlInitParser  ( ); }\n' | clang -xc  -I/usr/local/include/libxml2 -L/usr/local/lib - -lxml2
    yes
        HAVE_XML2 = 1
        XML2_INCDIR = /usr/local/include/libxml2
        XML2_LIBDIR = /usr/local/lib
checking for ngs-sdk package...
    includes... /usr/local
    libraries... 
        checking /usr/local/FreeBSD/clang/amd64/rel/lib/libngs-c++.a
        checking /usr/local/lib/libngs-c++.a
    /usr/local/lib
includes: /usr/local/include
libraries: /usr/local/lib
checking for ncbi-vdb package source files and build results...
    includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
    src... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
    libraries... 
        checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/FreeBSD/clang/amd64/rel/lib/libncbi-vdb.a
        checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/lib/libncbi-vdb.a
        checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/FreeBSD/clang/amd64/dbg/lib/libncbi-vdb.a
    no
    src...  libraries... 
        checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/FreeBSD/clang/amd64/rel/lib/libncbi-vdb.a
        checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/lib/libncbi-vdb.a
        checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/FreeBSD/clang/amd64/dbg/lib/libncbi-vdb.a
    no
    src...      not found /usr/local/ncbi/ncbi-vdb
        not found /disk-samsung/freebsd-ports/biology/sra-tools/work/ncbi/ncbi-vdb
        ncbi-vdb: includes: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces; libs: ; ilibs: 
configure: error: required ncbi-vdb package not found.
===>  Script "configure" failed unexpectedly.
Please report the problem to mzaki@niid.go.jp [maintainer] and attach the
"/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
(e.g. a /usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

klymenko commented 3 years ago

Copy setup/os-arch.prl from ncbi-vdb to sra-tools.

yurivict commented 3 years ago

Now it passed the configure stage, and breaks at the build stage:

gcc -c  -std=c11  -DNDEBUG -m64   -DBSD -DUNIX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=bsd64 -D_ARCH_BITS=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/override -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/bsd -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/linux -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/unix -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc/amd64 -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/bsd -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/linux -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/unix -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ext -I/usr/local/include -I. -MD -o vdb-dump-tools.o /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-tools.c -D__mod_name__=tools/vdb-dump -D__file_name__=vdb-dump-tools -D__file_ext__=c
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/build/Makefile.config.bsd.x86_64:114: target '/usr/local/lib' given more than once in the same rule
gcc -c  -std=c11  -DNDEBUG -m64   -DBSD -DUNIX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=bsd64 -D_ARCH_BITS=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/override -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/bsd -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/linux -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/unix -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc/amd64 -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/bsd -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/linux -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/unix -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ext -I/usr/local/include -I. -MD -o vdb-dump-str.o /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-str.c -D__mod_name__=tools/vdb-dump -D__file_name__=vdb-dump-str -D__file_ext__=c
gmake[8]: Entering directory '/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/driver-tool/secure'
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-context.c:27:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-context.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-redir.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file.h:39:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file-v2.h:35:
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfc/refcount.h:39:10: fatal error: 'atomic32.h' file not found
#include <atomic32.h>
         ^~~~~~~~~~~~
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-str.c:34:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file.h:39:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file-v2.h:35:
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfc/refcount.h:39:10: fatal error: 'atomic32.h' file not found
#include <atomic32.h>
         ^~~~~~~~~~~~
1 error generated.
gmake[5]: *** [/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/build/Makefile.rules:31: vdb-dump-str.o] Error 1
gmake[5]: *** Waiting for unfinished jobs....
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-coldefs.c:27:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-helper.h:55:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-context.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-redir.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file.h:39:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file-v2.h:35:
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfc/refcount.h:39:10: fatal error: 'atomic32.h' file not found
#include <atomic32.h>
         ^~~~~~~~~~~~
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-tools.c:41:10: fatal error: 'bitstr.h' file not found
#include <bitstr.h>
         ^~~~~~~~~~
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/build/Makefile.config.bsd.x86_64:114: target '/usr/local/lib' given more than once in the same rule
1 error generated.

klymenko commented 3 years ago

What command did you run to build ncbi-vdb?

yurivict commented 3 years ago

configure:

MAKE=gmake ac_cv_path_PERL=/usr/local/bin/perl ac_cv_path_PERL_PATH=/usr/local/bin/perl  PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work PATH=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin SHELL=/bin/sh CONFIG_SHELL=/bin/sh CMAKE_PREFIX_PATH="/usr/local" CONFIG_SITE=/disk-samsung/freebsd-ports/Templates/config.site lt_cv_sys_max_cmd_len=524288 \
./configure \
--with-xml2-prefix=/usr/local  --with-hdf5-prefix=/usr/local  --with-mbedtls-prefix=/usr/local  CC="cc" CXX="c++" TOOLS="clang" --prefix=/usr/local

make:

PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work PATH=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe -fno-omit-frame-pointer  -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/lib -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  CPP="cpp" CPPFLAGS="-fno-omit-frame-pointer -isystem /usr/local/include"  LDFLAGS=" -fstack-protector-strong -L/usr/local/lib " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fno-omit-frame-pointer -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/lib -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -isystem /usr/local/include "  MANPREFIX="/usr/local" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" \
gmake \
CCNAME=cc CXXNAME=c++ PKGCFLAGS="-O2 -pipe -fno-omit-frame-pointer  -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/lib -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing " DESTDIR=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/stage

klymenko commented 3 years ago

I cannot build ncbi-vdb with your PR. See https://github.com/ncbi/ncbi-vdb/pull/37#issuecomment-828782328

outpaddling commented 3 years ago

Any more progress on this? I fixed a couple of build issues in sra-tools a while back: https://github.com/outpaddling/freebsd-ports-wip/tree/master/sra-tools/files Still getting link errors, though: g++ -o /usr/ports/wip/sra-tools/work/sra-tools-2.11.0/amd64-portbld-freebsd12.2/sra-tools/bsd/clang/x86_64/rel/bin/sratools.2.11.0 -DNDEBUG -m64 -L/usr/local/lib sratools.o sratools2.o imp_fasterq_dump.o imp_fastq_dump.o imp_prefetch.o imp_sam_dump.o imp_sra_pileup.o imp_srapath.o imp_vdb_dump.o cmdline.o config.o proc.o run-source.o uuid.o service.o -L/usr/ports/wip/sra-tools/work/sra-tools-2.11.0/amd64-portbld-freebsd12.2/sra-tools/bsd/clang/x86_64/rel/lib -L/usr/ports/wip/sra-tools/work/sra-tools-2.11.0/amd64-portbld-freebsd12.2/sra-tools/bsd/clang/x86_64/rel/ilib -L/usr/local/lib -L/usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib -L/usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/ilib -Wl,-Bstatic -lncbi-vdb -ljson -Wl,-Bdynamic -lepoll-shim -lncbi-vdb -lmbedtls -lexecinfo -lz -lbz2 -lpthread -lm /usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function MakeAwsAuthenticationHeader': aws-auth.c:(.text+0x76): undefined reference tovdb_mbedtls_md_info_from_type' /usr/local/bin/ld: aws-auth.c:(.text+0xb1): undefined reference to vdb_mbedtls_md_hmac' /usr/local/bin/ld: aws-auth.c:(.text+0xe6): undefined reference tovdb_mbedtls_base64_encode' /usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function Base64InIdentityDocument': aws-auth.c:(.text+0x603): undefined reference tovdb_mbedtls_base64_encode' /usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function Base64InIdentityPkcs7': aws-auth.c:(.text+0x6ba): undefined reference tovdb_mbedtls_base64_encode' /usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function MakeLocation': aws-auth.c:(.text+0x77c): undefined reference tovdb_mbedtls_base64_encode' /usr/local/bin/ld: aws-auth.c:(.text+0x7c6): undefined reference to vdb_mbedtls_base64_encode' /usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-gcp.pic.o): in functionSign_RSA_SHA256': gcp.c:(.text+0x9e): undefined reference to vdb_mbedtls_entropy_init' /usr/local/bin/ld: gcp.c:(.text+0xad): undefined reference tovdb_mbedtls_ctr_drbg_init' /usr/local/bin/ld: gcp.c:(.text+0xb7): undefined reference to vdb_mbedtls_entropy_func' /usr/local/bin/ld: gcp.c:(.text+0xc9): undefined reference tovdb_mbedtls_ctr_drbg_seed' /usr/local/bin/ld: gcp.c:(.text+0xde): undefined reference to vdb_mbedtls_pk_init' /usr/local/bin/ld: gcp.c:(.text+0xfb): undefined reference tovdb_mbedtls_pk_parse_key'

outpaddling commented 3 years ago

Looks like the mbedtls code is not being built in ncbi_vdb, even though configure is finding the library:

<<<ROOT@coral.acadix>>> /usr/ports/biology/ncbi-vdb 1083 # find work/ -name '*.a' -print -exec objdump -t '{}' \; | fgrep vdb_mbedtls_
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_base64_encode
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_md_hmac
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_md_info_from_type
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_ctr_drbg_free
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_ctr_drbg_init
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_ctr_drbg_random
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_ctr_drbg_seed
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_entropy_free
0000000000000000         *UND*  0000000000000000 vdb_mbedtls_entropy_func
...

===> ncbi-vdb-2.11.0_1 depends on shared library: libmbedtls.so - found (/usr/local/lib/libmbedtls.so) checking for mbedtls library... yes configured with: "'--with-xml2-prefix=/usr/local' '--with-hdf5-prefix=/usr/local' '--with-mbedtls-prefix=/usr/local' 'CC=cc' 'CXX=c++' 'TOOLS=clang' '--prefix=/usr/local' '--build=amd64-portbld-freebsd12.2'"

outpaddling commented 3 years ago

Looks like all the vdbmbedtls functions are defined in the bundled libs/ext/mbedtls in ncbi-vdb.

build/mbedtls-lib-name-replace.sed was used at some point (not during build) to add the vdb_ prefix to all the symbols.

So I'm guessing that the bundled mbedtls is heavily modified and cannot be easily replaced with a separate mbedtls. Andrew?

If that's the case, we'll have to revert the patches that unbundle mbedtls.

kwrodarmer commented 3 years ago

Use the included mbedtls. It has been modified to allow for static inclusion in libvdb without conflicting with other mbedtls linkage (if we included it verbatim, then anyone linking against it in addition to libvdb would get symbol conflicts).

I did not read the trail above with great detail, but why would you unbundle mbedtls? Please just build libvdb the way it was designed. Thanks!

outpaddling commented 3 years ago

Standard reasons: To get regular security updates and other improvements. Bundled libraries invariably fall behind upstream.

Thanks for verifying. I've already had a successful build with the bundled mbedtls and will post more info once I've cleaned up and tested more thoroughly.

outpaddling commented 3 years ago

I just committed to the FreeBSD ports tree. The one big issue remaining is how the build interacts with ncbi-vdb. Andrew mentioned that sra-tools requires access to the ncbi-vdb source tree. I got around this by treating ncbi-vdb like a Github submodule, building it as part of the sra-tools port instead of building and installing as a separate port. Are there plans to fully separate ncbi-vdb so dependent apps can build off a standard install?

outpaddling commented 3 years ago

I did not read the trail above with great detail, but why would you unbundle mbedtls? Please just build libvdb the way it was designed. Thanks!

A little more about this: In addition to being a good practice for security and reliability, unbundling is actually policy for many mainstream package managers, e.g.

https://fedoraproject.org/wiki/Bundled_Libraries?rd=Packaging:Bundled_Libraries https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies https://docs.freebsd.org/en/books/porters-handbook/special/#bundled-libs You can add pkgsrc to the list as well. I can't find an official document, but I'm also a pkgsrc committer and it was part of my training. We can make exceptions if unbundling proves too difficult, but we have to make a sincere effort or our colleagues will throw tomatoes at us. I'm not sure how much NCBI values inclusion of its tools in package managers like Debian packages, FreeBSD ports, Yum, etc., but making it easier to work with external libraries would make our jobs as package maintainers easier and probably increase availability of NCBI software via these deployment methods.

klymenko commented 1 year ago

We switched to cmake build since you created this issue.

outpaddling commented 1 year ago

Thanks, does this mean the build should work with separately installed ngs and ncbi-vdb now?

klymenko commented 1 year ago

It means that you probable will have to retry your steps. ngs is not needed anymore. Recommended build sequence is:

git clone https://github.com/ncbi/ncbi-vdb.git
git clone https://github.com/ncbi/sra-tools.git
cd ncbi-vdb && ./configure && make
cd ../sra-tools && ./configure && make

outpaddling commented 1 year ago

My question is, does sra-tools still require access to the ncbi-vdb source tree during build? Or can we build and install ncbi-vdb and point the sra-tools build to the installation prefix? The former is problematic for building within a package manager, but I can make it happen as I did with 2.x.

outpaddling commented 1 year ago

Also, you stated earlier that you switched to cmake, but your recommended build sequence above uses the old configure scripts. Do you recommend one method over the other? I built ncbi-vdb successfully using standard cmake commands.

klymenko commented 1 year ago

sra-tools should not require ncbi-vdb source tree.

You can configure/make or build using cmake commands. There is no preference.

outpaddling commented 1 year ago

OK, I should be able to take it from here. This can probably be closed now. I don't have the authority since Yuri opened it.

ncbi / sra-tools

2.11.0 doesn't find ncbi-vdb: configure: error: required ncbi-vdb package not found #489