search

pmqs / DB_File

DB_File - Perl5 access to Berkeley DB version 1.x
8 stars 3 forks source link

Difficulty building on HPC (Linux) #15

Open chentytina opened 7 months ago

chentytina commented 7 months ago

Hi, I've referenced previous resolved issue #14 to cpan DB_File using:

export LDFLAGS="-L/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib"
export CPPFLAGS="-I/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/include"
export DB_FILE_LIB="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib"
export DB_FILE_INCLUDE="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/include"

Here is the CPAN output:

cpanm (App::cpanminus) 1.7047 on perl 5.026002 built for x86_64-linux-thread-multi
Work directory is /gpfs1/home/tchen/.cpanm/work/1711638695.1356205
You have make /usr/bin/make
You have /gpfs1/home/tchen/anaconda3/bin/wget
You have /usr/bin/tar: tar (GNU tar) 1.30
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
You have /usr/bin/unzip
Searching DB_File () on cpanmetadb ...
--> Working on DB_File
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/DB_File-1.859.tar.gz
-> OK
Unpacking DB_File-1.859.tar.gz
Entering DB_File-1.859
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.24)
Configuring DB_File-1.859
Running Makefile.PL
Parsing config.in...
Looks Good.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for DB_File
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have ExtUtils::MakeMaker 0 ... Yes (7.24)
Building and testing DB_File-1.859
cp DB_File.pm blib/lib/DB_File.pm
Running Mkbootstrap for DB_File ()
chmod 644 "DB_File.bs"
"/gpfs2/home/tchen/anaconda3/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- DB_File.bs blib/arch/auto/DB_File/DB_File.bs 644
gcc -c  -I/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/include -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -O2   -DVERSION=\"1.859\" -DXS_VERSION=\"1.859\" -fPIC "-I/gpfs2/home/tchen/anaconda3/lib/5.26.2/x86_64-linux-thread-multi/CORE"  -D_NOT_CORE  -DmDB_Prefix_t=size_t -DmDB_Hash_t=u_int32_t   version.c
"/gpfs2/home/tchen/anaconda3/bin/perl" "/gpfs1/home/tchen/perl5/lib/perl5/ExtUtils/xsubpp" -noprototypes -typemap '/gpfs2/home/tchen/anaconda3/lib/5.26.2/ExtUtils/typemap' -typemap '/gpfs2/home/tchen/.cpanm/work/1711638695.1356205/DB_File-1.859/typemap'  DB_File.xs > DB_File.xsc
mv DB_File.xsc DB_File.c
gcc -c  -I/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/include -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -O2   -DVERSION=\"1.859\" -DXS_VERSION=\"1.859\" -fPIC "-I/gpfs2/home/tchen/anaconda3/lib/5.26.2/x86_64-linux-thread-multi/CORE"  -D_NOT_CORE  -DmDB_Prefix_t=size_t -DmDB_Hash_t=u_int32_t   DB_File.c
rm -f blib/arch/auto/DB_File/DB_File.so
LD_RUN_PATH="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib" gcc  -shared -O2 -L/gpfs1/home/tchen/anaconda3/lib -L/usr/local/lib -fstack-protector-strong version.o DB_File.o  -o blib/arch/auto/DB_File/DB_File.so  \
   -L/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib -ldb   \

chmod 755 blib/arch/auto/DB_File/DB_File.so
Manifying 1 pod document
"/gpfs2/home/tchen/anaconda3/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- DB_File.bs blib/arch/auto/DB_File/DB_File.bs 644
PERL_DL_NONLAZY=1 "/gpfs2/home/tchen/anaconda3/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t

#   Failed test 'use DB_File;'
#   at t/000prereq.t line 23.
#     Tried to use 'DB_File'.
#     Error:  
# DB_File was build with libdb version 5.3.28,
# but you are attempting to run it with libdb version 6.2.32
# Compilation failed in require at t/000prereq.t line 23.
# BEGIN failed--compilation aborted at t/000prereq.t line 23.
Use of uninitialized value $DB_File::db_version in concatenation (.) or string at t/000prereq.t line 28.
Use of uninitialized value $DB_File::db_ver in concatenation (.) or string at t/000prereq.t line 28.
# 
# 
# DB_File version            1.859
# DB_File::db_version        
# DB_File::db_ver            
# 
# Looks like you failed 1 test of 1.
t/000prereq.t ... 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 

DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-btree.t line 32.
BEGIN failed--compilation aborted at t/db-btree.t line 32.
t/db-btree.t .... 
Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 

DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-hash.t line 17.
BEGIN failed--compilation aborted at t/db-hash.t line 17.
t/db-hash.t ..... 
Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 

DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-recno.t line 15.
BEGIN failed--compilation aborted at t/db-recno.t line 15.
t/db-recno.t .... 
Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 

DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-threads.t line 8.
BEGIN failed--compilation aborted at t/db-threads.t line 8.
t/db-threads.t .. 
Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 
t/meta-json.t ... skipped: Test::CPAN::Meta::JSON required for testing META.json
t/meta-yaml.t ... skipped: Test::CPAN::Meta required for testing META.yml
t/pod.t ......... skipped: Test::Pod 1.00 required for testing POD

Test Summary Report
-------------------
t/000prereq.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t/db-btree.t  (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/db-hash.t   (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/db-recno.t  (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/db-threads.t (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
Files=8, Tests=1,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.21 cusr  0.08 csys =  0.33 CPU)
Result: FAIL
Failed 5/8 test programs. 1/1 subtests failed.
make: *** [Makefile:1055: test_dynamic] Error 255
-> FAIL Installing DB_File failed. See /gpfs1/home/tchen/.cpanm/work/1711638695.1356205/build.log for details. Retry with --force to force install it.

Any help/insight would be greatly appreciated, I am not the best at perl but I did try to troubleshoot but to no avail, thank you!

pmqs commented 7 months ago

That should be enough. What OS are you running?

Can you try running it like this in case cpanm is interfering with your environment variables

export LDFLAGS="-L/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib"
export CPPFLAGS="-I/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/include"
export DB_FILE_LIB="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib"
export DB_FILE_INCLUDE="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/include"

wget https://cpan.metacpan.org/authors/id/P/PM/PMQS/DB_File-1.859.tar.gz
tar xvf db
cd DB_File-1.859
perl Makefile.PL
make
make test
make install
chentytina commented 7 months ago

running MacOS sonoma, but this is run on HPC. This some how has to do with the different version of libd used? I am confused why this would be the case as I had assigned the path for libd via export. still getting the same error:

"/gpfs2/home/tchen/anaconda3/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- DB_File.bs blib/arch/auto/DB_File/DB_File.bs 644
PERL_DL_NONLAZY=1 "/gpfs2/home/tchen/anaconda3/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/000prereq.t ... 1/1 
#   Failed test 'use DB_File;'
#   at t/000prereq.t line 23.
#     Tried to use 'DB_File'.
#     Error:  
# DB_File was build with libdb version 5.3.28,
# but you are attempting to run it with libdb version 6.2.32
# Compilation failed in require at t/000prereq.t line 23.
# BEGIN failed--compilation aborted at t/000prereq.t line 23.
Use of uninitialized value $DB_File::db_version in concatenation (.) or string at t/000prereq.t line 28.
Use of uninitialized value $DB_File::db_ver in concatenation (.) or string at t/000prereq.t line 28.
# 
# 
# DB_File version            1.859
# DB_File::db_version        
# DB_File::db_ver            
# 
# Looks like you failed 1 test of 1.
t/000prereq.t ... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
t/db-btree.t .... 
DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-btree.t line 32.
BEGIN failed--compilation aborted at t/db-btree.t line 32.
t/db-btree.t .... Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 
t/db-hash.t ..... 
DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-hash.t line 17.
BEGIN failed--compilation aborted at t/db-hash.t line 17.
t/db-hash.t ..... Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 
t/db-recno.t .... 
DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-recno.t line 15.
BEGIN failed--compilation aborted at t/db-recno.t line 15.
t/db-recno.t .... Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 
t/db-threads.t .. 
DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32
Compilation failed in require at t/db-threads.t line 8.
BEGIN failed--compilation aborted at t/db-threads.t line 8.
t/db-threads.t .. Dubious, test returned 2 (wstat 512, 0x200)
No subtests run 
t/meta-json.t ... skipped: Test::CPAN::Meta::JSON required for testing META.json
t/meta-yaml.t ... skipped: Test::CPAN::Meta required for testing META.yml
t/pod.t ......... skipped: Test::Pod 1.00 required for testing POD

Test Summary Report
-------------------
t/000prereq.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t/db-btree.t  (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/db-hash.t   (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/db-recno.t  (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/db-threads.t (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
Files=8, Tests=1,  1 wallclock secs ( 0.03 usr  0.02 sys +  0.20 cusr  0.08 csys =  0.33 CPU)
Result: FAIL
Failed 5/8 test programs. 1/1 subtests failed.
make: *** [Makefile:1055: test_dynamic] Error 255
pmqs commented 7 months ago

Just to be clear -- the code is running on MacOS? Correct?

Can you post the output from running perl -V please? Note the uppercase "V" in the command I need run.

MacOS isn't an OS I know that well, but I believe that dynamic libraries are handled slightly differently from Linux (which I do know).

Do you need to use Berkeley DB 5.3.28? If not - try setting the environment variables to point to the 6.2.32 files instead.

chentytina commented 7 months ago

It's running on the HPC clusters, sorry didn't clarify that well enough. Heres the output from perl -V:

Summary of my perl5 (revision 5 version 26 subversion 2) configuration:

  Platform:
    osname=linux
    osvers=4.4.0-96-generic
    archname=x86_64-linux-thread-multi
    uname='linux 1d1a811e559a 4.4.0-96-generic #119~14.04.1-ubuntu smp wed sep 13 08:40:48 utc 2017 x86_64 x86_64 x86_64 gnulinux '
    config_args='-de -Dprefix=/gpfs1/home/tchen/anaconda3 -Dusethreads -Duserelocatableinc -Dcccdlflags=-fPIC -Dldflags= -Wl,-rpath,/gpfs1/home/tchen/anaconda3/lib -L/gpfs1/home/tchen/anaconda3/lib -Ddefault_inc_excludes_dot=n -Dcc=gcc'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=undef
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='4.8.2 20140120 (Red Hat 4.8.2-15)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='gcc'
    ldflags =' -Wl,-rpath,/gpfs1/home/tchen/anaconda3/lib -L/gpfs1/home/tchen/anaconda3/lib -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /opt/rh/devtoolset-2/root/usr/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.12.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.12'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/gpfs1/home/tchen/anaconda3/lib -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
  Built under linux
  Compiled at Jul  6 2018 15:53:32
  %ENV:
    PERL5LIB="/gpfs1/home/tchen/perl5/lib/perl5:/gpfs1/home/tchen/perl5/lib/perl5:/gpfs1/home/tchen/perl5/lib/perl5"
    PERLBREW_HOME="/gpfs1/home/tchen/.perlbrew"
    PERLBREW_ROOT="/gpfs1/home/tchen/perl5/perlbrew"
    PERLBREW_SHELLRC_VERSION="0.98"
    PERL_LOCAL_LIB_ROOT="/gpfs1/home/tchen/perl5:/gpfs1/home/tchen/perl5:/gpfs1/home/tchen/perl5"
    PERL_MB_OPT="--install_base "/gpfs1/home/tchen/perl5""
    PERL_MM_OPT="INSTALL_BASE=/gpfs1/home/tchen/perl5"
  @INC:
    /gpfs1/home/tchen/perl5/lib/perl5/x86_64-linux-thread-multi
    /gpfs1/home/tchen/perl5/lib/perl5
    /gpfs1/home/tchen/perl5/lib/perl5/x86_64-linux-thread-multi
    /gpfs1/home/tchen/perl5/lib/perl5
    /gpfs1/home/tchen/perl5/lib/perl5/x86_64-linux-thread-multi
    /gpfs1/home/tchen/perl5/lib/perl5
    /gpfs2/home/tchen/anaconda3/lib/site_perl/5.26.2/x86_64-linux-thread-multi
    /gpfs2/home/tchen/anaconda3/lib/site_perl/5.26.2
    /gpfs2/home/tchen/anaconda3/lib/5.26.2/x86_64-linux-thread-multi
    /gpfs2/home/tchen/anaconda3/lib/5.26.2

I don't need to use Berkeley DB 5.3.28, but I couldn't find the 6.2.32 files. I had download Berkeley DB 5.3.28 via homebrew.

pmqs commented 7 months ago

Good, Linux.

Try

export LD_LIBRARY_PATH="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib"
chentytina commented 7 months ago

Getting the same error

DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32

Could it perhaps have something to do with the LD_PRELOAD variable?

pmqs commented 7 months ago

Getting the same error

DB_File was build with libdb version 5.3.28,
but you are attempting to run it with libdb version 6.2.32

Could it perhaps have something to do with the LD_PRELOAD variable?

Do you have LD_PRELOAD set? If so what is it set to?

Try unsetting it

chentytina commented 7 months ago

I didn't set it, it's still kept as default, should I set it to something specific?

pmqs commented 7 months ago

I didn't set it, it's still kept as default, should I set it to something specific?

It should not have a value at all. Do you mean that it is set to blank, like this?

$ echo $LD_PRELOAD

Using LD_PRELOAD is a possibility - try this, assuming there is a .so file in /gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib/l called libdb.so . 

export LD_PRELOAD="/gpfs2/home/tchen/homebrew/Cellar/berkeley-db@5/5.3.28_1/lib/libdb.so"

Be careful though - that will make ALL programs run load the Berkeley DB library. That may or may not cause issues.

chentytina commented 7 months ago

$ echo $LD_PRELOAD This is blank for me and yow were right setting LD_PRELOAD causes a lot of other issues.

pmqs commented 7 months ago

If your OS already has another version of Berkeley DB installed, try uninstalling the one you got from homebrew & see if DB_File can use the version with the OS.

If Perl complains about not being able to find Berkeley DB, try running locate libdb.so and locate db.h to see where they are installed - then set DB_FILE_LIB and DB_FILE_INCLUDE appropriately.