Open brianv0 opened 4 years ago
cc @isuruf this looks somewhat serious and I'm stumped.
Should we pull the 9.3.0 builds for now?
Thank you!
@brianv0, can you confirm that this issue is fixed?
awesome, running our CI right now, I'll get back to you today
I still had the issue, but I'm a bit confused why. I did see libquadmath.so.0
[2020-08-01T20:58:48.010862Z] /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/x86_64-conda_cos6-linux-gnu-c++ -o tests/background -Wl,-rpath-link -Wl,/lsstsw/build/afw/lib:/lsstsw/stack/73733da/Linux64/astshim/20.0.0+f45b7d88f4/lib:/lsstsw/stack/73733da/Linux64/geom/20.0.0-2-gec03fae+4d82fa78bb/lib:/lsstsw/stack/73733da/Linux64/sphgeom/20.0.0-3-gd2e950e+f45b7d88f4/lib:/lsstsw/stack/73733da/Linux64/pex_config/20.0.0+9457a788fe/lib:/lsstsw/stack/73733da/Linux64/daf_persistence/20.0.0+ef667cdf05/lib:/lsstsw/stack/73733da/Linux64/pex_policy/20.0.0+eb0b68971d/lib:/lsstsw/stack/73733da/Linux64/log/20.0.0-1-gd1c87d7+54d12611dd/lib:/lsstsw/stack/73733da/Linux64/daf_base/20.0.0-1-g8a53f90+e0c9560612/lib:/lsstsw/stack/73733da/Linux64/utils/20.0.0+95dc160249/lib:/lsstsw/stack/73733da/Linux64/pex_exceptions/20.0.0+117106b506/lib:/lsstsw/stack/73733da/Linux64/base/20.0.0-4-g4a2362f+f45b7d88f4/lib -fno-lto -Wl,-rpath=/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib tests/background.o -Llib -L/lsstsw/stack/73733da/Linux64/astshim/20.0.0+f45b7d88f4/lib -L/lsstsw/stack/73733da/Linux64/geom/20.0.0-2-gec03fae+4d82fa78bb/lib -L/lsstsw/stack/73733da/Linux64/sphgeom/20.0.0-3-gd2e950e+f45b7d88f4/lib -L/lsstsw/stack/73733da/Linux64/daf_persistence/20.0.0+ef667cdf05/lib -L/lsstsw/stack/73733da/Linux64/pex_policy/20.0.0+eb0b68971d/lib -L/lsstsw/stack/73733da/Linux64/log/20.0.0-1-gd1c87d7+54d12611dd/lib -L/lsstsw/stack/73733da/Linux64/daf_base/20.0.0-1-g8a53f90+e0c9560612/lib -L/lsstsw/stack/73733da/Linux64/utils/20.0.0+95dc160249/lib -L/lsstsw/stack/73733da/Linux64/pex_exceptions/20.0.0+117106b506/lib -L/lsstsw/stack/73733da/Linux64/base/20.0.0-4-g4a2362f+f45b7d88f4/lib -L/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib -L/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/python3.7/config-3.7m-x86_64-linux-gnu -lafw -lastshim -last -last_pal -last_grf_2.0 -last_grf_3.2 -last_grf_5.6 -last_grf3d -last_pass2 -last_err -lm -lboost_serialization -lcfitsio -lgsl -lgslcblas -lMinuit2 -lgeom -lsphgeom -lfftw3f -lfftw3 -lboost_math_c99 -ldaf_persistence -lpex_policy -lboost_filesystem -lboost_system -llog -llog4cxx -ldaf_base -lboost_regex -lutils -lpex_exceptions -lbase -lpthread -lboost_unit_test_framework
[2020-08-01T20:58:48.630943Z] /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: warning: libquadmath.so.0, needed by /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/./libgfortran.so.4, not found (try using -rpath or -rpath-link)
[2020-08-01T20:58:48.631211Z] /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/./libgfortran.so.4: undefined reference to `fmodq@QUADMATH_1.0'
Cleaned up version:
[2020-08-01T20:58:48.010862Z] /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/x86_64-conda_cos6-linux-gnu-c++ \
-o tests/background \
-Wl,-rpath-link \
-Wl,/lsstsw/build/afw/lib:/lsstsw/stack/73733da/Linux64/astshim/20.0.0+f45b7d88f4/lib:/lsstsw/stack/73733da/Linux64/geom/20.0.0-2-gec03fae+4d82fa78bb/lib:/lsstsw/stack/73733da/Linux64/sphgeom/20.0.0-3-gd2e950e+f45b7d88f4/lib:/lsstsw/stack/73733da/Linux64/pex_config/20.0.0+9457a788fe/lib:/lsstsw/stack/73733da/Linux64/daf_persistence/20.0.0+ef667cdf05/lib:/lsstsw/stack/73733da/Linux64/pex_policy/20.0.0+eb0b68971d/lib:/lsstsw/stack/73733da/Linux64/log/20.0.0-1-gd1c87d7+54d12611dd/lib:/lsstsw/stack/73733da/Linux64/daf_base/20.0.0-1-g8a53f90+e0c9560612/lib:/lsstsw/stack/73733da/Linux64/utils/20.0.0+95dc160249/lib:/lsstsw/stack/73733da/Linux64/pex_exceptions/20.0.0+117106b506/lib:/lsstsw/stack/73733da/Linux64/base/20.0.0-4-g4a2362f+f45b7d88f4/lib \
-fno-lto \
##### This is the conda-env directory
-Wl,-rpath=/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib \
tests/background.o \
-Llib -L/lsstsw/stack/73733da/Linux64/astshim/20.0.0+f45b7d88f4/lib -L/lsstsw/stack/73733da/Linux64/geom/20.0.0-2-gec03fae+4d82fa78bb/lib \
-L/lsstsw/stack/73733da/Linux64/sphgeom/20.0.0-3-gd2e950e+f45b7d88f4/lib -L/lsstsw/stack/73733da/Linux64/daf_persistence/20.0.0+ef667cdf05/lib \
-L/lsstsw/stack/73733da/Linux64/pex_policy/20.0.0+eb0b68971d/lib -L/lsstsw/stack/73733da/Linux64/log/20.0.0-1-gd1c87d7+54d12611dd/lib \
-L/lsstsw/stack/73733da/Linux64/daf_base/20.0.0-1-g8a53f90+e0c9560612/lib -L/lsstsw/stack/73733da/Linux64/utils/20.0.0+95dc160249/lib \
-L/lsstsw/stack/73733da/Linux64/pex_exceptions/20.0.0+117106b506/lib -L/lsstsw/stack/73733da/Linux64/base/20.0.0-4-g4a2362f+f45b7d88f4/lib \
-L/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib -L/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/python3.7/config-3.7m-x86_64-linux-gnu\
-lafw -lastshim -last -last_pal -last_grf_2.0 -last_grf_3.2 -last_grf_5.6 -last_grf3d -last_pass2 -last_err -lm -lboost_serialization -lcfitsio \
-lgsl -lgslcblas -lMinuit2 -lgeom -lsphgeom -lfftw3f -lfftw3 -lboost_math_c99 -ldaf_persistence -lpex_policy -lboost_filesystem -lboost_system -llog \
-llog4cxx -ldaf_base -lboost_regex -lutils -lpex_exceptions -lbase -lpthread -lboost_unit_test_framework
[2020-08-01T20:58:48.630943Z] /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: warning: libquadmath.so.0, needed by /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/./libgfortran.so.4, not found (try using -rpath or -rpath-link)
[2020-08-01T20:58:48.631211Z] /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/./libgfortran.so.4: undefined reference to `fmodq@QUADMATH_1.0'
You'll see the line above with the env's lib directory:
-Wl,-rpath=/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib
And we verify that libquadmath.so.0 is in that directory:
(lsst-scipipe-tickets-DM-26148.73733da) root@9014056e77a5:/lsstsw/build/afw# ls /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquad*
/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquadmath.so /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquadmath.so.0 /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquadmath.so.0.0.0
Do I need to adjust my flags?
Can you try looking at which files are different in the two environments?
@brianv0 can you ls -lah the quadmath SO's in lib? I'd be good to make sure the links and files are setup correctly.
Also which compilers are you using? I see conda_cos6 in the HOST. You should not be seeing this if you are using the latest compilers.
# which $CC
/lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin/x86_64-conda_cos6-linux-gnu-cc
# ls -al /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/bin | grep gcc
lrwxrwxrwx 1 root root 31 Aug 1 19:59 cc -> x86_64-conda_cos6-linux-gnu-gcc
lrwxrwxrwx 1 root root 31 Aug 1 19:59 gcc -> x86_64-conda_cos6-linux-gnu-gcc
lrwxrwxrwx 1 root root 34 Aug 1 19:59 gcc-ar -> x86_64-conda_cos6-linux-gnu-gcc-ar
lrwxrwxrwx 1 root root 34 Aug 1 19:59 gcc-nm -> x86_64-conda_cos6-linux-gnu-gcc-nm
lrwxrwxrwx 1 root root 38 Aug 1 19:59 gcc-ranlib -> x86_64-conda_cos6-linux-gnu-gcc-ranlib
lrwxrwxrwx 1 root root 31 Aug 1 19:59 x86_64-conda_cos6-linux-gnu-cc -> x86_64-conda_cos6-linux-gnu-gcc
-rwxrwxr-x 2 root root 904032 May 14 14:41 x86_64-conda_cos6-linux-gnu-gcc
-rwxrwxr-x 2 root root 25520 May 14 14:41 x86_64-conda_cos6-linux-gnu-gcc-ar
-rwxrwxr-x 2 root root 25488 May 14 14:41 x86_64-conda_cos6-linux-gnu-gcc-nm
-rwxrwxr-x 2 root root 25488 May 14 14:41 x86_64-conda_cos6-linux-gnu-gcc-ranlib
#
# ls -lah /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libqua*
lrwxrwxrwx 1 root root 20 Aug 1 19:59 /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquadmath.so -> libquadmath.so.0.0.0
lrwxrwxrwx 1 root root 20 Aug 1 19:59 /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquadmath.so.0 -> libquadmath.so.0.0.0
-rwxrwxr-x 2 root root 945K Aug 1 08:37 /lsstsw/miniconda/envs/lsst-scipipe-tickets-DM-26148.73733da/lib/libquadmath.so.0.0.0
#
# conda list | grep "gfortr\|gcc"
_libgcc_mutex 0.1 conda_forge conda-forge
gcc_impl_linux-64 7.5.0 hd420e75_6 conda-forge
gcc_linux-64 7.5.0 h09487f9_20 conda-forge
gfortran_impl_linux-64 7.5.0 hdf63c60_6 conda-forge
gfortran_linux-64 7.5.0 h09487f9_20 conda-forge
libgcc-ng 9.3.0 h24d8f2e_13 conda-forge
libgfortran 3.0.0 1 conda-forge
libgfortran-ng 7.5.0 hdf63c60_13 conda-forge
# conda list | grep "compiler"
c-compiler 1.1.1 h516909a_0 conda-forge
compilers 1.1.1 0 conda-forge
cxx-compiler 1.1.1 hc9558a2_0 conda-forge
fortran-compiler 1.1.1 he991be0_0 conda-forge
Those builds are very old. They should work, but maybe they are broken. Can you try builds of *_impl_linux-64
with build number 13 or 14?
also, what is pulling in this?
libgfortran 3.0.0 1 conda-forge
IDK where that other libgfortran comes from...
Issue: When using the
compilers
package to compile code, I get linking errors against libquadmath.so.0 on linux seemingly with 9.3.0 build 11 which I did not get a week ago with a slightly different environment.Below is a snippet from our logs:
Here is a diff of my environment export between versions which work and don't work:
Environment (
conda list
):Details about
conda
and system (conda info
):