search

ACCESS-NRI / spack-packages

Spack package repository maintained by ACCESS-NRI
Apache License 2.0
5 stars 4 forks source link

cice5: shared libraries not found #14

Closed harshula closed 1 year ago

harshula commented 1 year ago 
$ ldd opt/spack/linux-rocky8-x86_64/intel-2019.5.281/cice5-master-byagur66qd4y7za7vauskh5w34t3cb54/bin/cice_auscom_1440x1080_48x40_480p.exe | grep not
    libnetcdf.so.18 => not found
    libnetcdff.so.7 => not found
harshula commented 1 year ago

Non-spack build:

$ ldd bin/cice_auscom_1440x1080_480p_edcfa6f_libaccessom2_d750b4b.exe | grep netcdf
    libnetcdf_ompi3.so.18 => /apps/netcdf/4.7.4p/lib/libnetcdf_ompi3.so.18 (0x00007f04ac271000)
    libnetcdff_ompi3_Intel.so.7 => /apps/netcdf/4.7.4p/lib/libnetcdff_ompi3_Intel.so.7 (0x00007f04abc9e000)
harshula commented 1 year ago

My understanding is that we do not want pnetcdf support. We want hdf5 support for parallel I/O

Variant pnetcdf is disabled in the SPD:

depends_on("parallelio~pnetcdf~timing~shared")

The SPD of netcdf-c contains:

        if "+parallel-netcdf" in self.spec:
            config_args.append("--enable-pnetcdf")
            pnetcdf = self.spec["parallel-netcdf"]
            cppflags.append(pnetcdf.headers.cpp_flags)
            # TODO: change to pnetcdf.libs.search_flags once 'parallel-netcdf'
            # package gets custom implementation of 'libs'
            ldflags.append("-L" + pnetcdf.prefix.lib)
        else:
            config_args.append("--disable-pnetcdf")

https://docs.unidata.ucar.edu/nug/current/getting_and_building_netcdf.html#build_parallel

harshula commented 1 year ago

Notes

Issue is not caused by Commit 540f54f260eed64e0c33e52504c4f5e1992fea22 (cice5: simplify build script and Makefile snippet). I checked Spack compiles on Gadi with and without this patch.

harshula commented 1 year ago

[Updated 2023/07/21]

An Issue https://github.com/ACCESS-NRI/spack_packages/issues/19 addresses the following (and is not directly relevant to this Issue):

This subtle difference was not captured in the SPD!

cice5: bld/config.nci.auscom.1440x1080

module load netcdf/4.7.4p

mom5: bin/environs.nci

module load netcdf/4.7.4
--- /apps/netcdf/4.7.4/bin/nc-config --all
+++ /apps/netcdf/4.7.4p/bin/nc-config --all
@@ -1,9 +1,9 @@

 This netCDF 4.7.4 has been built with the following features: 

-  --cc            -> gcc
-  --cflags        -> -I/apps/netcdf/4.7.4/include 
-  --libs          -> -L/apps/netcdf/4.7.4/lib -lnetcdf
+  --cc            -> mpicc
+  --cflags        -> -I/apps/netcdf/4.7.4p/include 
+  --libs          -> -L/apps/netcdf/4.7.4p/lib/ompi2 -lnetcdf
   --static        -> -ljpeg -lmfhdf -ldf -ljpeg -lhdf5_hl -lhdf5 -lm -ldl -lz -lcurl 

   --has-c++       -> no
@@ -22,13 +22,13 @@
   --has-hdf4      -> yes
   --has-logging   -> no
   --has-pnetcdf   -> no
-  --has-szlib     -> no
+  --has-szlib     -> yes
   --has-cdf5      -> yes
-  --has-parallel4 -> no
-  --has-parallel  -> no
+  --has-parallel4 -> yes
+  --has-parallel  -> yes

-  --prefix        -> /apps/netcdf/4.7.4
-  --includedir    -> /apps/netcdf/4.7.4/include
-  --libdir        -> /apps/netcdf/4.7.4/lib
+  --prefix        -> /apps/netcdf/4.7.4p
+  --includedir    -> /apps/netcdf/4.7.4p/include
+  --libdir        -> /apps/netcdf/4.7.4p/lib
   --version       -> netCDF 4.7.4
/apps/netcdf/4.7.4/bin/nc-config 
Usage: nc-config [OPTION]

Available values for OPTION include:

  --help          display this help message and exit
  --all           display all options
  --cc            C compiler
  --cflags        pre-processor and compiler flags
  --has-c++       whether C++ API is installed
  --has-c++4      whether netCDF-4 C++ API is installed
  --has-fortran   whether Fortran API is installed
  --has-dap2      whether OPeNDAP (DAP2) is enabled in this build
  --has-dap4      whether DAP4 is enabled in this build
  --has-dap       same as --has-dap2 (Deprecated)
  --has-nc2       whether NetCDF-2 API is enabled
  --has-nc4       whether NetCDF-4/HDF-5 is enabled in this build
  --has-hdf5      whether HDF5 is used in build (always the same as --has-nc4)
  --has-hdf4      whether HDF4 was used in build
  --has-logging   whether logging is enabled with --enable-logging.
  --has-pnetcdf   whether PnetCDF was used in build
  --has-szlib     whether szlib is included in build
  --has-cdf5      whether cdf5 support is included in build
  --has-parallel4 whether has parallel IO support via HDF5
  --has-parallel  whether has parallel IO support via HDF5 or PnetCDF
  --libs          library linking information for netcdf
  --static        library linking information for statically-compiled netcdf
  --prefix        Install prefix
  --includedir    Include directory
  --libdir        Library directory
  --version       Library version
harshula commented 1 year ago

NOTE, netcdf library directory is missing from the rpath of the Spack build:

$ readelf -d access-om2.git/bin/cice_auscom_1440x1080_480p_edcfa6f_libaccessom2_d750b4b.exe | grep -i path

0x000000000000000f (RPATH) Library rpath: [$ROOT/access-om2.git/src/cice5/ParallelIO/build/lib/:/apps/openmpi/4.0.2/lib/Intel:/apps/openmpi/4.0.2/lib:/apps/openmpi/4.0.2/lib/profilers:/apps/netcdf/4.7.4p/lib/ompi3/Intel:/apps/netcdf/4.7.4p/lib/ompi3:/apps/netcdf/4.7.4p/lib:/apps/intel-ct/2019.5.281/compiler/lib]

$ readelf -d spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/cice5-master-5ydjnpc3y6vv3eqyba7xbht2m455qh3q/bin/cice_auscom_1440x1080_48x40_480p.exe | grep -i path

0x000000000000000f (RPATH) Library rpath: [/apps/intel-ct/2019.5.281/compiler/lib:/apps/intel/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin:/half-root/usr/lib/gcc/x86_64-redhat-linux/8:/half-root/usr/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/hwloc-2.8.0-r5ru36jmxtsrmwxak4imiecmgo7neplo/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libevent-2.1.12-2tgpkknzc4gzg76lg6cv5qg5rtdaj3kq/lib:/usr/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/openmpi-4.0.2-fxqhczcsknc3t4gklr3e4nmtg7pmpxlm/lib]

harshula commented 1 year ago

MOM5 Spack build:

$ readelf -d spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/mom5-master-c6d6kywqic62upj66wz6ocw5c2waxial/bin/fms_ACCESS-OM.x | grep -i path```

0x000000000000000f (RPATH) Library rpath: [/apps/intel-ct/2019.5.281/compiler/lib:/apps/intel/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin:/half-root/usr/lib/gcc/x86_64-redhat-linux/8:/half-root/usr/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/hwloc-2.8.0-r5ru36jmxtsrmwxak4imiecmgo7neplo/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libevent-2.1.12-2tgpkknzc4gzg76lg6cv5qg5rtdaj3kq/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/openmpi-4.0.2-fxqhczcsknc3t4gklr3e4nmtg7pmpxlm/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/mom5-master-c6d6kywqic62upj66wz6ocw5c2waxial/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/mom5-master-c6d6kywqic62upj66wz6ocw5c2waxial/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libaccessom2-development-kdvfx6n5vd4nvjgeuoufuwym5yjkoawm/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/oasis3-mct-master-qf7xfj2fn2mcqoinut5l3za3rtv6mx4r/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-fortran-4.5.2-vxvt4qxhct6hpeiumgw2ygpq2tiot4br/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-c-4.7.4-k35uzgfuvzbydypggn246lifjprqw7sb/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/hdf5-1.12.2-3d53czshr26zauwqowejryjfn4nhy4zn/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/pmix-4.1.2-vmxxqlrmd42irdhiknkr6cdcbn5gisgi/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/numactl-2.0.14-araczq4x24mm7pyh7gzc2hsgv44dsjpo/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libxml2-2.10.1-afvgcekn2ronkoal6tx6sawpnklnwdhb/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/xz-5.2.7-fup6kkdhcfg4hzzj4e2u7cj5glb5zrzt/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libpciaccess-0.16-k3vn63w4bevjf22e2pv6vlthobfh2rji/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libiconv-1.16-mcmhdrfqpdtwihshzbkcnixotv3eipng/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/json-fortran-8.3.0-3ppdux7jxbp7xynhsrvokazzmkfkmdue/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/openssl-1.1.1s-dlyq5hli6cifgu23swg3rj7vjgfcoosk/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/zlib-1.2.13-voddumzgksrhnjdog7vd7i7jca5wh37k/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/ncurses-6.3-2pt667h723axnbvsqgcdiiiedovoyrno/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/datetime-fortran-1.7.0-ym4543pcm7j3zbmdqwpg6hd7c63efzmv/lib:/usr/lib64]

harshula commented 1 year ago

WORKAROUND

$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-fortran-4.5.2-vxvt4qxhct6hpeiumgw2ygpq2tiot4br/lib/:opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-c-4.7.4-k35uzgfuvzbydypggn246lifjprqw7sb/lib ldd opt/spack/linux-rocky8-x86_64/intel-2019.5.281/cice5-master-5ydjnpc3y6vv3eqyba7xbht2m455qh3q/bin/cice_auscom_1440x1080_48x40_480p.exe | grep netcdf
    libnetcdf.so.18 => opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-c-4.7.4-k35uzgfuvzbydypggn246lifjprqw7sb/lib/libnetcdf.so.18 (0x00007f44a4730000)
    libnetcdff.so.7 => opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-fortran-4.5.2-vxvt4qxhct6hpeiumgw2ygpq2tiot4br/lib/libnetcdff.so.7 (0x00007f44a4299000)
aidanheerdegen commented 1 year ago

If the MOM5 build sets the RPATH correctly why doesn't the CICE5 build also do so?

Is something missing in the CICE5 SPD?

harshula commented 1 year ago

MOM5 and CICE5 have different build systems. e.g. CICE5 does an explicit ld call.

harshula commented 1 year ago

Notes

This is the likely culprit. The explicit rpath arguments below correspond to the rpath directories found via readelf:

ld [...] -o cice_auscom_1440x1080_48x40_480p.exe [...] -lpiof -lpioc -lpsmile.MPI1 -lmct -lmpeu -lscrip -laccessom2 -lnetcdf -lnetcdff -ldatetime -lmpi -rpath /apps/intel-ct/2019.5.281/compiler/lib -rpath /apps/intel/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin -rpath /half-root/usr/lib/gcc/x86_64-redhat-linux/8 -rpath /half-root/usr/lib64 -rpath $ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/hwloc-2.8.0-r5ru36jmxtsrmwxak4imiecmgo7neplo/lib -rpath $ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libevent-2.1.12-2tgpkknzc4gzg76lg6cv5qg5rtdaj3kq/lib -rpath /usr/lib64 -rpath $ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/openmpi-4.0.2-fxqhczcsknc3t4gklr3e4nmtg7pmpxlm/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -Bdynamic -Bstatic -lifport -lifcoremt -limf -lsvml -Bdynamic -lm -Bstatic -lipgo -lirc -Bdynamic -lpthread -Bstatic -lsvml -Bdynamic -ldl -lc -lgcc -lgcc_s -Bstatic -lirc_s -Bdynamic -ldl -lc /half-root/usr/bin/../lib/gcc/x86_64-redhat-linux/8/crtend.o /half-root/usr/bin/../lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o

harshula commented 1 year ago

Notes

Using --disable-new-dtags and -rpath= with gcc 12.2.0 can resolve the problem using rpath:

$ ldd spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/cice5-master-ic3ha6ack66yotucvjjrnlmrodvlv5zf/bin/cice_auscom_1440x1080_48x40_480p.exe | grep netcdf
    libnetcdf.so.18 => $ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/netcdf-c-4.7.4-jzburrwbsqhi6cjth7nm4zdnunhcyhaq/lib/libnetcdf.so.18 (0x00007f8c0fb12000)
    libnetcdff.so.7 => $ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/netcdf-fortran-4.5.2-hlwgeozb6ierhgr3imrpzauehzm5sh63/lib/libnetcdff.so.7 (0x00007f8c0fa7a000)

0x000000000000000f (RPATH) Library rpath: [$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/parallelio-2.5.9-wepr5juoh7z4ru7lo3r4lkls73oh4w4s/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/oasis3-mct-master-2xghicufd5fh327fwmst7evyzlzwjby2/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/libaccessom2-development-uajwyu2477we2nwmzuldr22nsmlfbaof/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/netcdf-c-4.7.4-jzburrwbsqhi6cjth7nm4zdnunhcyhaq/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/netcdf-fortran-4.5.2-hlwgeozb6ierhgr3imrpzauehzm5sh63/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/datetime-fortran-1.7.0-w2m7dj2vuz7ql2hcfh22hjxk2mg65hit/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/hwloc-2.8.0-yiuypylbriyxjyqpwrguocbb7n4a3qf3/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/libevent-2.1.12-rluncdqrloqt4ki7pkbgampgybdbkqpv/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/openmpi-4.0.2-f3ryz6laankci7rzmyjtmquqdkisymoi/lib]

Using just -rpath= with gcc 12.2.0 can also resolve the problem, but defaults to runpath: 0x000000000000001d (RUNPATH) Library runpath: [$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/parallelio-2.5.9-wepr5juoh7z4ru7lo3r4lkls73oh4w4s/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/oasis3-mct-master-2xghicufd5fh327fwmst7evyzlzwjby2/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/libaccessom2-development-uajwyu2477we2nwmzuldr22nsmlfbaof/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/netcdf-c-4.7.4-jzburrwbsqhi6cjth7nm4zdnunhcyhaq/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/netcdf-fortran-4.5.2-hlwgeozb6ierhgr3imrpzauehzm5sh63/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/datetime-fortran-1.7.0-w2m7dj2vuz7ql2hcfh22hjxk2mg65hit/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/hwloc-2.8.0-yiuypylbriyxjyqpwrguocbb7n4a3qf3/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/libevent-2.1.12-rluncdqrloqt4ki7pkbgampgybdbkqpv/lib:$ROOT/spack-upstream.git/opt/spack/linux-debian12-x86_64/gcc-12.2.0/openmpi-4.0.2-f3ryz6laankci7rzmyjtmquqdkisymoi/lib]

harshula commented 1 year ago

Using just -rpath= with the intel compiler 2019.5.281, also resolves the problem:

$ ldd $ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/cice5-master-dfkucjdck2cpr3wdonyfr6r2qruqg32z/bin/cice_auscom_1440x1080_48x40_480p.exe | grep netcdf
    libnetcdf.so.18 => $ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-c-4.7.4-k35uzgfuvzbydypggn246lifjprqw7sb/lib/libnetcdf.so.18 (0x00007f980d787000)
    libnetcdff.so.7 => $ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-fortran-4.5.2-vxvt4qxhct6hpeiumgw2ygpq2tiot4br/lib/libnetcdff.so.7 (0x00007f980d2f0000)

0x000000000000000f (RPATH) Library rpath: [$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/parallelio-2.5.9-oqckdhkv4aji7rehadhnsy3o3lmtjndt/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/oasis3-mct-master-qf7xfj2fn2mcqoinut5l3za3rtv6mx4r/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libaccessom2-development-kdvfx6n5vd4nvjgeuoufuwym5yjkoawm/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-c-4.7.4-k35uzgfuvzbydypggn246lifjprqw7sb/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/netcdf-fortran-4.5.2-vxvt4qxhct6hpeiumgw2ygpq2tiot4br/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/datetime-fortran-1.7.0-ym4543pcm7j3zbmdqwpg6hd7c63efzmv/lib:/apps/intel-ct/2019.5.281/compiler/lib:/apps/intel/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin:/half-root/usr/lib/gcc/x86_64-redhat-linux/8:/half-root/usr/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/hwloc-2.8.0-r5ru36jmxtsrmwxak4imiecmgo7neplo/lib:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/libevent-2.1.12-2tgpkknzc4gzg76lg6cv5qg5rtdaj3kq/lib:/usr/lib64:$ROOT/spack-upstream.git/opt/spack/linux-rocky8-x86_64/intel-2019.5.281/openmpi-4.0.2-fxqhczcsknc3t4gklr3e4nmtg7pmpxlm/lib]

harshula commented 1 year ago

Notes

src/cice5/bld/Makefile.std:

        $(LD) -o $(EXEC) $(LDFLAGS) $(OBJS) $(ULIBS) $(SLIBS)

The following ld arguments during ACCESS-OM2 COSIMA build:

-L/apps/netcdf/4.7.4p/lib/ompi3/Intel -L/apps/netcdf/4.7.4p/lib/ompi3 -L/apps/netcdf/4.7.4p/lib

are not from any of the following variables: $(EXEC) $(LDFLAGS) $(OBJS) $(ULIBS) $(SLIBS)

Need to identify the environment variables used by ld to find out where they are coming from.

where LD is defined in src/cice5/bld/Macros.nci:

FC         := mpifort
LD         := $(FC)
harshula commented 1 year ago

Notes

$ module show netcdf/4.7.4p
/apps/Modules/modulefiles/netcdf/4.7.4p:

prepend-path    PATH /apps/netcdf/4.7.4p/bin
prepend-path    C_INCLUDE_PATH /apps/netcdf/4.7.4p/include
prepend-path    CPLUS_INCLUDE_PATH /apps/netcdf/4.7.4p/include
prepend-path    CPATH /apps/netcdf/4.7.4p/include
prepend-path    FPATH /apps/netcdf/4.7.4p/include
prepend-path    LIBRARY_PATH /apps/netcdf/4.7.4p/lib
prepend-path    LD_LIBRARY_PATH /apps/netcdf/4.7.4p/lib
prepend-path    LD_RUN_PATH /apps/netcdf/4.7.4p/lib
prepend-path    PKG_CONFIG_PATH /apps/netcdf/4.7.4p/lib/pkgconfig
prepend-path    MANPATH /apps/netcdf/4.7.4p/share/man
prepend-path    INFOPATH /apps/netcdf/4.7.4p/share/info
setenv          NETCDF /apps/netcdf/4.7.4p
conflict        netcdf
setenv          NETCDF_BASE /apps/netcdf/4.7.4p
setenv          NETCDF_ROOT /apps/netcdf/4.7.4p
setenv          NETCDF_VERSION 4.7.4p
module-whatis   {netcdf, version 4.7.4p}
harshula commented 1 year ago 
$ which ld
/opt/nci/bin/ld
$ ls -l /opt/nci/bin/ld                                   
lrwxrwxrwx. 1 root root 14 Mar 17  2020 /opt/nci/bin/ld -> ../wrappers/ld   
#!/bin/bash
# This script adjusts the behaviour of ld.

# Specifically, we want the default behaviour when using -rpath or $LD_RUN_PATH
# to be to set DT_RUNPATH as well as DT_RPATH in the resulting binary object.
# This will enable the dynamic linker to override the -rpath setting at runtime
# with $LD_LIBRARY_PATH.

...

# The use of the -rpath flag (at all) completely inhibits $LD_RUN_PATH.
# This means that anything we have put in there, including important
# stuff like compiler runtimes, won't be rpath'd.
# So the solution is to take the contents of $LD_RUN_PATH and stuff it
# into a -rpath flag at the end of the command line.
function finalise () {
    if [ -z $LD_RUN_PATH ]; then
        launch
    else
        launch_append -rpath "$LD_RUN_PATH"
    fi
}
harshula commented 1 year ago 
/opt/nci/bin/ld    [...] -L/apps/netcdf/4.7.4p/lib [...] -o cice_auscom_360x300_24p.ld-test.nci-ld.exe [...] -L/apps/netcdf/4.7.4p/lib/ompi3/Intel -L/apps/netcdf/4.7.4p/lib/ompi3 -L/apps/netcdf/4.7.4p/lib -L/apps/netcdf/4.7.4p/lib/ompi3/Intel/ -L/apps/netcdf/4.7.4p/lib/ompi3/GNU/ -L/apps/netcdf/4.7.4p/lib/ompi3/ -L/apps/netcdf/4.7.4p/lib/ [...] -lnetcdf -lnetcdff [...]

/usr/bin/ld.bfd    [...] -L/apps/netcdf/4.7.4p/lib [...] -o cice_auscom_360x300_24p.ld-test.ld.bfd.exe [...] -L/apps/netcdf/4.7.4p/lib/ompi3/Intel -L/apps/netcdf/4.7.4p/lib/ompi3 -L/apps/netcdf/4.7.4p/lib -L/apps/netcdf/4.7.4p/lib/ompi3/Intel/ -L/apps/netcdf/4.7.4p/lib/ompi3/GNU/ -L/apps/netcdf/4.7.4p/lib/ompi3/ -L/apps/netcdf/4.7.4p/lib/ [...] -lnetcdf -lnetcdff [...]

The above ld commands called without module load netcdf/4.7.4p:

$ readelf -d cice_auscom_360x300_24p.ld-test.nci-ld.exe | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [$GDATA/access-om2-cice5-test.git/src/cice5/ParallelIO/build/lib/]

$ readelf -d cice_auscom_360x300_24p.ld-test.ld.bfd.exe | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [$GDATA/access-om2-cice5-test.git/src/cice5/ParallelIO/build/lib/]

The above ld commands called after module load netcdf/4.7.4p:

$ readelf -d cice_auscom_360x300_24p.ld-test.nci-ld.exe | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [$GDATA/access-om2-cice5-test.git/src/cice5/ParallelIO/build/lib/:/apps/netcdf/4.7.4p/lib]

$ readelf -d cice_auscom_360x300_24p.ld-test.ld.bfd.exe | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [$GDATA/access-om2-cice5-test.git/src/cice5/ParallelIO/build/lib/]

It is safe to conclude that gadi's ld takes $LD_RUN_PATH and adds it to rpath resulting in the netcdf libraries being found at runtime, even when the netcdf module is NOT loaded.

harshula commented 1 year ago

NOTE: Spack also has a ld wrapper

$ ls -l lib/spack/env/ld
lrwxrwxrwx 1 harshula harshula 2 Aug 28  2022 lib/spack/env/ld -> cc

$SPACKDIR/lib/spack/env/cc:

# RPATHs arguments
case "$mode" in
    ccld)
        if [ -n "$dtags_to_add" ] ; then
            append args_list "$linker_arg$dtags_to_add"
        fi
        extend args_list rpath_dirs_list "$rpath"
        extend args_list system_rpath_dirs_list "$rpath"
        ;;
    ld)
        if [ -n "$dtags_to_add" ] ; then
            append args_list "$dtags_to_add"
        fi
        extend args_list rpath_dirs_list "-rpath${lsep}"
        extend args_list system_rpath_dirs_list "-rpath${lsep}"
        ;;
esac
harshula commented 1 year ago

Notes

@penguian made a good suggestion to explicitly run ld while altering the library search directories.

Original: ld [...] -L/apps/netcdf/4.7.4p/lib [...] -o cice_auscom_360x300_24p.ld-test.exe [...] -L/apps/netcdf/4.7.4p/lib/ompi3/Intel -L/apps/netcdf/4.7.4p/lib/ompi3 -L/apps/netcdf/4.7.4p/lib [...] -L/apps/netcdf/4.7.4p/lib/ompi3/Intel/ -L/apps/netcdf/4.7.4p/lib/ompi3/GNU/ -L/apps/netcdf/4.7.4p/lib/ompi3/ -L/apps/netcdf/4.7.4p/lib/ [...] -lnetcdf -lnetcdff => Creates an executable

Remove: -L/apps/netcdf/4.7.4p/lib/ompi3/Intel/ -L/apps/netcdf/4.7.4p/lib/ompi3/GNU/ -L/apps/netcdf/4.7.4p/lib/ompi3/ ld [...] -L/apps/netcdf/4.7.4p/lib [...] -o cice_auscom_360x300_24p.ld-test.exe [...] -L/apps/netcdf/4.7.4p/lib [...] -lnetcdf -lnetcdff .. /bin/ld: cannot find -lnetcdf /bin/ld: cannot find -lnetcdff => Does not create an executable

Remove: -L/apps/netcdf/4.7.4p/lib/ ld [...] -o cice_auscom_360x300_24p.ld-test.exe [...] -L/apps/netcdf/4.7.4p/lib/ompi3/Intel -L/apps/netcdf/4.7.4p/lib/ompi3 [...] -L/apps/netcdf/4.7.4p/lib/ompi3/Intel/ -L/apps/netcdf/4.7.4p/lib/ompi3/GNU/ -L/apps/netcdf/4.7.4p/lib/ompi3/ [...] -lnetcdf -lnetcdff => Creates an executable

harshula commented 1 year ago

Notes

Spack build with the aforementioned patch, results in: ld [...] -L$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/netcdf-c-4.7.4-2pnrht2gvdji7afpxemqaag7alpujbm6/lib -L$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/netcdf-fortran-4.5.2-iodr42hphp2ihgpkegoo4peyrpk3g7x6/lib [...] -o cice_auscom_360x300_24x1_24p.exe [...] -lnetcdf -rpath=$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/netcdf-c-4.7.4-2pnrht2gvdji7afpxemqaag7alpujbm6/lib -lnetcdff -rpath=$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/netcdf-fortran-4.5.2-iodr42hphp2ihgpkegoo4peyrpk3g7x6/lib [...]

$ readelf -d opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/cice5-master-sfqeshtw7dca6kgbncmk24frgqgdmq3p/bin/cice_auscom_360x300_24x1_24p.exe 0x000000000000000f (RPATH) Library rpath: [$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/parallelio-2.5.10-6cqex6f5q47ajhmt5p2pvl7ibawig6ic/lib:$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/oasis3-mct-master-mjvolqwye5yv57aky5faeasq47byw67c/lib:$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-master-ytsfc7nfiuvs4jzubyvwgcutqkuqmnxo/lib:$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/netcdf-c-4.7.4-2pnrht2gvdji7afpxemqaag7alpujbm6/lib:$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/netcdf-fortran-4.5.2-iodr42hphp2ihgpkegoo4peyrpk3g7x6/lib:$GDATA/spack-system-openmpi-from-main.git/opt/spack/linux-rocky8-x86_64/intel-19.0.5.281/datetime-fortran-1.7.0-aretozixwsdz42owabgzeiuxooyxrspg/lib:/apps/openmpi/4.0.2/lib/Intel:/apps/openmpi/4.0.2/lib:/apps/openmpi/4.0.2/lib/profilers:/apps/intel-ct/2019.5.281/compiler/lib]