craype when building serial application

[jedwards4b]

Currently Loaded Modules: 1) ncarenv/22.12 (S) 4) mpi-serial/2.3.0 7) cray-libsci/22.11.1.2 10) parallelio/2.5.10-debug 2) craype/2.7.19 5) cmake/3.25.0 8) hdf5/1.12.2 11) esmf/8.4.1b02-debug 3) gcc/12.1.0 6) ncarcompilers/0.7.2 9) netcdf/4.9.0

Link errors:

    /usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_strerror@FABRIC_1.0'
    /usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_fabric@FABRIC_1.1'
    /usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_getinfo@FABRIC_1.3'
    /usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_version@FABRIC_1.0'        
    /usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_dupinfo@FABRIC_1.3'
    /usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_freeinfo@FABRIC_1.3'

[vanderwb]

@jedwards - can you dump your env either here or in a file? I'm not able to replicate this with my toy code:

[09:57] hw_omp$ ml

Currently Loaded Modules:
  1) ncarenv/22.12 (S)   3) gcc/12.1.0            5) hdf5/1.12.2    7) cray-libsci/22.11.1.2   9) parallelio/2.5.10-debug
  2) craype/2.7.19       4) ncarcompilers/0.7.2   6) netcdf/4.9.0   8) mpi-serial/2.3.0       10) esmf/8.4.1b02-debug

  Where:
   S:  Module is Sticky, requires --force to unload or purge

[09:57] hw_omp$ gcc -o test openmp.c -fopenmp
openmp.c: In function 'main':
openmp.c:29:31: warning: implicit declaration of function 'sched_getcpu'; did you mean 'sched_getparam'? [-Wimplicit-function-declaration]
   29 |                 core        = sched_getcpu() % TASKS_PER_NODE;
      |                               ^~~~~~~~~~~~
      |                               sched_getparam
[09:57] hw_omp$ ls
openmp.c  test

[jedwards4b]

/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230203_103848_p3ppcf/software_environment.txt

[vanderwb]

@jedwards4b - can you still reproduce this problem after the module updates?

[jedwards4b]

Now I'm having another issue that I thought I had resolved.

spack concretize -f
==> Starting concretization pool with 10 processes
==> Error: errors occurred during concretization of the environment:
{'-fPIC', '-std=c90'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC', '-std=c90'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()

In the past I had -fPIC in the environment for my mpi-serial build, but I have removed it and replaced it with args = ['CFLAGS={0}'.format(self.compiler.cc_pic_flag)] in the mpi-serial package.py. I uninstalled the original mpi-serial and reinstalled with the change. I ran spack clean -a but I still get that error on concretize.

Furthermore a module avail shows:

------------------------------------------- MPI-dependent Software - [cce/15.0.1 + mpi-serial/2.3.0] -------------------------------------------
   esmf/8.5.0b16-debug    esmf/8.5.0b16 (D)    parallelio/2.5.10-debug    parallelio/2.5.10 (D)

However the parallelio modules are not there:

 ls -ltr /glade/u/apps/cseg/modules/22.12/mpi-serial/2.3.0/cce/15.0.1
total 1
drwxrwsr-x+ 2 jedwards cseg 4096 Feb 16 14:57 esmf

[jedwards4b]

I see in module show mpi-serial/2.3.0 whatis("Configure options : CFLAGS=-fPIC") I don't see this line in for example the intel/2021.7.1 mpi-serial module.

[jedwards4b]

Manually commenting out that line in the module file seems to have solved that problem.

[vanderwb]

I can work with you on Slack, or another GitHub issue, on the other things, but just to be clear, you cannot reproduce the original reported issue any longer?

[jedwards4b]

I was wrong - the -FPIC issue is still happening.

[jedwards4b]

/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no

gfortran -o /glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/cesm.exe ensemble_driver.o esm.o esmApp.o esmFlds.o esmFldsExchange_cesm_mod.o esmFldsExchange_hafs_mod.o esmFldsExchange_nems_mod.o esm_time_mod.o med.o med_constants_mod.o med_diag_mod.o med_fraction_mod.o med_internalstate_mod.o med_io_mod.o med_kind_mod.o med_map_mod.o med_merge_mod.o med_methods_mod.o med_phases_aofluxes_mod.o med_phases_history_mod.o med_phases_ocnalb_mod.o med_phases_post_atm_mod.o med_phases_post_glc_mod.o med_phases_post_ice_mod.o med_phases_post_lnd_mod.o med_phases_post_ocn_mod.o med_phases_post_rof_mod.o med_phases_post_wav_mod.o med_phases_prep_atm_mod.o med_phases_prep_glc_mod.o med_phases_prep_ice_mod.o med_phases_prep_lnd_mod.o med_phases_prep_ocn_mod.o med_phases_prep_rof_mod.o med_phases_prep_wav_mod.o med_phases_profile_mod.o med_phases_restart_mod.o med_time_mod.o med_utils_mod.o shr_flux_mod.o t_driver_timers_mod.o util.o  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -latm  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lice  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -llnd  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -locn  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lrof  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lglc  -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lwav -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/CDEPS/dshr -ldshr -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/CDEPS/streams -lstreams -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1g1w1/lib -lcsm_share -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/lib -lpiof -lpioc -lgptl -lmct -lmpeu   -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/CDEPS/fox/lib -lFoX_dom -lFoX_sax -lFoX_utils -lFoX_fsys -lFoX_wxml -lFoX_common -lFoX_fsys -L/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/esmf-8.4.1b02-5doca6ye42dnlonjpcajvwatq6ulmcgg/lib -Wl,-rpath,/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/esmf-8.4.1b02-5doca6ye42dnlonjpcajvwatq6ulmcgg/lib -lesmf  -lrt -lstdc++ -ldl -lsci_gnu -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lpioc -L/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/esmf-8.4.1b02-5doca6ye42dnlonjpcajvwatq6ulmcgg/lib -L/glade/work/csgteam/spack-deployments/gust/22.12/envs/build/opt/__spack_path_placeholder__/__spack_path_placeholder__/__spack_pa/netcdf/4.9.0/packages/netcdf-c/4.9.0/gcc/12.1.0/lib -L/glade/u/apps/gust/22.12/spack/opt/spack/netcdf/4.9.0/packages/netcdf-fortran/4.6.0/gcc/12.1.0/lib -L/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/parallelio-2.5.10-gjpniwdkbvi5qpozxs2djqxia43wcb7b/lib -L/opt/cray/pe/gcc/12.1.0/snos/lib/gcc/x86_64-suse-linux/12.1.0/../../../../lib64/  /glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/lib/libmpi-serial.a
err=f951: Warning: Nonexistent include directory '/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/finclude' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory '/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/iac/obj' [-Wmissing-include-dirs]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:66:

  757 |   subroutine InitAdvertize(driver, importState, exportState, clock, rc)
      |                                                                  1
Warning: Unused dummy argument 'clock' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:33:

  757 |   subroutine InitAdvertize(driver, importState, exportState, clock, rc)
      |                                 1
Warning: Unused dummy argument 'driver' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:59:

  757 |   subroutine InitAdvertize(driver, importState, exportState, clock, rc)
      |                                                           1
Warning: Unused dummy argument 'exportstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:46:

  757 |   subroutine InitAdvertize(driver, importState, exportState, clock, rc)
      |                                              1
Warning: Unused dummy argument 'importstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:712:74:

  712 |   subroutine ReadAttributes(gcomp, config, label, relaxedflag, formatprint, rc)
      |                                                                          1
Warning: Unused dummy argument 'formatprint' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:317:77:

  317 |   recursive subroutine ModifyCplLists(driver, importState, exportState, clock, rc)
      |                                                                             1
Warning: Unused dummy argument 'clock' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:317:70:

  317 |   recursive subroutine ModifyCplLists(driver, importState, exportState, clock, rc)
      |                                                                      1
Warning: Unused dummy argument 'exportstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:317:57:

  317 |   recursive subroutine ModifyCplLists(driver, importState, exportState, clock, rc)
      |                                                         1
Warning: Unused dummy argument 'importstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:279:42:

  279 |   subroutine pretty_print_nuopc_freeformat(ffstuff, label, rc)
      |                                          ^
Warning: 'pretty_print_nuopc_freeformat' defined but not used [-Wunused-function]
/usr/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/lib64/libmpifort_gnu_91.so.12, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_strerror@FABRIC_1.0'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_fabric@FABRIC_1.1'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_getinfo@FABRIC_1.3'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_version@FABRIC_1.0'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_dupinfo@FABRIC_1.3'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_freeinfo@FABRIC_1.3'