Closed jedwards4b closed 1 year ago
@jedwards - can you dump your env either here or in a file? I'm not able to replicate this with my toy code:
[09:57] hw_omp$ ml
Currently Loaded Modules:
1) ncarenv/22.12 (S) 3) gcc/12.1.0 5) hdf5/1.12.2 7) cray-libsci/22.11.1.2 9) parallelio/2.5.10-debug
2) craype/2.7.19 4) ncarcompilers/0.7.2 6) netcdf/4.9.0 8) mpi-serial/2.3.0 10) esmf/8.4.1b02-debug
Where:
S: Module is Sticky, requires --force to unload or purge
[09:57] hw_omp$ gcc -o test openmp.c -fopenmp
openmp.c: In function 'main':
openmp.c:29:31: warning: implicit declaration of function 'sched_getcpu'; did you mean 'sched_getparam'? [-Wimplicit-function-declaration]
29 | core = sched_getcpu() % TASKS_PER_NODE;
| ^~~~~~~~~~~~
| sched_getparam
[09:57] hw_omp$ ls
openmp.c test
/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230203_103848_p3ppcf/software_environment.txt
@jedwards4b - can you still reproduce this problem after the module updates?
Now I'm having another issue that I thought I had resolved.
spack concretize -f
==> Starting concretization pool with 10 processes
==> Error: errors occurred during concretization of the environment:
{'-fPIC', '-std=c90'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC', '-std=c90'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
{'-fPIC'} does not equal set()
In the past I had -fPIC in the environment for my mpi-serial build, but I have removed it and replaced it with args = ['CFLAGS={0}'.format(self.compiler.cc_pic_flag)]
in the mpi-serial package.py. I uninstalled the original mpi-serial and reinstalled with the change. I ran spack clean -a
but I still get that error on concretize.
Furthermore a module avail shows:
------------------------------------------- MPI-dependent Software - [cce/15.0.1 + mpi-serial/2.3.0] -------------------------------------------
esmf/8.5.0b16-debug esmf/8.5.0b16 (D) parallelio/2.5.10-debug parallelio/2.5.10 (D)
However the parallelio modules are not there:
ls -ltr /glade/u/apps/cseg/modules/22.12/mpi-serial/2.3.0/cce/15.0.1
total 1
drwxrwsr-x+ 2 jedwards cseg 4096 Feb 16 14:57 esmf
I see in module show mpi-serial/2.3.0
whatis("Configure options : CFLAGS=-fPIC")
I don't see this line in for example the intel/2021.7.1 mpi-serial module.
Manually commenting out that line in the module file seems to have solved that problem.
I can work with you on Slack, or another GitHub issue, on the other things, but just to be clear, you cannot reproduce the original reported issue any longer?
I was wrong - the -FPIC issue is still happening.
/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no
gfortran -o /glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/cesm.exe ensemble_driver.o esm.o esmApp.o esmFlds.o esmFldsExchange_cesm_mod.o esmFldsExchange_hafs_mod.o esmFldsExchange_nems_mod.o esm_time_mod.o med.o med_constants_mod.o med_diag_mod.o med_fraction_mod.o med_internalstate_mod.o med_io_mod.o med_kind_mod.o med_map_mod.o med_merge_mod.o med_methods_mod.o med_phases_aofluxes_mod.o med_phases_history_mod.o med_phases_ocnalb_mod.o med_phases_post_atm_mod.o med_phases_post_glc_mod.o med_phases_post_ice_mod.o med_phases_post_lnd_mod.o med_phases_post_ocn_mod.o med_phases_post_rof_mod.o med_phases_post_wav_mod.o med_phases_prep_atm_mod.o med_phases_prep_glc_mod.o med_phases_prep_ice_mod.o med_phases_prep_lnd_mod.o med_phases_prep_ocn_mod.o med_phases_prep_rof_mod.o med_phases_prep_wav_mod.o med_phases_profile_mod.o med_phases_restart_mod.o med_time_mod.o med_utils_mod.o shr_flux_mod.o t_driver_timers_mod.o util.o -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -latm -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lice -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -llnd -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -locn -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lrof -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lglc -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/lib/ -lwav -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/CDEPS/dshr -ldshr -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/CDEPS/streams -lstreams -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1g1w1/lib -lcsm_share -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/lib -lpiof -lpioc -lgptl -lmct -lmpeu -L/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/CDEPS/fox/lib -lFoX_dom -lFoX_sax -lFoX_utils -lFoX_fsys -lFoX_wxml -lFoX_common -lFoX_fsys -L/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/esmf-8.4.1b02-5doca6ye42dnlonjpcajvwatq6ulmcgg/lib -Wl,-rpath,/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/esmf-8.4.1b02-5doca6ye42dnlonjpcajvwatq6ulmcgg/lib -lesmf -lrt -lstdc++ -ldl -lsci_gnu -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lpioc -L/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/esmf-8.4.1b02-5doca6ye42dnlonjpcajvwatq6ulmcgg/lib -L/glade/work/csgteam/spack-deployments/gust/22.12/envs/build/opt/__spack_path_placeholder__/__spack_path_placeholder__/__spack_pa/netcdf/4.9.0/packages/netcdf-c/4.9.0/gcc/12.1.0/lib -L/glade/u/apps/gust/22.12/spack/opt/spack/netcdf/4.9.0/packages/netcdf-fortran/4.6.0/gcc/12.1.0/lib -L/glade/u/apps/cseg/spack/opt/spack/linux-sles15-zen3/gcc-12.1.0/parallelio-2.5.10-gjpniwdkbvi5qpozxs2djqxia43wcb7b/lib -L/opt/cray/pe/gcc/12.1.0/snos/lib/gcc/x86_64-suse-linux/12.1.0/../../../../lib64/ /glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/lib/libmpi-serial.a
err=f951: Warning: Nonexistent include directory '/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/gnu/mpi-serial/debug/nothreads/nuopc/finclude' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory '/glade/gust/scratch/jedwards/SMS_D_Mmpi-serial.f19_g17.X.gust_gnu.20230303_082100_b796no/bld/iac/obj' [-Wmissing-include-dirs]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:66:
757 | subroutine InitAdvertize(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'clock' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:33:
757 | subroutine InitAdvertize(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'driver' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:59:
757 | subroutine InitAdvertize(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'exportstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:757:46:
757 | subroutine InitAdvertize(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'importstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:712:74:
712 | subroutine ReadAttributes(gcomp, config, label, relaxedflag, formatprint, rc)
| 1
Warning: Unused dummy argument 'formatprint' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:317:77:
317 | recursive subroutine ModifyCplLists(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'clock' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:317:70:
317 | recursive subroutine ModifyCplLists(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'exportstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:317:57:
317 | recursive subroutine ModifyCplLists(driver, importState, exportState, clock, rc)
| 1
Warning: Unused dummy argument 'importstate' at (1) [-Wunused-dummy-argument]
/glade/work/jedwards/sandboxes/cesm2_x_alpha/components/cmeps/cime_config/../cesm/driver/esm.F90:279:42:
279 | subroutine pretty_print_nuopc_freeformat(ffstuff, label, rc)
| ^
Warning: 'pretty_print_nuopc_freeformat' defined but not used [-Wunused-function]
/usr/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/lib64/libmpifort_gnu_91.so.12, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_strerror@FABRIC_1.0'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_fabric@FABRIC_1.1'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_getinfo@FABRIC_1.3'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_version@FABRIC_1.0'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_dupinfo@FABRIC_1.3'
/usr/bin/ld: /opt/cray/pe/lib64/libmpi_gnu_91.so.12: undefined reference to `fi_freeinfo@FABRIC_1.3'
Currently Loaded Modules: 1) ncarenv/22.12 (S) 4) mpi-serial/2.3.0 7) cray-libsci/22.11.1.2 10) parallelio/2.5.10-debug 2) craype/2.7.19 5) cmake/3.25.0 8) hdf5/1.12.2 11) esmf/8.4.1b02-debug 3) gcc/12.1.0 6) ncarcompilers/0.7.2 9) netcdf/4.9.0
Link errors: