Closed sarats closed 6 months ago
I did check and the following file exists on the compute node.
$ ls -l /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne30_20220428.nc
-rw-rw-r-- 1 acmetest cli115 9914470292 Apr 27 17:41 /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne30_20220428.nc
I notice you're trying to read an ne30 file during an ne4 simulation. I think this is expected - I think we remap an ne30 file to ne4 to confirm that horizontal remapping is correct. But worth checking with @AaronDonahue, who implemented this part of the code. Aaron - can you take a quick look at this stack trace and see if you have other ideas about what could be wrong.
I think you can bypass this horiz remap step by using an ne4 SPA file instead and setting SPARemapFile to none in namelist_scream.xml.
It's also weird that we're having so much trouble on Summit, but have been doing perlmutter gpu runs for a long while. Is there a particular case you want @ndkeen to try to confirm the problem is isolated to Summit?
the ne4 spa file to use is /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne4_20220428.nc
All of us have been trying to just do the basic ne4 test run after checking out master.
But there is something specific in how Summit fails when we have 2 tasks on same node vs. 2 tasks on different nodes.
./create_newcase --compset F2010-SCREAMv1 --res ne4_ne4 --case scr-ne4-nt2 --compiler gnugpu --queue debug --walltime 00:30
./xmlchange DEBUG=TRUE
./xmlchange NTASKS=2
./case.setup
./case.build
./case.submit
based on the stack trace it looks like it errors when trying to open the file. @sarats , can you confirm the files SPA is looking for are actually there? You can check the path by looking in run/data/scream_input.yaml
and looking in the SPA section.
As I noted above, this file /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne30_20220428.nc
is there. Do you want to check something else?
yes, can you check if the mapping file is also present?
oh wait, yeah, this is looking for the data file. So it should work...
what are the permissions on that file?
Permissions seem ok.
64 spa:
65 SPA Remap File: /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/map_ne30_to_ne4_mono_20220502.nc
66 SPA Data File: /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne30_20220428.nc
[sarat@batch1 data ]$ ls /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne30_20220428.nc -l
-rw-rw-r-- 1 acmetest cli115 9914470292 Apr 27 17:41 /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/spa_file_unified_and_complete_ne30_20220428.nc
[sarat@batch1 data ]$ ls -l /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/map_ne30_to_ne4_mono_20220502.nc
-rw-rw-r-- 1 acmetest cli115 16979268 May 2 19:38 /gpfs/alpine/cli115/world-shared/e3sm/inputdata/atm/scream/init/map_ne30_to_ne4_mono_20220502.nc
When I change the data file to the one @PeterCaldwell pointed above, the program crashes at
(gdb) bt
#0 0x0000200006133618 in raise () from /lib64/power9/libc.so.6
#1 0x0000200006113a2c in abort () from /lib64/power9/libc.so.6
#2 0x0000200003fbba28 in __gnu_cxx::__verbose_terminate_handler () at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x0000200003fb7004 in __cxxabiv1::__terminate (handler=<optimized out>) at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
#4 0x0000200003fb70d0 in std::terminate () at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
#5 0x0000200003fb75a8 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x200004166bf0 <typeinfo for std::logic_error>, dest=0x200003fd7070 <std::logic_error::~logic_error()>) at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/eh_throw.cc:95
#6 0x000000001374e200 in scream::spa::SPAFunctions<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >::update_spa_data_from_file (spa_data_file_name=..., time_index=1, nswbands=14, nlwbands=16, spa_horiz_interp=..., spa_data=...) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/spa_functions_impl.hpp:518
#7 0x00000000137499f8 in scream::spa::SPAFunctions<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >::update_spa_timestate (spa_data_file_name=..., nswbands=14, nlwbands=16, ts=..., spa_horiz_interp=..., time_state=..., spa_beg=..., spa_end=...)
at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/spa_functions_impl.hpp:704
#8 0x000000001373d338 in scream::SPA::initialize_impl (this=0x40954ea0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/atmosphere_prescribed_aerosol.cpp:189
#9 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x40954ea0, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#10 0x00000000137d0888 in scream::AtmosphereProcessGroup::initialize_impl (this=0x37a2c970, run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process_group.cpp:186
#11 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x37a2c970, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#12 0x00000000137d0888 in scream::AtmosphereProcessGroup::initialize_impl (this=0x383d0e40, run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process_group.cpp:186
#13 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x383d0e40, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#14 0x00000000137d0888 in scream::AtmosphereProcessGroup::initialize_impl (this=0x3e600480, run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process_group.cpp:186
#15 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x3e600480, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#16 0x000000001265db50 in scream::control::AtmosphereDriver::initialize_atm_procs (this=0x40a198b0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/control/atmosphere_driver.cpp:872
#17 0x000000001024ac70 in <lambda()>::operator()(void) const (__closure=0x7fffe9acb6a0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/scream_cxx_f90_interface.cpp:190
#18 0x0000000010252c04 in _GLOBAL__N__e19a5351_28_scream_cxx_f90_interface_cpp_babe8b2b::fpe_guard_wrapper<scream_init_atm(int, int, int, int)::<lambda()> >(const <lambda()> &) (f=...) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/scream_cxx_f90_interface.cpp:51
#19 0x000000001024ad6c in scream_init_atm (run_start_ymd=10101, run_start_tod=0, case_start_ymd=10101, case_start_tod=0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/scream_cxx_f90_interface.cpp:164
#20 0x0000000010246684 in atm_comp_mct::atm_init_mct (eclock=..., cdata=..., x2a=..., a2x=..., nlfilename=..., _nlfilename=6) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/atm_comp_mct.F90:174
#21 0x000000001006823c in component_mod::component_init_cc (eclock=..., comp=..., comp_init=0x102444e0 <atm_comp_mct::atm_init_mct>, infodata=..., nlfilename=..., seq_flds_x2c_fluxes=..., seq_flds_c2x_fluxes=..., _nlfilename=6, _seq_flds_x2c_fluxes=0, _seq_flds_c2x_fluxes=0)
at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/component_mod.F90:248
#22 0x00000000100473f0 in cime_comp_mod::cime_init () at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/cime_comp_mod.F90:1438
#23 0x000000001005fa40 in cime_driver () at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/cime_driver.F90:122
#24 0x000000001005fbf0 in main (argc=1, argv=0x7fffe9ad4ba6) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/cime_driver.F90:23
#25 0x0000200006114078 in generic_start_main.isra () from /lib64/power9/libc.so.6
#26 0x0000200006114264 in __libc_start_main () from /lib64/power9/libc.so.6
#27 0x0000000000000000 in ?? ()
Second process
(gdb) bt
#0 0x0000200006133618 in raise () from /lib64/power9/libc.so.6
#1 0x0000200006113a2c in abort () from /lib64/power9/libc.so.6
#2 0x0000200003fbba28 in __gnu_cxx::__verbose_terminate_handler () at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x0000200003fb7004 in __cxxabiv1::__terminate (handler=<optimized out>) at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
#4 0x0000200003fb70d0 in std::terminate () at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
#5 0x0000200003fb75a8 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x200004166bf0 <typeinfo for std::logic_error>, dest=0x200003fd7070 <std::logic_error::~logic_error()>) at /gpfs/alpine/scratch/belhorn/stf007/builds/gcc-build-9.3.0-3/gcc-9.3.0/libstdc++-v3/libsupc++/eh_throw.cc:95
#6 0x000000001374e200 in scream::spa::SPAFunctions<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >::update_spa_data_from_file (spa_data_file_name=..., time_index=1, nswbands=14, nlwbands=16, spa_horiz_interp=..., spa_data=...) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/spa_functions_impl.hpp:518
#7 0x00000000137499f8 in scream::spa::SPAFunctions<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >::update_spa_timestate (spa_data_file_name=..., nswbands=14, nlwbands=16, ts=..., spa_horiz_interp=..., time_state=..., spa_beg=..., spa_end=...)
at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/spa_functions_impl.hpp:704
#8 0x000000001373d338 in scream::SPA::initialize_impl (this=0x59c56ea0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/atmosphere_prescribed_aerosol.cpp:189
#9 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x59c56ea0, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#10 0x00000000137d0888 in scream::AtmosphereProcessGroup::initialize_impl (this=0x53b521e0, run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process_group.cpp:186
#11 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x53b521e0, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#12 0x00000000137d0888 in scream::AtmosphereProcessGroup::initialize_impl (this=0x540299f0, run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process_group.cpp:186
#13 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x540299f0, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#14 0x00000000137d0888 in scream::AtmosphereProcessGroup::initialize_impl (this=0x55211b70, run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process_group.cpp:186
#15 0x00000000137b53f4 in scream::AtmosphereProcess::initialize (this=0x55211b70, t0=..., run_type=scream::RunType::Initial) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/share/atm_process/atmosphere_process.cpp:43
#16 0x000000001265db50 in scream::control::AtmosphereDriver::initialize_atm_procs (this=0x59679e30) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/control/atmosphere_driver.cpp:872
#17 0x000000001024ac70 in <lambda()>::operator()(void) const (__closure=0x7ffffd93e9f0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/scream_cxx_f90_interface.cpp:190
#18 0x0000000010252c04 in _GLOBAL__N__e19a5351_28_scream_cxx_f90_interface_cpp_babe8b2b::fpe_guard_wrapper<scream_init_atm(int, int, int, int)::<lambda()> >(const <lambda()> &) (f=...) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/scream_cxx_f90_interface.cpp:51
#19 0x000000001024ad6c in scream_init_atm (run_start_ymd=10101, run_start_tod=0, case_start_ymd=10101, case_start_tod=0) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/scream_cxx_f90_interface.cpp:164
#20 0x0000000010246684 in atm_comp_mct::atm_init_mct (eclock=..., cdata=..., x2a=..., a2x=..., nlfilename=..., _nlfilename=6) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/mct_coupling/atm_comp_mct.F90:174
#21 0x000000001006823c in component_mod::component_init_cc (eclock=..., comp=..., comp_init=0x102444e0 <atm_comp_mct::atm_init_mct>, infodata=..., nlfilename=..., seq_flds_x2c_fluxes=..., seq_flds_c2x_fluxes=..., _nlfilename=6, _seq_flds_x2c_fluxes=0, _seq_flds_c2x_fluxes=0)
at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/component_mod.F90:248
#22 0x00000000100473f0 in cime_comp_mod::cime_init () at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/cime_comp_mod.F90:1438
#23 0x000000001005fa40 in cime_driver () at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/cime_driver.F90:122
#24 0x000000001005fbf0 in main (argc=1, argv=0x7ffffd944ba6) at /gpfs/alpine/cli115/scratch/sarat/repos/scream/driver-mct/main/cime_driver.F90:23
#25 0x0000200006114078 in generic_start_main.isra () from /lib64/power9/libc.so.6
#26 0x0000200006114264 in __libc_start_main () from /lib64/power9/libc.so.6
#27 0x0000000000000000 in ?? ()
@sarats , did you set the
64 spa:
65 SPA Remap File: none
?? otherwise it will try to remap the ne4 data using and ne30->ne4 map.
Oh ok, that might explain this error message then:
14: 1: terminate called after throwing an instance of 'std::logic_error'
191 14: 1: what(): /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/physics/spa/spa_functions_impl.hpp:518: FAIL:
192 14: 1: ncol==spa_horiz_interp.source_grid_ncols
193 14: 1: ERROR update_spa_data_from_file: Number of columns in remap data (866 doesn't match the SPA data file (48602).
194 14: 1:
can i ask why there is discussion about ne4 IC on summit? I think Luca already ran on 1 rank (though did he report T<0), and i was able to run SMS.ne4_ne4.F2010-SCREAMv1 on 1 rank and saw
Atmosphere step = 119
model time = 0001-01-05 23:00:00
[EAMXX] Finalize ...
[EAMXX] Finalize ... done!
So I assume it was successful?
This is 2 ranks right? Maybe there is something wrong with multiple ranks...
@sarat -- can you run ./preview_run in your folder? for me, NRES and tasks-per-resource are messed up every time.
But 2 ranks (1 per node) running manually seem to work (I think Luca did the same earlier):
jsrun -X 1 --nrs 2 --rs_per_host 1 --tasks_per_rs 1 -d plane:1 --cpu_per_rs 7 --gpu_per_rs 1 --bind packed:smt:1 --latency_priority gpu-cpu --stdio_mode prepended --smpiargs="-gpu" /gpfs/alpine/cli115/proj-shared/onguba/e3sm_scratch/SMS.ne4_ne4.F2010-SCREAMv1.summit_gnugpu.20220525_170033_r56ez7/bld/e3sm.exe
bash-4.4$ ls -la atm.log.220525-193250
-rw------- 1 onguba cli115 8333 May 25 19:46 atm.log.220525-193250
bash-4.4$ tail atm.log.220525-193250
model time = 0001-01-05 21:00:00
Atmosphere step = 118
model time = 0001-01-05 22:00:00
Atmosphere step = 119
model time = 0001-01-05 23:00:00
[EAMXX] Finalize ...
[EAMXX] Finalize ... done!
jsrun -X 1 -n 2 -r 1 -a 1 -c 7 -g 1 --stdio_mode prepended <path>/e3sm.exe
This is a DEBUG build.
With the ne4 data file, the crash occurs as there is an attempt to use a large team size. This seems to be from the exception handler in scream/components/scream/src/dynamics/homme/atmosphere_dynamics.cpp:428.
4: 1: Kokkos::Impl::ParallelFor< Cuda > requested too large team size.
4: 0: WARNING: SPA Remap File has been set to 'NONE', assuming that SPA data and simulation are on the same grid - skipping horizontal interpolation
4: 1: WARNING: SPA Remap File has been set to 'NONE', assuming that SPA data and simulation are on the same grid - skipping horizontal interpolation
4: 0: Create Pool
4: 0: Tesla V100-SXM2-16GB
4: 0: INFORM: Automatically inserting fence() after every parallel_for
4: 0: [EAMXX] initialize_atm_procs ... done!
4: 0: [EAMxx::init] resolution-dependent device memory footprint: 70.230680MB
4: 0: [EAMxx::init] resolution-dependent host memory footprint: 26.815656MB
4: 0: Atmosphere step = 0
4: 0: model time = 0001-01-01 00:00:00
4: 0:
4: 0: 4: 1: terminate called after throwing an instance of 'std::logic_error'
4: 1: what(): /gpfs/alpine/cli115/scratch/sarat/repos/scream/components/scream/src/dynamics/homme/atmosphere_dynamics.cpp:428: FAIL:
4: 1: false
4: 1: Kokkos::Impl::ParallelFor< Cuda > requested too large team size.
4: 1: Traceback functionality not available
4: 1:
I found this related issue: https://github.com/E3SM-Project/HOMMEXX/issues/293
Note that some folks saw similar errors in Trilinos only when doing debug build/runs. https://github.com/kokkos/kokkos/issues/4095
That might explain why some folks might get a successful run if they are not using DEBUG.
Yes, that's similar to the workaround the Trilinos folks used as well, cap it to 512.
So, I can confirm that we can get a successful run with a non-DEBUG build with 2 ranks on 2 nodes. It even finalized properly and exited. Anyway, one mystery solved regarding why DEBUG builds didn't work.
Even the 2 ranks on single node with HOMMEXX_MPI_ON_DEVICE OFF worked. So, there seems to be an issue with GPU to GPU peer communication.
A minor thing to note is the creation of the 'fort.98' file. $ cat fort.98 P3_INIT (reading/creating look-up tables) ...
Just in case it is lost on slack -- the PAMI error shows up with standalone homme when using modules from cime. If using different gnu/cuda, homme standalone works on 1 node, on 2 nodes 12 resources, etc.
i am trying to search a combination of gnu/cuda, so that scream is build with it and homme does not have PAMI errors. So far, the problem is to build ekat.
@sarats when you did these experiments, were you always using the same modules? And were they the current CIME defaults (gnu/9.3 and cuda/11.5)? Or were they the ones you changed to in #1690?
can i ask why there is discussion about ne4 IC on summit? I think Luca already ran on 1 rank (though did he report T<0)
After updating master last week (to get the sub cycling fix), I was able to run (interactively) with 2+ ranks on 2+ nodes, by specifying correct resources to jsrun, and disabling MPI_ON_DEVICE in homme. I thought I was also able to run on 2 nodes with 12 ranks using case.submit. All of this was with summit's default modules (gcc/9.3 and cuda/11.5).
However, today I can't run using case.submit. Still have to try manually via interactive nodes.
Closing b/c stale (and we probably don't run on Summit anymore)
v1 with ne4 using 2 tasks with one task per node gets past initialization but still gets stuck.
This is the backtrace for the two processes.
cc @bartgol @jgfouca @oksanaguba @whannah1