Closed KateFriedman-NOAA closed 11 months ago
KateFriedman-NOAA
commented
1 year ago Opened https://github.com/NOAA-EMC/spack-stack/issues/471 to get the following modules added to the global-workflow-env/unified-dev module:
ncl/6.6.2
g2tmpl/1.10.2
gsi-ncdiag/1.0.0
crtm/2.4.0Also need gempak but it's not currently within spack-stack. Will load it separately for now and look at submitting a spack-stack issue to get it added to spack-stack.
KateFriedman-NOAA
commented
1 year ago I am able to run most of the system using the provided spack-stack install on Orion, however there are a few stumbling blocks to resolve:
interp_inc.xWorking with @climbfuji and @ulmononian I was able to get past hanging python scripts by switching to a different spack-stack environment put together by @ulmononian:
In attempt to fix the hanging Python script issue, I installed a version of the unified environment with py-netcdf4
and py-h5py built without mpi, as Dom suggested this might have been the root cause of the problem. - Cameronmodule use /work2/noaa/epic-ps/cbook/spack_work/spack-stack-1.2.0/envs/unified-env/install/modulefiles/Core
module load stack-intel
module load stack-intel-oneapi-mpiThis helped with issue 1. Issue 2 now occurs (once able to get to that step).
Last reply to email thread:
The second python script that was having issues got past the prior issues but hit a new one. The second python script
invokes an executable that appears to be hanging. Please see the following log on Orion (jump to the very bottom for
the timeout, python script starts line 1028 and the executable starts line 1074):
/work/noaa/stmp/kfriedma/comrot/spackcyc192/logs/2022010200/gfsanalcalc.log
I tried this analcalc job both with and without setting I_MPI_HYDRA_BOOTSTRAP=ssh --> no difference.
The chgres_inc.x exec is this exec: /work/noaa/global/kfriedma/git/develop_spack/exec/interp_inc.x
Orion-login-3[190] /work/noaa/stmp/kfriedma/comrot/spackcyc192$ ll /work/noaa/global/kfriedma/git/develop_spack/exec/interp_inc.x
lrwxrwxrwx 1 kfriedma global 87 Feb 16 15:45 /work/noaa/global/kfriedma/git/develop_spack/exec/interp_inc.x -> /work/noaa/global/kfriedma/git/develop_spack/sorc/gsi_utils.fd/install/bin/interp_inc.x*
That interp_inc.x exec comes from the GSI utils component, which was built with the intel 2018 spack-stack install.
Perhaps I need an intel 2018 version with the same mpi-less python within? And then rebuild the GSI components with that?This is where I am currently stuck.
KateFriedman-NOAA
commented
1 year ago Status Summary
I was able to get past the hanging python utilities by using a different spack-stack install made by @ulmononian (without MPI):
/work2/noaa/epic-ps/cbook/spack_work/spack-stack-1.2.0/envs/unified-env/install/modulefiles/CoreThe python in the analcalc jobs completed without hanging...however, the chgres_inc.x/interp_inc.x exec now fails with what appears to be an HDF5 error:
FATAL ERROR: CREATING FILE=inc.fullres.03: Permission denied
STOP.
999
999
999
999
999
999
999
999
999
srun: launch/slurm: _task_finish: Received task exit notification for 9 tasks of StepId=9900781.0 (status=0xe700).
srun: error: Orion-05-22: tasks 0-8: Exited with exit code 231
srun: launch/slurm: _step_signal: Terminating StepId=9900781.0
slurmstepd: error: *** STEP 9900781.0 ON Orion-05-22 CANCELLED AT 2023-03-31T09:48:00 ***
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
chgres_inc.x 0000000000493D1E Unknown Unknown Unknown
libpthread-2.17.s 00002B82A20825D0 Unknown Unknown Unknown
libc-2.17.so 00002B82A2D55603 epoll_wait Unknown Unknown
libucs.so.0.0.0 00002B82AC35F3AB ucs_event_set_wai Unknown Unknown
libucs.so.0.0.0 00002B82AC34642C Unknown Unknown Unknown
libpthread-2.17.s 00002B82A207ADD5 Unknown Unknown Unknown
libc-2.17.so 00002B82A2D5502D clone Unknown Unknown
HDF5-DIAG: Error detected in HDF5 (1.14.0) MPI-process 0:
#000: /work2/noaa/epic-ps/cbook/spack_work/spack-stack-1.2.0/cache/build_stage/spack-stage-hdf5-1.14.0-wcprhcxjrt7qbgfi3sqgdrj37wgd3mdc/spack-src/src/H5T.c line 1911 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.14.0) MPI-process 0:Log (starting line 1075): /work/noaa/stmp/kfriedma/comrot/spackcyc192/logs/2022010200/gfsanalcalc.log
Word from @GeorgeVandenberghe-NOAA is that the GSI fails with hdf5/1.1.40 (have previously used hdf5/1.10.6 with the GSI). Still using intel 2018 with the GSI execs.
Currently stuck here...need to retest or rebuild the GSI with hdf5/1.10.6 to see if we can get past this error and run the few remaining jobs to complete a full cycle with spack-stack.
Test on Orion:
clone with updated components: /work/noaa/global/kfriedma/git/develop_spack
EXPDIR: /work/noaa/global/kfriedma/expdir/spackcyc192
COMROT: /work/noaa/stmp/kfriedma/comrot/spackcyc192Note: clone is now several months old and does not have all GFSv17-dev updates since then, particularly the COM reorg updates that went in this week. Likely need to redo spack-stack updates in all components in new clone using global-workflow develop.
AlexanderRichert-NOAA
commented
1 year ago @KateFriedman-NOAA for what it's worth, looking at your UFS build logs, it looks like it's still using hpc-stack's parallelio somehow. The one in spack-stack is a shared library, so I think your UFS CMakeLists.txt and CMakeModules will need to be updated (see the current UFS develop branch; basically, use a more recent commit of CMakeModules, and remove the STATIC from the PIO line in ufs_model.fd/CMakeLists.txt).
KateFriedman-NOAA
commented
1 year ago @KateFriedman-NOAA for what it's worth, looking at your UFS build logs, it looks like it's still using hpc-stack's parallelio somehow. The one in spack-stack is a shared library, so I think your UFS CMakeLists.txt and CMakeModules will need to be updated (see the current UFS develop branch; basically, use a more recent commit of CMakeModules, and remove the
STATICfrom the PIO line in ufs_model.fd/CMakeLists.txt).
Ah, yes, let me double check this, I now remember having an issue with the UFS build. Let me get back to you on this...
GeorgeVandenberghe-NOAA
commented
1 year ago Remembering the good old days when we just integrated or numerically solved partial differential equations. :-)
On Mon, May 1, 2023 at 9:55 AM Kate Friedman @.***> wrote:
@KateFriedman-NOAA https://github.com/KateFriedman-NOAA for what it's worth, looking at your UFS build logs, it looks like it's still using hpc-stack's parallelio somehow. The one in spack-stack is a shared library, so I think your UFS CMakeLists.txt and CMakeModules will need to be updated (see the current UFS develop branch; basically, use a more recent commit of CMakeModules, and remove the STATIC from the PIO line in ufs_model.fd/CMakeLists.txt).
Ah, yes, let me double check this, I now remember having an issue with the UFS build. Let me get back to you on this...
— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/1310#issuecomment-1529733856, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FSVPEL7LOCVESUY5T3XD66DJANCNFSM6AAAAAAU2UPJQE . You are receiving this because you were mentioned.Message ID: @.***>
--
George W Vandenberghe
Lynker Technologies at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
@.***
301-683-3769(work) 3017751547(cell)
junwang-noaa
commented
1 year ago One concern I have with the shared library is that we can't run a executable on different platforms, e.g. run executable compiled on wcoss2 on gaea C5, or hera executable on Orion (we recently confirmed the baselines on hera/orion reproduce).
AlexanderRichert-NOAA
commented
1 year ago @junwang-noaa have you done the former (build on WCOSS2->run on C5) with executables built with hpc-stack? With WCOSS2, I think there are some inevitable shared dependencies because of Cray, but I haven't tried it (I guess those same libraries exist on C5, just different versions potentially, like cray-mpich).
KateFriedman-NOAA
commented
1 year ago @AlexanderRichert-NOAA
@KateFriedman-NOAA for what it's worth, looking at your UFS build logs, it looks like it's still using hpc-stack's parallelio somehow. The one in spack-stack is a shared library, so I think your UFS CMakeLists.txt and CMakeModules will need to be updated (see the current UFS develop branch; basically, use a more recent commit of CMakeModules, and remove the
STATICfrom the PIO line in ufs_model.fd/CMakeLists.txt).
Okie dokie...so I had changed modulefiles/ufs_common.lua to use the spack-stack parallelio module:
+parallelio_ver=os.getenv("parallelio_ver") or "2.5.7"
+load(pathJoin("parallelio", parallelio_ver))...but I remember trying to make that change and having issues so, like you said, using a more recent version would help. I will see if I can find time to try that, I'd probably want to update to newer versions of all components and remake the module/stack changes too. Will see if I can find time to do that before I go on leave this month. I have a bunch of higher priority tasks to complete first though. Thanks!
AlexanderRichert-NOAA
commented
1 year ago @KateFriedman-NOAA looking at your gfsanalcalc.log, I just noticed you're getting an issue that I was running into: MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case. I think this means it's running the tasks as separate jobs without MPI comms, but I could be wrong (so if this is normal/expected, please disregard this). In any case, I got around it in exglobal_forecast.sh by adding export I_MPI_PMI_LIBRARY=/opt/slurm/lib/libpmi.so right before the $APRUN_FV3 line (this isn't the ideal way to do it of course). Again assuming this isn't the expected behavior, can you try making a similar change to this job and see if it changes anything?
It looks like this variable gets set by the impi/2022.1.2 module and the value looks correct, so I don't know why it's not getting propagated...
AlexanderRichert-NOAA
commented
1 year ago By setting I_MPI_PMI_LIBRARY I'm able to get to the point where chgres_inc.x just hangs indefinitely. Switching to hdf5/1.12.2 at runtime does not fix it, but I have not yet tried building with it. I have also tried building and running the executable using intel 2022 to avoid the mismatch between modules (it was building with old intel but running with new intel), but no luck. It appears that none of the processes are making it through the nf90_get_var call in gsi_utils.fd/src/netcdf_io/interp_inc.fd/driver.f90 (line 346). I'll keep digging and see if I can figure out where exactly it's getting stuck...
AlexanderRichert-NOAA
commented
1 year ago No luck with netcdf-c@4.9.2, nor with hdf5@1.12.2...
climbfuji
commented
1 year ago Can yoy try building with hdf5 1.12.2 and the default netcdf combination in the spack-stack ue?
On May 2, 2023, at 11:44 PM, Alex Richert @.***> wrote:
No luck with @.*** :(
— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/1310#issuecomment-1532477641, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RIAR42ZLGMICWNNHW3XEHWE7ANCNFSM6AAAAAAU2UPJQE. You are receiving this because you were mentioned.
AlexanderRichert-NOAA
commented
1 year ago Yeah, I tried building and running that executable with spack-stack-1.3.1 (hdf5@1.12.2/netcdf-c@4.9.2/netcdf-fortran@4.6.0) and it was still hanging in the same place.
GeorgeVandenberghe-NOAA
commented
1 year ago Does the executable built with the older hpc-stack stuff, run to completion or does that also hang?
On Wed, May 3, 2023 at 4:13 PM Alex Richert @.***> wrote:
Yeah, I tried building and running that executable with spack-stack-1.3.1 @.**@*.**@*.***) and it was still hanging in the same place.
— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/1310#issuecomment-1533324990, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FVJ75Q3ZR24VZVIXN3XEJ7YPANCNFSM6AAAAAAU2UPJQE . You are receiving this because you were mentioned.Message ID: @.***>
--
George W Vandenberghe
Lynker Technologies at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
@.***
301-683-3769(work) 3017751547(cell)
AlexanderRichert-NOAA
commented
1 year ago I just tested that-- It does run to completion building with the existing hpc-stack libraries, so it's not a matter of misconfiguration at runtime of MPI settings, etc. I'm building my own stack with intel 2018 and I'll just keep changing things until it works (probably starting with hdf5 version)...
For my own memory-- I've now tried hdf5 builds with and without thread safety enabled, so that doesn't seem to be the issue.
AlexanderRichert-NOAA
commented
1 year ago I built a spack-stack-based stack with hdf@1.10.6 and, for better or for worse, that fixed it. I'm going to see if I can narrow down the version where it breaks. It seems odd that it would break in the context of a high-level netcdf function (n*_get_var), so I wonder if it has to do with how it's being used in the context of MPI.
GeorgeVandenberghe-NOAA
commented
1 year ago I ran into gsi hang issues in the benchmark package with hdf5/1.14.0 and GSI people have not run it down so I had to give up and provide and build a separate set of hdf5/1.10.6 dependencies for the GSI. To avoid dependency clashes (mistakes happen) I also separated the benchmarks so they're now GSI and .. everything.else. Running this down and backtracking delayed benchmark release for ten days
A separate set of issues occurred with netcdf/-fortran/4.6.0 in the GFDL MOM6 ocean model coupled with UFS. This appears to be a MOM6 problem but it is masked by using netcdf-fortran/4.5.3 so I backtracked and used that.
On Thu, May 4, 2023 at 6:52 PM Alex Richert @.***> wrote:
I built a spack-stack-based stack with @.** and, for better or for worse, that fixed it. I'm going to see if I can narrow down the version where it breaks. It seems odd that it would break in the context of a high-level netcdf function (n_get_var), so I wonder if it has to do with how it's being used in the context of MPI.
— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/1310#issuecomment-1535249891, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FQA7RJCTIPUFQFEV6DXEP3FHANCNFSM6AAAAAAU2UPJQE . You are receiving this because you were mentioned.Message ID: @.***>
--
George W Vandenberghe
Lynker Technologies at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
@.***
301-683-3769(work) 3017751547(cell)
AlexanderRichert-NOAA
commented
1 year ago Here's a summary of my findings on Orion in terms of hdf5 versions and interp_inc.x: Working (no hang): 1.10.6, 1.10.7, 1.10.8 Not working (hanging): 1.10.9, 1.12.2, 1.14.0
So whatever the issue is, it appears to have emerged in the 1.10.8->1.10.9 transition. I'll see if I can pin down the culprit.
KateFriedman-NOAA
commented
1 year ago To document this from an email thread with @climbfuji:
I have wgrib2@3.1.1 on Orion in:
/work2/noaa/da/dheinzel-new/spack-stack-unified-env-io-updates/envs/unified-dev-test4-intel-18
and
/work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1/envs/unified-env
(the latter has a default=newer Intel and a GNU environment)Before I go digging deeper in terms of hdf5 code, @KateFriedman-NOAA if we had an intel 18-based stack with hdf5@1.10.6, would that at least take care of things in the short-to-mid term, especially in terms of transitioning to spack-stack?
KateFriedman-NOAA
commented
1 year ago if we had an intel 18-based stack with hdf5@1.10.6, would that at least take care of things in the short-to-mid term, especially in terms of transitioning to spack-stack?
For supporting the GSI moving to spack-stack, yes, probably. We would need to decide if and how we mix intel versions across the GFS or not...meaning, GSI builds with 2018 but all other pieces build with 2022 and the whole system (from the workflow level) runs with 2022 loaded.
Our move to EPIC stack is planning to move everything to 2022 with the GSI being a question right now as well.
GeorgeVandenberghe-NOAA
commented
1 year ago Is there any path forward for the GSI to address it's issues with intel/2022.x and with HDF5/1.12 and 1.14?
On Tue, May 9, 2023 at 4:49 PM Kate Friedman @.***> wrote:
if we had an intel 18-based stack with @.***, would that at least take care of things in the short-to-mid term, especially in terms of transitioning to spack-stack?
For supporting the GSI moving to spack-stack, yes, probably. We would need to decide if and how we mix intel versions across the GFS or not...meaning, GSI builds with 2018 but all other pieces build with 2022 and the whole system (from the workflow level) runs with 2022 loaded.
Our move to EPIC stack is planning to move everything to 2022 with the GSI being a question right now as well.
— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/1310#issuecomment-1540526436, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FSRMKU2DB5LFKEKMYDXFJYSPANCNFSM6AAAAAAU2UPJQE . You are receiving this because you were mentioned.Message ID: @.***>
--
George W Vandenberghe
Lynker Technologies at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
@.***
301-683-3769(work) 3017751547(cell)
AlexanderRichert-NOAA
commented
1 year ago Okay, I at least have a fix for the gfsanalcalc/interp_inc.x issue. For whatever reason, as of hdf@1.10.9, certain parallel operations require the involvement of the root process (MPI_RANK==0), and if it's not available for work, it hangs. So my workaround is to make [MPI world size]-1 the root process (replace mype == 0 with mype = npes-1 and so on in netcdf_io/interp_inc.fd/driver.f90), such that the parallel work is done on ranks 0 through 8 and rank 9 is the "root" MPI process. See /work/noaa/nems/arichert/develop_spack/sorc/gsi_utils.fd/src/netcdf_io/interp_inc.fd/driver.f90_WORKING
I'll submit a bug report to HDF5 and see what they say, and I'll cross-reference the issue here. @GeorgeVandenberghe-NOAA any chance this could explain other issues you've encountered? And if so, is this kind of workaround viable as far as level of effort required?
GeorgeVandenberghe-NOAA
commented
1 year ago I will have to think about this one but first thought is that mpi_io operations are collective and all ranks in a collective must complete or the collective hangs.. standard behavior. If so this is an application design flaw or error.
On Friday, May 19, 2023, Alex Richert @.***> wrote:
Okay, I at least have a fix for the gfsanalcalc/interp_inc.x issue. For whatever reason, as of @.***, certain parallel operations require the involvement of the root process (MPI_RANK==0), and if it's not available for work, it hangs. So my workaround is to make [MPI world size]-1 the root process (replace mype == 0 with mype = npes-1 and so on in netcdf_io/interp_inc.fd/driver.f90), such that the parallel work is done on ranks 0 through 8 and rank 9 is the "root" MPI process. See /work/noaa/nems/arichert/develop_spack/sorc/gsi_utils.fd/src/netcdf_io/interp_inc.fd/driver.f90_WORKING
I'll submit a bug report to HDF5 and see what they say, and I'll cross-reference the issue here. @GeorgeVandenberghe-NOAA any chance this could explain other issues you've encountered? And if so, is this kind of workaround viable as far as level of effort required?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci4.googleusercontent.com/proxy/igY4pgW27fDPW2tVGIhq89-aS00g4jIKd_lGXhN7vn7bLpVe0FK6i43Pg-slbD3lfVUMid2yAvohaxcei5lyn0TCrq3Qx_DzqXyLktAK6uhlG6CYiL0vEk79oqh0ywd8co5pCSVu2T6gYqB-hc8VZkIphZpgcrb41nmZldIZLEgVXWNrnF2oqx7zfTZDWyqV_JISDK2cXHDI7WKUEJwWscvxdkqCS-jHPE2W_E_YfjeYnH705Q=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FVTHBOO4P4JTGYNEELXHABOVA5CNFSM6AAAAAAU2UPJQGWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTS4WU2AC.gif>Message ID: @.***>
--
George W Vandenberghe
Lynker Technologies at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
@.***
301-683-3769(work) 3017751547(cell)
KateFriedman-NOAA
commented
11 months ago Being continued in issue #1868. Closing.
Description
This issue documents work to test out a test install of spack-stack on Orion provided by @climbfuji and others. Details provided by Dom:
Issues encountered during testing and requests to add modules will be submitted via the spack-stack repo: https://github.com/NOAA-EMC/spack-stack
Primary relevant spack-stack issue: https://github.com/NOAA-EMC/spack-stack/pull/454