Closed StevePny closed 2 years ago
The 'fix_orog' files are publicly available. See our documentation for the community release: https://noaa-emcufs-utils.readthedocs.io/en/ufs-v2.0.0/ufs_utils.html#program-inputs-and-outputs-for-global-applications
I am puzzled by the error in sfc_climo_gen. It should run with a multiple of six mpi tasks. I usually use 24 tasks. But I can try running with six tasks on WCOSS.
Hi George, the site that you provided is what I was following. It lists the files needed (see below), but it doesn't provide links. One of the links on the page might lead to these files, but if so, I haven't been able to find the right one.
Input data:
The “grid” files (CRES_grid.tile#.nc) containing the geo-reference records for the grid - (NetCDF). Created by the make_hgrid or regional_esg_grid programs.
Global 30-arc-second University of Maryland land cover data. Used to create the land-sea mask.
./fix/fix_orog/landcover30.fixed (unformatted binary)
Global 30-arc-second USGS GMTED2010 orography data.
./fix/fix_orog/gmted2010.30sec.int (unformatted binary)
30-arc-second RAMP Antarctic terrain data (Radarsat Antarctic Mapping Project)
./fix/fix_orog/thirty.second.antarctic.new.bin (unformatted binary)
The UFS_UTILS script "fix/link_fixdirs.sh" pulls these from NOAA HPC:
#------------------------------
#--model fix fields
#------------------------------
if [ $machine == "cray" ]; then
FIX_DIR="/gpfs/hps3/emc/global/noscrub/emc.glopara/git/fv3gfs/fix"
elif [ $machine = "dell" ]; then
FIX_DIR="/gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/fv3gfs/fix"
elif [ $machine = "hera" ]; then
FIX_DIR="/scratch1/NCEPDEV/global/glopara/fix"
elif [ $machine = "jet" ]; then
FIX_DIR="/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix"
elif [ $machine = "orion" ]; then
FIX_DIR="/work/noaa/global/glopara/fix"
elif [ $machine = "s4" ]; then
FIX_DIR="/data/prod/glopara/fix"
fi
Thanks George. Those sites don't have all the necessary files. I found the rest (e.g. vegetation_type.modis.igbp.0.05.nc) here: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/fix_sfc_climo/
Could you add these links to the corresponding section 3.6.3 on the https://noaa-emcufs-utils.readthedocs.io website?
I found the solution to the initial MPI problem -
When I run the sfc_climo_gen exe I see:
+ mpirun --allow-run-as-root -np 6 /UFS_UTILS/exec/sfc_climo_gen
- INITIALIZE ESMF
- INITIALIZE ESMF
- INITIALIZE ESMF
- CALL VMGetGlobal
- CALL VMGet
- NPETS IS 1
- LOCAL PET 0
- READ SETUP NAMELIST, LOCALPET: 0
- OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc
- INITIALIZE ESMF
- CALL VMGetGlobal
- CALL VMGet
- NPETS IS 1
- LOCAL PET 0
- READ SETUP NAMELIST, LOCALPET: 0
- OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc
- INITIALIZE ESMF
- CALL VMGetGlobal
- CALL VMGet
- NPETS IS 1
- LOCAL PET 0
- READ SETUP NAMELIST, LOCALPET: 0
- OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc
- CALL VMGetGlobal
- CALL VMGet
- NPETS IS 1
- LOCAL PET 0
- READ SETUP NAMELIST, LOCALPET: 0
- OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc
- READ NUMBER OF TILES
- READ TILE NAMES
- CALL VMGetGlobal
- CALL VMGet
- NPETS IS 1
- LOCAL PET 0
- READ SETUP NAMELIST, LOCALPET: 0
- OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc
- READ NUMBER OF TILES
- READ TILE NAMES
- READ NUMBER OF TILES
- READ TILE NAMES
- NUMBER OF TILES, MODEL GRID IS 6
- FATAL ERROR: MUST RUN THIS PROGRAM WITH A TASK COUNT THAT
- IS A MULTIPLE OF THE NUMBER OF TILES.
- NUMBER OF TILES, MODEL GRID IS 6
- FATAL ERROR: MUST RUN THIS PROGRAM WITH A TASK COUNT THAT
- IS A MULTIPLE OF THE NUMBER OF TILES.
- NUMBER OF TILES, MODEL GRID IS 6
- FATAL ERROR: MUST RUN THIS PROGRAM WITH A TASK COUNT THAT
- IS A MULTIPLE OF THE NUMBER OF TILES.
There is a required option when building ESMF. Even though openmpi is present, it defaults to a single-processor bypass mode when building specifically on Linux systems (e.g. in a Linux docker container):
From their documentation: "ESMF_COMM Possible value: system-dependent ... Alternatively, ESMF comes with a single-processor MPI-bypass library which is the default for Linux and Darwin systems. To force the use of this bypass library set ESMF_COMM equal to "mpiuni".
Also, ESMF must be built with this environment variable to enable netcdf:
ENV ESMF_NETCDF="standard"
Otherwise, I get a cryptic error about IOSTAT 49, which I tracked down with these resources: https://earthsystemmodeling.org/docs/release/ESMF_5_2_0/ESMC_crefdoc/node9.html#SECTION09020000000000000000 https://github.com/NOAA-EMC/fv3gfs/blob/master/sorc/fv3gfs.fd/NEMS/src/module_MEDIATOR_SpaceWeather.F90 https://earthsystemmodeling.org/docs/release/ESMF_5_2_0/ESMF_usrdoc/node9.html
I still have a problem with the routine ending prematurely:
- OPEN SOURCE FILE /fix_sfc_climo/snowfree_albedo.4comp.0.05.nc
- CALL FieldScatter FOR SOURCE GRID DATA.
- CALL FieldRegridStore.
- CALL FieldRegridStore.
- CALL FieldRegridStore.
- CALL FieldRegridStore.
- CALL FieldRegridStore.
- CALL FieldRegridStore.
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node 549771a6c3cd exited on signal 9 (Killed).
--------------------------------------------------------------------------
@GeorgeGayno-NOAA do you have any idea what this could be? Perhaps a netcdf error?
Thanks George. Those sites don't have all the necessary files. I found the rest (e.g. vegetation_type.modis.igbp.0.05.nc) here: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/fix_sfc_climo/
Could you add these links to the corresponding section 3.6.3 on the https://noaa-emcufs-utils.readthedocs.io website?
I found the solution to the initial MPI problem -
When I run the sfc_climo_gen exe I see:
+ mpirun --allow-run-as-root -np 6 /UFS_UTILS/exec/sfc_climo_gen - INITIALIZE ESMF - INITIALIZE ESMF - INITIALIZE ESMF - CALL VMGetGlobal - CALL VMGet - NPETS IS 1 - LOCAL PET 0 - READ SETUP NAMELIST, LOCALPET: 0 - OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc - INITIALIZE ESMF - CALL VMGetGlobal - CALL VMGet - NPETS IS 1 - LOCAL PET 0 - READ SETUP NAMELIST, LOCALPET: 0 - OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc - INITIALIZE ESMF - CALL VMGetGlobal - CALL VMGet - NPETS IS 1 - LOCAL PET 0 - READ SETUP NAMELIST, LOCALPET: 0 - OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc - CALL VMGetGlobal - CALL VMGet - NPETS IS 1 - LOCAL PET 0 - READ SETUP NAMELIST, LOCALPET: 0 - OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc - READ NUMBER OF TILES - READ TILE NAMES - CALL VMGetGlobal - CALL VMGet - NPETS IS 1 - LOCAL PET 0 - READ SETUP NAMELIST, LOCALPET: 0 - OPEN MODEL GRID MOSAIC FILE: /tmp/out/C96/C96_mosaic.nc - READ NUMBER OF TILES - READ TILE NAMES - READ NUMBER OF TILES - READ TILE NAMES - NUMBER OF TILES, MODEL GRID IS 6 - FATAL ERROR: MUST RUN THIS PROGRAM WITH A TASK COUNT THAT - IS A MULTIPLE OF THE NUMBER OF TILES. - NUMBER OF TILES, MODEL GRID IS 6 - FATAL ERROR: MUST RUN THIS PROGRAM WITH A TASK COUNT THAT - IS A MULTIPLE OF THE NUMBER OF TILES. - NUMBER OF TILES, MODEL GRID IS 6 - FATAL ERROR: MUST RUN THIS PROGRAM WITH A TASK COUNT THAT - IS A MULTIPLE OF THE NUMBER OF TILES.
There is a required option when building ESMF. Even though openmpi is present, it defaults to a single-processor bypass mode when building specifically on Linux systems (e.g. in a Linux docker container):
From their documentation: "ESMF_COMM Possible value: system-dependent ... Alternatively, ESMF comes with a single-processor MPI-bypass library which is the default for Linux and Darwin systems. To force the use of this bypass library set ESMF_COMM equal to "mpiuni".
Also, ESMF must be built with this environment variable to enable netcdf:
ENV ESMF_NETCDF="standard"
Otherwise, I get a cryptic error about IOSTAT 49, which I tracked down with these resources: https://earthsystemmodeling.org/docs/release/ESMF_5_2_0/ESMC_crefdoc/node9.html#SECTION09020000000000000000 https://github.com/NOAA-EMC/fv3gfs/blob/master/sorc/fv3gfs.fd/NEMS/src/module_MEDIATOR_SpaceWeather.F90 https://earthsystemmodeling.org/docs/release/ESMF_5_2_0/ESMF_usrdoc/node9.html
To do: The problem with the platform dependent "aprun" being hard-coded in the final script sfc_climo_gen.sh still needs to be corrected in the repo. Elsewhere in the scripts this type of command is passed as an environment variable.
Can you use the APRUN_SFC variable?
For now, every time I build I use: sed -i 's/aprun -j 1 -n 6 -N 6/mpirun --allow-run-as-root -n 6/' /UFS_UTILS/ush/sfc_climo_gen.sh
I still have a problem with the routine ending prematurely:
- OPEN SOURCE FILE /fix_sfc_climo/snowfree_albedo.4comp.0.05.nc - CALL FieldScatter FOR SOURCE GRID DATA. - CALL FieldRegridStore. - CALL FieldRegridStore. - CALL FieldRegridStore. - CALL FieldRegridStore. - CALL FieldRegridStore. - CALL FieldRegridStore. -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 0 on node 549771a6c3cd exited on signal 9 (Killed). --------------------------------------------------------------------------
@GeorgeGayno-NOAA do you have any idea what this could be? Perhaps a netcdf error?
Looks like it stops in the ESMF regrid step. Check the ESMF log files. They begin with "PET".
There is no error message. What should the output look like? This is as far as they get (see below).
In the past when the process has stopped with no explanation it has been a memory issue, but I've increased the memory of the Docker container quite a bit and it still halts in the same place.
20220329 025220.269 INFO PET0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.269 INFO PET0 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.269 INFO PET0 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.269 INFO PET0 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.269 INFO PET0 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.269 INFO PET0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.269 INFO PET0 Running with ESMF Version : v8.3.0b11
20220329 025220.269 INFO PET0 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.269 INFO PET0 ESMF library build location : /tmp/esmf
20220329 025220.269 INFO PET0 ESMF_COMM : openmpi
20220329 025220.271 INFO PET0 ESMF_MOAB : enabled
20220329 025220.271 INFO PET0 ESMF_LAPACK : enabled
20220329 025220.271 INFO PET0 ESMF_NETCDF : enabled
20220329 025220.271 INFO PET0 ESMF_PNETCDF : disabled
20220329 025220.271 INFO PET0 ESMF_PIO : enabled
20220329 025220.271 INFO PET0 ESMF_YAMLCPP : enabled
~
"Can you use the APRUN_SFC variable?" Yes, I'm not sure how I missed that - I think it is my editor putting dark blue on a black background. That is what I was looking for.
There is no error message. What should the output look like? This is as far as they get (see below).
In the past when the process has stopped with no explanation it has been a memory issue, but I've increased the memory of the Docker container quite a bit and it still halts in the same place.
20220329 025220.269 INFO PET0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 20220329 025220.269 INFO PET0 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!! 20220329 025220.269 INFO PET0 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!! 20220329 025220.269 INFO PET0 !!! FOR PRODUCTION RUNS, USE: !!! 20220329 025220.269 INFO PET0 !!! ESMF_LOGKIND_Multi_On_Error !!! 20220329 025220.269 INFO PET0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 20220329 025220.269 INFO PET0 Running with ESMF Version : v8.3.0b11 20220329 025220.269 INFO PET0 ESMF library build date/time: "Mar 29 2022" "02:20:31" 20220329 025220.269 INFO PET0 ESMF library build location : /tmp/esmf 20220329 025220.269 INFO PET0 ESMF_COMM : openmpi 20220329 025220.271 INFO PET0 ESMF_MOAB : enabled 20220329 025220.271 INFO PET0 ESMF_LAPACK : enabled 20220329 025220.271 INFO PET0 ESMF_NETCDF : enabled 20220329 025220.271 INFO PET0 ESMF_PNETCDF : disabled 20220329 025220.271 INFO PET0 ESMF_PIO : enabled 20220329 025220.271 INFO PET0 ESMF_YAMLCPP : enabled ~
It looks like it is not an ESMF problem.
I am not familiar with your system. Are you requesting enough wall clock time?
Each process writes its own log file. Do any of the other PET files show an error?
"I am not familiar with your system. Are you requesting enough wall clock time?" I'm on a Mac M1 running in an ubuntu docker container using arm64. Docker is currently configured with:
"Each process writes its own log file. Do any of the other PET files show an error?" Not that I see:
root@1c2934f2fa6a:/UFS_UTILS/ush# cat /rundir/tmp/sfcfields/PET*
20220329 025220.269 INFO PET0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.269 INFO PET0 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.269 INFO PET0 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.269 INFO PET0 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.269 INFO PET0 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.269 INFO PET0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.269 INFO PET0 Running with ESMF Version : v8.3.0b11
20220329 025220.269 INFO PET0 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.269 INFO PET0 ESMF library build location : /tmp/esmf
20220329 025220.269 INFO PET0 ESMF_COMM : openmpi
20220329 025220.271 INFO PET0 ESMF_MOAB : enabled
20220329 025220.271 INFO PET0 ESMF_LAPACK : enabled
20220329 025220.271 INFO PET0 ESMF_NETCDF : enabled
20220329 025220.271 INFO PET0 ESMF_PNETCDF : disabled
20220329 025220.271 INFO PET0 ESMF_PIO : enabled
20220329 025220.271 INFO PET0 ESMF_YAMLCPP : enabled
20220329 025220.253 INFO PET1 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.253 INFO PET1 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.253 INFO PET1 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.253 INFO PET1 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.253 INFO PET1 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.253 INFO PET1 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.253 INFO PET1 Running with ESMF Version : v8.3.0b11
20220329 025220.253 INFO PET1 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.253 INFO PET1 ESMF library build location : /tmp/esmf
20220329 025220.253 INFO PET1 ESMF_COMM : openmpi
20220329 025220.256 INFO PET1 ESMF_MOAB : enabled
20220329 025220.256 INFO PET1 ESMF_LAPACK : enabled
20220329 025220.256 INFO PET1 ESMF_NETCDF : enabled
20220329 025220.256 INFO PET1 ESMF_PNETCDF : disabled
20220329 025220.256 INFO PET1 ESMF_PIO : enabled
20220329 025220.256 INFO PET1 ESMF_YAMLCPP : enabled
20220329 025220.249 INFO PET2 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.249 INFO PET2 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.249 INFO PET2 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.249 INFO PET2 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.249 INFO PET2 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.249 INFO PET2 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.249 INFO PET2 Running with ESMF Version : v8.3.0b11
20220329 025220.250 INFO PET2 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.250 INFO PET2 ESMF library build location : /tmp/esmf
20220329 025220.250 INFO PET2 ESMF_COMM : openmpi
20220329 025220.253 INFO PET2 ESMF_MOAB : enabled
20220329 025220.253 INFO PET2 ESMF_LAPACK : enabled
20220329 025220.253 INFO PET2 ESMF_NETCDF : enabled
20220329 025220.253 INFO PET2 ESMF_PNETCDF : disabled
20220329 025220.253 INFO PET2 ESMF_PIO : enabled
20220329 025220.253 INFO PET2 ESMF_YAMLCPP : enabled
20220329 025220.276 INFO PET3 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.276 INFO PET3 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.276 INFO PET3 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.276 INFO PET3 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.276 INFO PET3 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.276 INFO PET3 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.276 INFO PET3 Running with ESMF Version : v8.3.0b11
20220329 025220.276 INFO PET3 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.276 INFO PET3 ESMF library build location : /tmp/esmf
20220329 025220.276 INFO PET3 ESMF_COMM : openmpi
20220329 025220.278 INFO PET3 ESMF_MOAB : enabled
20220329 025220.278 INFO PET3 ESMF_LAPACK : enabled
20220329 025220.278 INFO PET3 ESMF_NETCDF : enabled
20220329 025220.278 INFO PET3 ESMF_PNETCDF : disabled
20220329 025220.278 INFO PET3 ESMF_PIO : enabled
20220329 025220.278 INFO PET3 ESMF_YAMLCPP : enabled
20220329 025220.261 INFO PET4 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.261 INFO PET4 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.261 INFO PET4 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.261 INFO PET4 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.261 INFO PET4 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.261 INFO PET4 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.261 INFO PET4 Running with ESMF Version : v8.3.0b11
20220329 025220.261 INFO PET4 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.261 INFO PET4 ESMF library build location : /tmp/esmf
20220329 025220.261 INFO PET4 ESMF_COMM : openmpi
20220329 025220.265 INFO PET4 ESMF_MOAB : enabled
20220329 025220.265 INFO PET4 ESMF_LAPACK : enabled
20220329 025220.265 INFO PET4 ESMF_NETCDF : enabled
20220329 025220.265 INFO PET4 ESMF_PNETCDF : disabled
20220329 025220.265 INFO PET4 ESMF_PIO : enabled
20220329 025220.265 INFO PET4 ESMF_YAMLCPP : enabled
20220329 025220.257 INFO PET5 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.257 INFO PET5 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20220329 025220.257 INFO PET5 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20220329 025220.257 INFO PET5 !!! FOR PRODUCTION RUNS, USE: !!!
20220329 025220.257 INFO PET5 !!! ESMF_LOGKIND_Multi_On_Error !!!
20220329 025220.257 INFO PET5 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20220329 025220.257 INFO PET5 Running with ESMF Version : v8.3.0b11
20220329 025220.257 INFO PET5 ESMF library build date/time: "Mar 29 2022" "02:20:31"
20220329 025220.257 INFO PET5 ESMF library build location : /tmp/esmf
20220329 025220.257 INFO PET5 ESMF_COMM : openmpi
20220329 025220.259 INFO PET5 ESMF_MOAB : enabled
20220329 025220.259 INFO PET5 ESMF_LAPACK : enabled
20220329 025220.259 INFO PET5 ESMF_NETCDF : enabled
20220329 025220.259 INFO PET5 ESMF_PNETCDF : disabled
20220329 025220.259 INFO PET5 ESMF_PIO : enabled
20220329 025220.259 INFO PET5 ESMF_YAMLCPP : enabled
Ok, solved. It turns out it was in fact a resource issue, I just didn't increase the memory enough. I maxed it out at 32 GB and the whole thing runs to completion:
finishing with:
+ mv vegetation_type.tile1.nc /rundir/out/C96/fix_sfc/C96.vegetation_type.tile1.nc
+ for files in *.nc
+ [[ -f vegetation_type.tile2.nc ]]
+ mv vegetation_type.tile2.nc /rundir/out/C96/fix_sfc/C96.vegetation_type.tile2.nc
+ for files in *.nc
+ [[ -f vegetation_type.tile3.nc ]]
+ mv vegetation_type.tile3.nc /rundir/out/C96/fix_sfc/C96.vegetation_type.tile3.nc
+ for files in *.nc
+ [[ -f vegetation_type.tile4.nc ]]
+ mv vegetation_type.tile4.nc /rundir/out/C96/fix_sfc/C96.vegetation_type.tile4.nc
+ for files in *.nc
+ [[ -f vegetation_type.tile5.nc ]]
+ mv vegetation_type.tile5.nc /rundir/out/C96/fix_sfc/C96.vegetation_type.tile5.nc
+ for files in *.nc
+ [[ -f vegetation_type.tile6.nc ]]
+ mv vegetation_type.tile6.nc /rundir/out/C96/fix_sfc/C96.vegetation_type.tile6.nc
+ exit 0
+ err=0
+ '[' 0 '!=' 0 ']'
+ '[' uniform = regional_gfdl ']'
+ '[' uniform = regional_esg ']'
+ '[' uniform = nest ']'
+ exit
@StevePny Is everything working for you? Can we close this issue?
It seems as if the UFS_UTILS has not been run outside of NOAA HPC environments. In order to run locally (in a docker container), there are a couple instances where the build expects aprun to be used. First, if APRUN is not provided as an environment variable, then in fv3gfs_make_grid.sh it defaults to "time" (APRUN=${APRUN:-time}), but this causes a problem within the bash shell (since 'time' is a built-in in bash). To address this, I installed time using "apt-get install -y time" and then provided an explicit location for APRUN=/usr/bin/time as an environment variable. However, "aprun" is also hard-coded in the final script sfc_climo_gen.sh. Here, it is not clear how to modify the script to run.
I tried to hard-code a correction, but it comes up with a strange error, since the number of tiles is 6 and so is the number of tasks:
Also, the fix_orog/ files were not publicly accessible (as far as I could find). I had to track them down via HPC access. I would recommend providing these files on a public ftp.