dudhia
commented
2 years ago My recollection is that the Z staggering only applies to the output/input dimension and the variable in the code is always kde. What would we need to do to repeat your error with standard code? Sounds like a memory size issue. The urban array can be quite large. Does the number of processors you use for real.exe matter?
On Tue, Feb 22, 2022 at 10:05 AM William Gustafson @.***> wrote:
Heng Xiao @.*** https://github.com/hengxiao80) and I have been trying to work through an issue where real.exe is crashing while trying to read URB_PARAM. This only happens when we use large domains. One thing I noticed is that URB_PARAM is not indicated to have Z-staggering in Registry.EM_COMMON. I vaguely remember from work I did many years ago in WRF-Chem that variables that use one of the constant dimensions must have that marked as Z-staggered, even if it really isn't thought of that way. Is that correct? If so, then this is likely a bug for the definition of URB_PARAM that may also require a few changes to looping parameters in the code.
The symptom we are seeing is a buffer overflow in the read routines. We actually have the urban parameterization turned off, so URB_PARAM is not in our input met_em files. So, this read attempt is superfluous. Our workaround is to just turn off the "i1" read in Registry.EM_COMMON, but this may not work for others' situations if they run into this same problem.
Note that we tried changing the staggering to "Z", but we still got the crash if we left the read turned on. So, I do not know what the actual issue is. However, I wanted to report the possible Z-staggering issue for the urban developers to be aware of.
The issue happens for the 2nd domain when processing domains set up with
max_dom = 2 e_we = 751, 2146 e_sn = 866, 2776 e_vert = 150, 150
The error messages reported in rsl.error.0000 are:
d02 2019-01-29_12:00:00 NetCDF error in wrf_io.F90, line 2885 Varname URB_PARAM 0: MPICH2 Error: Failed to register memory address 0x2aabb2fcc000 with length of 0x641000 (6557696) bytes. 0: Unable to register memory at the requested address. This may indicate an application bug. See process virtual memory mappings below:
< EXCLUDING LOTS OF EXTRA OUTPUT WE HAVE TURNED ON THAT IS IRRELEVANT... >
Rank 0 [Fri Feb 4 20:43:06 2022] [c0-0c0s3n3] Fatal error in PMPI_Scatterv: Other MPI error, error stack: PMPI_Scatterv(448)........................: MPI_Scatterv(sbuf=0x2aac0229e020, scnts=0x13f9f4f0, displs=0x14d21e80, MPI_CHAR, rbuf=0x7fffff9934d0, rcount=6614784, MPI_CHAR, root=0, comm=0xc4000002) failed MPIR_Scatterv_impl(234)...................: MPIR_CRAY_Scatterv(178)...................: MPIR_Scatterv(164)........................: MPIR_Waitall_impl(221)....................: MPIDI_CH3I_Progress(537)..................: MPID_nem_mpich_blocking_recv(1140)........: MPID_nem_gni_poll(1632)...................: MPID_nem_gni_progress_lmt_start_send(2010): MPID_nem_gni_lmt_mem_register(130)........: MPID_nem_gni_dreg_register(569)...........: UDREG_Register 1 forrtl: error (76): Abort trap signal Image PC Routine Line Source real_debug.exe 000000000BA3E624 Unknown Unknown Unknown libpthread-2.26.s 00002AAAADE272D0 Unknown Unknown Unknownlibc-2.26.so 00002AAAAE270520 gsignal Unknown Unknownlibc-2.26.so 00002AAAAE271B01 abort Unknown Unknown libmpich_intel.so 00002AAAADA545F8 Unknown Unknown Unknown libmpich_intel.so 00002AAAAD9DD462 MPIR_Handle_fatal Unknown Unknown libmpich_intel.so 00002AAAAD9DD59E MPIR_Err_return_c Unknown Unknown libmpich_intel.so 00002AAAAD8F90E2 MPI_Scatterv Unknown Unknown real_debug.exe 0000000001E854D6 Unknown Unknown Unknown real_debug.exe 00000000014DCBCB wrf_global_to_pat 8393 module_dm.f90 real_debug.exe 00000000014DA6C0 wrf_global_to_pat 8223 module_dm.f90 real_debug.exe 000000000140EF92 call_pkg_and_dist 23240 module_io.f90 real_debug.exe 0000000001407BA5 call_pkg_and_dist 22691 module_io.f90 real_debug.exe 0000000001406CB9 call_pkg_and_dist 22600 module_io.f90 real_debug.exe 00000000013FB046 wrf_readfield1 21178 module_io.f90 real_debug.exe 00000000013FA8B8 wrf_readfield 20968 module_io.f90 real_debug.exe 000000000512067A wrf_ext_read_fiel 130 wrf_ext_read_field.f90 real_debug.exe 000000000420D54C inputwrf 1643 input_wrf.f90 real_debug.exe 0000000004031DC3 module_iodomain 898 module_io_domain.f90 real_debug.exe 0000000000417676 med_sidatainput 414 real_em.f90 real_debug.exe 0000000000415289 MAIN 244 real_em.f90 real_debug.exe 0000000000413852 Unknown Unknown Unknownlibc-2.26.so 00002AAAAE25B34A libc_start_main Unknown Unknown real_debug.exe 000000000041376A Unknown Unknown Unknown
We can provide sample inputs from met_em, namelist, etc. if desired. But I will not attach them all here due to size.
— Reply to this email directly, view it on GitHub https://github.com/wrf-model/WRF/issues/1687, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIZ77DIEKV2JTTRGOGZ7IDU4O66XANCNFSM5PB7LZZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
hengxiao80
jamiebresch
wgustafson
Heng Xiao (@hengxiao80) and I have been trying to work through an issue where real.exe is crashing while trying to read URB_PARAM. This only happens when we use large domains. One thing I noticed is that URB_PARAM is not indicated to have Z-staggering in Registry.EM_COMMON. I vaguely remember from work I did many years ago in WRF-Chem that variables that use one of the constant dimensions must have that marked as Z-staggered, even if it really isn't thought of that way. Is that correct? If so, then this is likely a bug for the definition of URB_PARAM that may also require a few changes to looping parameters in the code.
The symptom we are seeing is a buffer overflow in the read routines. We actually have the urban parameterization turned off, so URB_PARAM is not in our input met_em files. So, this read attempt is superfluous. Our workaround is to just turn off the "i1" read in Registry.EM_COMMON, but this may not work for others' situations if they run into this same problem.
Note that we tried changing the staggering to "Z", but we still got the crash if we left the read turned on. So, I do not know what the actual issue is. However, I wanted to report the possible Z-staggering issue for the urban developers to be aware of.
The issue happens for the 2nd domain when processing domains set up with
The error messages reported in rsl.error.0000 are:
We can provide sample inputs from met_em, namelist, etc. if desired. But I will not attach them all here due to size.