Open yunfangsun opened 1 month ago
uturuncoglu
commented
1 month ago @yunfangsun It seems that there is an issue with the mesh. So, I think it is an issue with WW3 mesh. I think there is no way in WW3 side to write the mesh but CMEPs mediator could dump the mesh in VTK format. Please set dbug_flag = 30in the ufs.configure all component section (or med section). Once, you run it it might write VTK files of the mesh and then we could check them to see any issue in the mesh.
yunfangsun
commented
1 month ago Hi @uturuncoglu ,
I have used dbug_flag = 30 in the ufs.configure
# MED #
MED_model: cmeps
MED_petlist_bounds: 0 1599
MED_omp_num_threads: 1
MED_attributes::
ATM_model = datm
OCN_model = schism
WAV_model = ww3
history_n = 1
history_option = nhours
history_ymd = -999
coupling_mode = coastal
pio_typename = PNETCDF
pio_numiotasks = 32
dbug_flag = 30
::And I didn't see the vtk files in the folder /work2/noaa/nosofs/yunfangs/stmp/yunfangs/FV3_RT/rt_324728_atmschww3_06132024/coastal_ian_subset_atm2sch2ww3_intel_5_4
If there anything wrong in the ufs.configure
yunfangsun
commented
1 month ago Hi @sbanihash,
Thank you for helping me check the log files,
There are another copies of the files located at ATM+WW3: /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_3388781_atmww3_052024/coastal_ian_subset_atm2ww3_intel
ATM+SCH+WW3: /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_324728_atmschww3_06132024/coastal_ian_subset_atm2sch2ww3_intel_test4_1
uturuncoglu
commented
1 month ago @yunfangsun you could also try to set it in the code. Goto mediator/med_constants_mod.F90 and set med_constants_dbug_flag something big like 30. You need to compile the code from scratch before run.
yunfangsun
commented
1 month ago To test the forcing issue, in the ATM+SCH+WW3 case, the forcing flags are turned off
input%forcing%winds = 'F'
input%forcing%currents = 'F'The run can't go through
To test the memory issue, for the ATM+SCH+WW3, the WW3 parts are using the same number of cores (5000 cores), ATM+SCH+WW3 is still failed.
yunfangsun
commented
1 month ago The mesh quality of the subset mesh is evaluated:
There are two points are almost identical, the modification is done
felicio93
commented
1 month ago Per Yunfang's request I reran the mesh subseting/merging processes using OCSMesh latest functions.
This is the code I used:
import ocsmesh
import geopandas as gpd
path = r"PATH/subset/"
shape = gpd.read_file(path+'file34.shp')
highres = ocsmesh.Mesh.open(path+'ECG120.2dm', crs=4326)
lowres = ocsmesh.Mesh.open(path+'final_mesh.2dm', crs=4326)
highres_clip = ocsmesh.utils.clip_mesh_by_shape(highres.msh_t,
shape=shape.union_all(),
inverse=False,
fit_inside=False,
check_cross_edges=False,
adjacent_layers=0)
merged = ocsmesh.utils.merge_overlapping_meshes([lowres.msh_t,highres_clip],adjacent_layers=2)
ocsmesh.utils.cleanup_duplicates(merged)
ocsmesh.utils.put_id_tags(merged)
ocsmesh.Mesh(merged).write(path+"merged.2dm", format='2dm', overwrite=True)@yunfangsun you will probably have to interpolate the DEM back to the mesh again.
The final mesh was uploaded here The mesh can be open on SMS with any errors, which means no invalid elements were created during the merging process.
Please let me know if there is anything else I can help with. Also, let know if you encounter any problems with this mesh.
yunfangsun
commented
3 weeks ago I have manually corrected the mesh by using SMS.
The original number of nodes is 2470094, number of cells is 4886065. After the duplicated nodes, and nearly duplicated nodes, the final mesh have 2470026 nodes, and 4885987 cells (no node position is changed).
And now the worst quality mesh calculated from MATLAB is as follows:
This mesh is acceptable.
yunfangsun
commented
3 weeks ago Based on the above mesh, the new ATM+SCHISM, ATM+WW3, ATM+SCHISM+WW3 configuration is developed for the current version of UFS-Coastal.
The ATM+SCHISM is located at /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_new/coastal_ian_subset_atm2sch_intel_1, this configuration is working for the Hurricane Ian case, and tested by @mansurjisan .
The ATM+WW3 is located at /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_ww_new/coastal_ian_subset_atm2ww3_intel_1, this configuration is also working, finished the simulation.
The ATM+SCHISM+WW3 is located at /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1, however, the same error occurs.
yunfangsun
commented
3 weeks ago Since now the mesh quality is validated, a few of ATM+SCHISM+WW3 configurations have been tested:
Since this mesh has the maximum numbers of elements for all of the existing ufs-coastal applications,
The may be possible memory issue, I have used different number of cores for this case:
In the folder of /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_1, 8000 cores are used, the job dropped at the processors of 6163, 6675.
/work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_2, 10000 cores are used, the job dropped at the processors of 8866, 977.
/work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_3, 12000 cores are used, the job dropped at the processors of 4494, 9876, 9875. all of the error messages are similar:
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Mesh/src/ESMCI_Mesh_Glue.C:6233 ESMCI_meshcreatedual() Internal error: Bad condition - /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Mesh/src/ESMCI_MeshDual.C, line:950: - there was a problem with triangulation (e.g. repeated points, clockwise poly, etc.)
20240821 131839.030 ERROR PET04494 ESMCI_MeshCap.C:258 MeshCap::meshcreatedual() Internal error: Bad condition - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_Mesh.F90:3707 ESMF_MeshCreateDual() Internal error: Bad condition - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_FieldRegrid.F90:1414 getMeshWithNodesOnFieldLoc Internal error: Bad condition - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_FieldRegrid.F90:1011 ESMF_FieldRegridStoreNX Internal error: Bad condition - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 med_map_mod.F90:503 Internal error: Bad condition - Passing error in return code
20240821 131839.030 ERROR PET04494 med_map_mod.F90:179 Internal error: Bad condition - Passing error in return code
20240821 131839.030 ERROR PET04494 med.F90:1871 Internal error: Bad condition - Passing error in return code
20240821 131839.030 ERROR PET04494 MED:src/addon/NUOPC/src/NUOPC_ModelBase.F90:1631 Internal error: Bad condition - Passing error in return code
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Trace/src/ESMCI_Trace.C:1707 ESMCI:TraceEventPhaseExit() Wrong argument specified - Trace regions not properly nested exiting from region: [MED] IPDv03p7 Expected exit from: MED: (med_map_mod: RouteHandles_init)
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Trace/src/ESMCI_Trace.C:1665 ESMCI::TraceEventCompPhaseExit() Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMCI_FTable.C:832 ESMCI_FTableCallEntryPointVMHop Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMCI_FTable.C:1100 c_esmc_compwait Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_Comp.F90:1285 ESMF_CompExecute Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_GridComp.F90:1433 ESMF_GridCompInitialize Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2895 Wrong argument specified - Failed calling phase 'IPDv03p7' Initialize for modelComp 4: MED
20240821 131839.030 ERROR PET04494 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2699 Wrong argument specified - Passing error in return code
20240821 131839.030 ERROR PET04494 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2457 Wrong argument specified - Passing error in return code
20240821 131839.030 ERROR PET04494 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:492 Wrong argument specified - Passing error in return code
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Trace/src/ESMCI_Trace.C:1707 ESMCI:TraceEventPhaseExit() Wrong argument specified - Trace regions not properly nested exiting from region: [UFS Driver Grid Comp] Init 1 Expected exit from: MED: (med_map_mod: RouteHandles_init)
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Trace/src/ESMCI_Trace.C:1665 ESMCI::TraceEventCompPhaseExit() Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMCI_FTable.C:832 ESMCI_FTableCallEntryPointVMHop Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMCI_FTable.C:1100 c_esmc_compwait Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_Comp.F90:1285 ESMF_CompExecute Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_GridComp.F90:1433 ESMF_GridCompInitialize Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 UFS.F90:393 Wrong argument specified - Aborting UFS
20240821 131839.030 INFO PET04494 Finalizing ESMF with endflag==ESMF_END_ABORT
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Trace/src/ESMCI_Trace.C:1816 ESMCI::TraceEventRegionExit() Wrong argument specified - Trace regions not properly nested exiting from region: [ESMF] Expected exit from: MED: (med_map_mod: RouteHandles_init)
20240821 131839.030 ERROR PET04494 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Trace/src/ESMCI_Trace.C:1258 ESMCI::TraceClose() Wrong argument specified - Internal subroutine call returned Error
20240821 131839.030 ERROR PET04494 ESMF_Trace.F90:102 ESMF_TraceClose() Wrong argument specified - Internal subroutine call returned Error
To test the exact location of the broken run.
The subset mesh /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_9_1_9_1 and coarse schism mesh /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_coarse_mesh_new_1 are both used and for the core usages:
100 for CMEPS
20 for DATM CDEPS, 40 SCHISM, 40 WW3
The option debug_level = 2 is used, and then the vtk format of mesh on each core for schism is written out, however, it is not working for ww3.
export_2022-09-15T00:00:00_ocean_mask.40.39.vtk
export_2022-09-15T00:00:00_ocn_current_zonal.40.30.vtkAnd for ww3 Diagnostic = 1 is used, but it gives the outputs for the whole domain not on subdomains on each core:
diagnostic_WAV_InitializeIPDv01p5_enter_import_2022_09_15_00_00_00_000_Sa_u10m.nc
diagnostic_WAV_InitializeIPDv01p5_enter_import_2022_09_15_00_00_00_000_Sa_v10m.nc
diagnostic_WAV_InitializeIPDv01p5_enter_import_2022_09_15_00_00_00_000_So_u.nc
diagnostic_WAV_InitializeIPDv01p5_enter_import_2022_09_15_00_00_00_000_So_v.nc
diagnostic_WAV_InitializeIPDvXp07_enter_import_2022_09_15_00_00_00_000_Sa_u10m.nc
diagnostic_WAV_InitializeIPDvXp07_enter_import_2022_09_15_00_00_00_000_Sa_v10m.nc
diagnostic_WAV_InitializeIPDvXp07_enter_import_2022_09_15_00_00_00_000_So_u.nc
diagnostic_WAV_InitializeIPDvXp07_enter_import_2022_09_15_00_00_00_000_So_v.ncBy comparing the up two cases:
The subset mesh configuration is stopped in the cmeps
Before the following:
PET84 after med_map_RouteHandles_init
20240825 130547.125 INFO PET84 Map type bilnr_nstod, destcomp ocn, mapnorm one Sa_pslv
20240825 130547.126 INFO PET84 Map type bilnr_nstod, destcomp ocn, mapnorm one Sw_wavsuu
20240825 130547.127 INFO PET84 Map type bilnr_nstod, destcomp wav, mapnorm one Sa_u10m
20240825 130547.127 INFO PET84 Map type bilnr_nstod, destcomp wav, mapnorm one So_u
20240825 130547.136 INFO PET84 (med.F90:DataInitialize): calledThe subset mesh is broken at Infrastructure/Mesh/src/ESMCI_MeshDual.C for the ghost mesh part
yunfangsun
commented
2 weeks ago Hi @uturuncoglu ,
The configuration with mesh problem is located at/work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1
The configuration without problem is located at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_coarse_mesh_new_1
Thank you!
uturuncoglu
commented
2 weeks ago @yunfangsun I could able to create VTK files but since they are very high resolution it is hard to fine the problematic region. So, I ask one of my colleague for help. I'll update you when I have some update about it. In the mean time, I plot those vtk files using Preview (a simile section of it can be seen in the following figure) and I wonder if they are exactly same or not. From plot, it seems that meshes are not same. Maybe the difference is coming because SCHSIM is creating mesh using ESMF API and WW3 is creating by reading SCRIP grid definition (netcdf file).
yunfangsun
commented
2 weeks ago Hi @uturuncoglu ,
The SCHISM and WW3 are using exactly the same mesh:
ww3 mesh
netcdf scrip_ww3_esmf {
dimensions:
nodeCount = 4995917 ;
elementCount = 2470026 ;
maxNodePElement = 11 ;
coordDim = 2 ;
variables:
double nodeCoords(nodeCount, coordDim) ;
nodeCoords:units = "degrees" ;
int elementConn(elementCount, maxNodePElement) ;
elementConn:long_name = "Node indices that define the element connectivity" ;
elementConn:_FillValue = -1 ;
int numElementConn(elementCount) ;
numElementConn:long_name = "Number of nodes per element" ;
double centerCoords(elementCount, coordDim) ;
centerCoords:units = "degrees" ;
int elementMask(elementCount) ;
elementMask:units = "unitless" ;
// global attributes:
:gridType = "unstructured mesh" ;
:version = "0.9" ;
:inputFile = "scrip.nc" ;
:timeGenerated = "Tue Aug 20 14:31:27 2024" ;
}
SCHISM hgrid file:
EPSG:4326
4885987 2470026
1 -77.95614850 35.30211970 -17.62350390
2 -77.95471320 35.30315010 -18.46252900
3 -77.95556400 35.30068700 -17.86483240@yunfangsun It seems that is an issue with the triangulation. I need to run the case with a specific version of ESMf that will output the region that causes issue. I think I could use your directory /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1. Right?
uturuncoglu
commented
2 weeks ago @yunfangsun They seems they are same but actually not. I think creating mesh with SCRIP file might use same coordinates but could end up different number of elements. So, maybe in WW3 side, ESMf call needs to get extra argument to have exact mesh but not sure.
yunfangsun
commented
2 weeks ago Hi @uturuncoglu , Yes, the folder is /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1
yunfangsun
commented
2 weeks ago Hi @uturuncoglu ,
To produce the scrip_ww3_esmf.nc file, I firstly ran the WW3 stand-alone case and get a scrip.nc file, and then use ESMF_Scrip2Unstruct scrip.nc scrip_ww3_esmf.nc 0 ESMF to produce the scrip_ww3_esmf.nc file.
uturuncoglu
commented
2 weeks ago @yunfangsun That is great. Yes, that is the preferred way. Once I have ESMF with debug feature, I'll try to run your case and we see the issue. After that we could look at WW3 side for its mesh generation.
uturuncoglu
commented
2 weeks ago @yunfangsun BTW, I am getting permission error for following files,
cp: cannot open '/work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1/atlantic.msh' for reading: Permission denied
cp: cannot open '/work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1/.atlantic.msh.swp' for reading: Permission deniedAre those used by the case? If not it is not important.
yunfangsun
commented
2 weeks ago Hi @uturuncoglu , I have changed the permission of atlantic.msh, could you please try it again? thank you!
uturuncoglu
commented
2 weeks ago @yunfangsun Do you remember your command that you used to convert scrip file to esmf mesh file. It seems that ESMf created dual mesh for WW3. The command should be something like ESMF_Scrip2Unstruct input_SCRIP.nc output_ESMFmesh.nc 0 but if you pass 1 as a last argument rather than 0, it converts the mesh to dual mesh (basically swaps nodes with elements). So, it would be nice to check that one. Of course this is not related with the issue that we seeing in the mesh but it could explain the issue of not having identical mesh in both side.
yunfangsun
commented
2 weeks ago Hi @uturuncoglu ,
I am using ESMF_Scrip2Unstruct scrip.nc scrip_ww3_esmf.nc 0 ESMF
and it works for the coarse SCHISM mesh and HOSFS mesh
uturuncoglu
commented
2 weeks ago @yunfangsun Okay but it does not mean both side has same mesh in the coarse case. It is just running. Do you still have scrip.nc file around?
yunfangsun
commented
2 weeks ago Hi @uturuncoglu ,
Yes, the scrip.nc is located at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1
Hi @uturuncoglu @saeed-moghimi-noaa @janahaddad @pvelissariou1
For the high-resolution subset mesh, I have finished the ATM+WW3 at
/work2/noaa/nosofs/yunfangs/stmp/yunfangs/FV3_RT/rt_3388781_atmww3_052024/coastal_ian_subset_atm2ww3_inteland ATM+SCH at/work2/noaa/nosofs/yunfangs/stmp/yunfangs/FV3_RT/rt_2446480_atmsch_052024/coastal_ian_atlantic_atm2sch_intel_subsetThe results are correct.However, for the ATM+SCH+WW3 (
/work2/noaa/nosofs/yunfangs/stmp/yunfangs/FV3_RT/rt_324728_atmschww3_06132024/coastal_ian_subset_atm2sch2ww3_intel_5_4), I am using the same input files, the case can't start, and ESMF is starting to complaining the mesh (which is the same as the two cases above).The error message is as follows:
Hi @uturuncoglu , do you know if there is a way for ESMF to locate the problematic triangulation and write the log file?
Thank you!