Closed uturuncoglu closed 6 months ago
uturuncoglu
commented
7 months ago Update from Hernan: In the old working run that masking was same. It has 1 over ocean and 1E20 over land. So, probably the masking is not the issue but in any case the mask in the mediator side needs to be same with the ROMS one (1 - ocean and 0 - land) and needs to be fixed.
uturuncoglu
commented
6 months ago @hga007 Do you have any idea why mask for psi points larger than 1 (actually 2) around the coastline for IRANE case. There might be a bug in the code but not sure. Here is the plot from the history file,
This is not ESMF/NUOPC layer and the internal data structure that holds the masking seems to have problem. I also checked the grid file irene_roms_grid.nc and it seems that the grid file has no this issue. Let me know what you think?
hga007
commented
6 months ago Hi Ufuk,
In the grid, the more important is the mask at RHO-points. For the other staggered locations, ROMS recompute the internally in metrics.F to set the no-slip conditions correctly. I don’t know why are you getting that issue. Anyway, the masks at PSI-points are never used for coupling in the NUOPC-cap module. They are only used internally in ROMS in the mixing tensors for high-order stencils and for vorticity output. You should compare your plot with the mask in the history file and not the grid.
Best, H
From: Ufuk Turunçoğlu @.> Date: Thursday, February 8, 2024 at 4:37 PM To: oceanmodeling/roms @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/roms] The coastal_irene_atm2roms RT test is failing (Issue #1)
@hga007https://github.com/hga007 Do you have any idea why mask for psi points larger than 1 (actually 2) around the coastline for IRANE case. There might be a bug in the code but not sure. Here is the plot from the history file, Screenshot.2024-02-08.at.2.33.53.PM.png (view on web)https://github.com/oceanmodeling/roms/assets/1228999/8c679238-67a4-4b17-be45-a511e52e982c
This is not ESMF/NUOPC layer and the internal data structure that holds the masking seems to have problem. I also checked the grid file irene_roms_grid.nc and it seems that the grid file has no this issue. Let me know what you think?
— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/roms/issues/1#issuecomment-1934972981, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TAG7MGOUO6WOSOEXD53YSVASVAVCNFSM6AAAAABC2YHNQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZUHE3TEOJYGE. You are receiving this because you were mentioned.Message ID: @.***>
uturuncoglu
commented
6 months ago @hga007 But, I am seeing this in ROMS history output. Maybe it is calculated wrongly internally. Since I am seeing very high temperature values in the very narrow regions. See following plot, the temperature falls to -160 in the region that has also issue in mask. So, I was thinking that maybe the issue in masking is causing crash.
uturuncoglu
commented
6 months ago @hga007 Please look at following code,
In their cff2 variable is 2 and in some if statements mask is assigned to cff2. So, maybe this is expected.
hga007
commented
6 months ago Hi Ufuk,
Yes, the PSI-mask needs to be two at coast to have the no-slip boundary conditions , and ROMS kernel will take care of it. This was done in an elegant way to avoid all the IF-statements and checks that you see in metrics.F. Otherwise, the code will be very messy and will break vectorization becoming slower. This piece of code was designed by Sasha many years ago.
The high values are happening in shallow-water and may due to strong winds. We have run this test so many times over the years that I don’t understand why we are having this problem now. The only thing that we did was changing ROMS_SetGridArrays. However, it was running fine after the change to the mesh coordinates that you did and debugged.
The error indicate that the model is blowing-up because of CFL violations, may required lower timestep. But why? Are u=you using different version of the ifort compiler? Maybe you should run in another computer to see if you get the same.
Best, H
From: Ufuk Turunçoğlu @.> Date: Thursday, February 8, 2024 at 5:46 PM To: oceanmodeling/roms @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/roms] The coastal_irene_atm2roms RT test is failing (Issue #1)
@hga007https://github.com/hga007 Please look at following code,
In their cff2 variable is 2 and in some if statements mask is assigned to cff2. So, maybe this is expected.
— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/roms/issues/1#issuecomment-1935052938, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TAEKWTN6NNIORLQL6DDYSVIVDAVCNFSM6AAAAABC2YHNQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZVGA2TEOJTHA. You are receiving this because you were mentioned.Message ID: @.***>
uturuncoglu
commented
6 months ago @hga007 I synced the UFS Coastal with UFS Weather Model. So, ESMF library version is changed (8.4.2 -> 8.5.0) along with couple of other libraries. As I know, the compiler versions are same. Let me try with 8.4.2 to see what happens.
uturuncoglu
commented
6 months ago @hga007 I tested with 8.4.2 also and it is still failing. I am not sure about the source of the issue. Maybe something else is changed in the UFS side and triggers the issue. I also find an old run directory that has no issue and compare mediator history files but could not find any difference. Maybe I could try to downgrade the ROMS to the same version that we used in that time to see what happens. I could also compare the namelist files. Anyway, I'll update you about it.
uturuncoglu
commented
6 months ago @hga007 I check different things but I could not find anything obvious. So, at this point I have no idea about crash. All forcing fields are very similar to the working case.
IRENE/Coupling/roms_data with ROMS data component by compiling UFS modules and it is running without any issue.At this point I am just thinking couple of option:
UFS might compile the code with some special flags that triggers issue. ROMS test build is using -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit compiler flags but UFS has more option -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -nowarn -sox -align array64byte -qno-opt-dynamic-align -real-size 64 -fp-model precise -ip -O3 -traceback -check uninit -module modules. So, maybe one of those extra options are triggering error.
We could try to create ESMF mesh file for era5_IRENE.nc again but in this case we could add mask information in it. At this point, era5_IRENE.nc file does not have mask information. Is it possible to create same file with mask in it? If I could have mask information, I could add it to ESMF mesh file and when we are doing interpolation I could just use ERA5 data over ocean and extrapolate to the coast. I am not sure but maybe it will work.
Another possibility is to test this configuration without ROMS internal bulk flux algorithm? We could provide heat and momentum fluxes directly from ERA5 rather than using ROMS internal bulk flux algorithm. I think this would be better approach. I think this is also valid for the fully coupled configuration to use same flux components in both atmosphere and ocean.
Anyway, let me know what you think?
uturuncoglu
commented
6 months ago @hga007 I tested the code with exact same compiler options but it is still failing. I am not sure what we need to do at this point. How easy to add mask information to era5_IRENE.nc? Maybe I could try it.
hga007
commented
6 months ago @uturuncoglu: It is not possible because the ERA dataset is in a different grid than ROMS. It has both water and land data. We could make another copy of those files and use editmask.m to add a land mask at the ERA resolution. It will take me a while to do so. This may be your problem. I had to modify the ESMF regridding when testing the WRF-ROMS coupling. If I have a chance this weekend, I can look at it.
unmap=ESMF_UNMAPPEDACTION_IGNORE
regridMethod=ESMF_REGRIDMETHOD_BILINEAR
!
srcTerm=0
!
CALL ESMF_FieldRegridStore (srcField=srcField, &
& dstField=dstField, &
& srcMaskValues=(/srcMaskVal/), &
& dstMaskValues=(/dstMaskVal/), &
& unmappedaction=unmap, &
& routeHandle=routeHandle, &
& regridmethod=regridMethod, &
& ignoreDegenerate=.TRUE., &
& srcTermProcessing=srcTerm, &
& rc=rc)
IF (ESMF_LogFoundError(rcToCheck=rc, &
& msg=ESMF_LOGERR_PASSTHRU, &
& line=__LINE__, &
& file=MyFile)) THEN
RETURN
END IF@hga007 I wonder if the ERA data set it is in original resolution. If so, we could download the land-sea mask from ERA5 and subset like we subset the ERA5 file. That will solve my problem but I have no idea how this data set is created. Is it in its original resolution. Since CDEPS is generic I could also download the ERA5 dataset from Copernicus and use it. Anyway, let me know the best way to go.
Okay, it seems that you are getting ERA5 data only over ocean and map it to the ROMS grid with extrapolation. Right? I so, I was also suspicious about contamination of the land data over ocean since we have issue in the very narrow coastline. We could do the same with CDEPS but I need to have ERA5 masking with the same size of the dataset.
uturuncoglu
commented
6 months ago @hga007 I found the issue. So, I think there is no need to update input files. With recent change in the UFS side, they decided to change the standard names started with mean to inst since actually those fields are not mean but instantaneous. The transition is not done for all the variables but just set of it at this point. there will be another PR in the UFS side in the near future. So, because of it couple of flux fields were not connected and it was creating an issue. So, to adjust ROMS to this change in the UFS side we need to change mean to inst in the roms_cmeps_era5.xml file and also the other xml files used by cmeps and cdeps in the roms_test repository like following,
227c225
< - standard_name: inst_down_lw_flx
---
> - standard_name: mean_down_lw_flx
257c255
< - standard_name: inst_net_sw_flx
---
> - standard_name: mean_net_sw_flx
332c330
< - standard_name: inst_prec_rate
---
> - standard_name: mean_prec_rateI'll change the file in my side. If you want I could create a PR to testing repo. It was a minor thing but it take time to figure out. We are fine now.
hga007
commented
6 months ago @uturuncoglu: Congratulations! That was tricky to find. I assume there is a need to report the export/import states somewhere in the logging so this problem can be quickly identified in the future. I go to the extreme of reporting the native ROMS coupling configurations in one of the log files.
uturuncoglu
commented
6 months ago @hga007 Thanks! The issue was seen in the PET log files. So, it was obvious but I missed it. Anyway, the RT is passing now again. I created PR in roms-test repository - https://github.com/myroms/roms_test/pull/20 - If you don't mind please review and merge it. I am closing this issue but we have still issue with GNU under UFS Coastal. I'll look at it too when I have time.
The existing data atmosphere coupled RT is failing with following error after UFS Coastal is sync with the authoritative repository,
This could be related with recent changes in CMEPS or CDEPS. there might be issue also with the ROMS masking in the CMEPS side since mediator history files show very large masking value over land region which might contribute interpolation issue.