WW3 Integration

[uturuncoglu]

@pvelissariou1 I am opening this issue for WW3 integration and testing. Since WW3 is integral part of the ufs-weather-model, I'll try to test existing configurations under CoastalApp-testsuite in here to see the issues that we might face.

[uturuncoglu]

@pvelissariou1 I tried to run ike_shinnecock.atm2ww3 with the WW3 that is found in ufs-coastal and I am getting following error from WW3 cap,

11:fv3_coastal.exe    0000000001E11790  wav_shel_inp_mp_r         774  wav_shel_inp.F90
11:fv3_coastal.exe    0000000001B29726  wav_comp_nuopc_mp        1598  wav_comp_nuopc.F90

The error is coming from following line in WW3,

      call print_logmsg(740+IAPROC, ' fnmpre'//trim(fnmpre), w3_debuginit_flag)
      open (newunit=ndsi,file=trim(fnmpre)//'ww3_shel.inp',status='old',iostat=ierr)
      rewind (ndsi)

So, it seems that ww3_shel.inp has some issue. Maybe it is because we are using newer version of the model. Do you have any idea?

[uturuncoglu]

I think this configuration does not have ww3_shel.inp. Do we need to create one?

[pvelissariou1]

@Ufuk Turuncoglu @.***> We don't use ww3_shel.inp in the WW3 included in CoastalApp. Need to check on this.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jun 20, 2023 at 3:24 PM Ufuk Turunçoğlu @.***> wrote:

I think this configuration does not have ww3_shel.inp. Do we need to create one?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1599447557, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP5XHOJXOL3SQFF42HLXMIBH5ANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Thanks. Let me try with other cap. Maybe that will work.

[uturuncoglu]

@pvelissariou1 Okay. It am able to build with old cap which seems CoastallApp was using. Anyway, I need to create input files with ww3_grid and ww3_bounc. I think those are not part of the WW3. Could you give me more information about it? Thanks.

[pvelissariou1]

@Ufuk Turuncoglu @.**> Look at the run/setup.job scripts how to create these files. When WW3 is compiled in CoastalApp, among other things it creates the required ww3 executables to create these files.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jun 20, 2023 at 5:27 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. It could able to build with old cap which seems CoastallApp was using. Anyway, I need to create input files with ww3_grid and ww3_bounc. I think those are not part of the WW3. Could you give me more information about it? Thanks.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1599655803, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP54WJBL4V55LH5VMYLXMIPWFANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 The existing WW3 is not building those tools like ww3_grid. So, let me check the CostallApp to see how those tools are building and maybe I could make some change in the WW3 CMake build to make it work.

[uturuncoglu]

It seems that WW3 is not building tools when it is build for coupling. I am not sure why but I could fix it.

[pvelissariou1]

@Ufuk Turuncoglu @.**> Ufuk, please look at the WW3/model/esmf/Makefile in CoastalApp. I have modified the file to build the ww3 binaries.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jun 21, 2023 at 11:06 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 The existing WW3 is not building those tools like ww3_grid. So, let me check the CostallApp to see how those tools are building and maybe I could make some change in the WW3 CMake build to make it work.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1601126184, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP54D436ZTRVQ55T3WTXMMLZFANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I opened an issue in WW3 repo. https://github.com/NOAA-EMC/WW3/issues/1025 Let's wait for their response and I could make a very simple PR to WW3 to fix it.

[pvelissariou1]

@Ufuk Turuncoglu @.**> Ufuk let's wait to see their response, but to build the ww3_ executables is not part of the NUOPC/ESMF build but rather is the user's responsibility to build those externally. I had this discussion in the past thus I decided to build them within the CoastalApp/WW3/model/esmf code for the user's convenience. Anyway, let's see if they are willing to add a rule in the make/CMake files to build those.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jun 21, 2023 at 11:17 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I opened an issue in WW3 repo. NOAA-EMC/WW3#1025 https://github.com/NOAA-EMC/WW3/issues/1025 Let's wait for their response and I could make a very simple PR to WW3 to fix it.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1601141812, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPY2BYSIMSL65RINAOLXMMM77ANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I am very close to make it work. The only issue again ATM forcing for CDEPS. Which forcing do we need to use for this case? I could again force with ERA5 but I think it is low resolution for this case.

[pvelissariou1]

@Ufuk Turuncoglu @.**> The test cases ike_shinnecock.atm2ww3 and ike_shinnecock.ww3 are self-contained. The atm forcing is wind_atm_fin_ch_time_vec.nc http://wind_atm_fin_ch_time_vec.nc* in the run directory. For the hsofs/ww3 cases you need to download the data required to run the test cases (please read https://github.com/noaa-ocs-modeling/CoastalApp-testsuite/blob/main/README.md#downloading-required-data-optional )

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jun 21, 2023 at 11:57 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I am very close to make it work. The only issue again ATM forcing for CDEPS. Which forcing do we need to use for this case?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1601227805, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7RUKVYUKZE55GFPCTXMMRXRANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Thanks. Yes. I saw that one but it is in vector format and designed to work with ATMMESH. Since we are using CDEPS at this point, I need to have same dataset but in its original form (on lat-on grid). I could not use this 1d data under CDEPS.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Yes, I forgot about this. Can you use the same atm. forcing you use for ADCIRC/CDEPS ?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jun 21, 2023 at 12:14 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Thanks. Yes. I saw that one but it is in vector format and designed to work with ATMMESH. Since we are using CDEPS at this point, I need to have same dataset but in its original form (on lat-on grid). I could not use this 1d data under CDEPS.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1601269559, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZP3AQX3DV53M37XS3XMMTXVANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I am using Wind_Pressure_HWRF_Florence_ExtendedSmooth.nc for it but this case cover different dates (I am not sure why and I think all the tests need to use same dataset - by this way you could compare their results). The data used by ADCRIC covers only 2018-09-09 06:00:00 to 2018-09-18 17:00:00 but the WW3 configuration starts from 2018-09-04 12:00. I think that we need to make consistent all the configurations in terms of used data and time of simulation. So, we could use the same ATM input for all the similar configurations.

[uturuncoglu]

@pvelissariou1 I could run ike_shinnecock.atm2ww3 test case with ERA5 input. We could change the input if you want or we could keep like it is. Let me know what do you want. My run directory is in /glade/scratch/turuncu/COASTAL_APP/ike_shinnecock.atm2ww3 on Cheyenne. It might be nice to check the output but I am not sure which file do I need to look with which tool? Anyway, if you give me more information about it, I could check the output. If everything is fine and then we could add this as a RT.

[uturuncoglu]

@pvelissariou1 Maybe I have no output but model seems working. If you don't mind could you check my run directory. If this run is okay, maybe we could try to run fvcom with ww3 case but I need to first solve the build issue.

[pvelissariou1]

@Ufuk Turuncoglu @.***> I have a couple of meetings early in the morning, I will check immediately after. I replied regarding the fproj build in FVCOM. You need to first build the PROJ4 library and then build fproj. Details can be found in the function compileFVCOMLibs in CoastalApp/scripts/functions_build.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jun 21, 2023 at 11:54 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Maybe I have no output but model seems working. If you don't mind could you check my run directory. If this run is okay, maybe we could try to run fvcom with ww3 case but I need to first solve the build issue.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1602007978, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP5GAQWCXDX36GMUHI3XMPFX3ANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa I could able to run WW3 ike_shinnecock.atm2ww3 case with CMEPS mediator after I got some key information from Jessica from NOAA and fixing CMEPS. I did not check the results but I'll finalize the fork update the configuration files and define as a RT under ufs-coastal. so, you could run and test. This will also give us a capability to extend exiting RTs with three way coupling by including WW3.

[saeed-moghimi-noaa]

This is a great news! Thanks @uturuncoglu

FYI @yunfangsun

[pvelissariou1]

Thank you Ufuk. Will try all WW3 cases. Hopefully FVCOM 2D case as well.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Aug 1, 2023 at 12:51 PM Saeed Moghimi @.***> wrote:

This is a great news! Thanks @uturuncoglu https://github.com/uturuncoglu

FYI @yunfangsun https://github.com/yunfangsun

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1660814357, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4Q26WYIGCPD2BT363XTE6XXANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[yunfangsun]

@uturuncoglu Hi Ufuk,

I can't access to Cheyenne, could I know if the codes have been updated on Github?

Thank you!

Yunfang

[uturuncoglu]

@yunfangsun I did not push the modifications to repository yet but I'll do it soon. Cheyenne is down this week. So, at this point I am continue to development on Orion.

[uturuncoglu]

@pvelissariou1 Did you modify the configuration of WW3 related tests? Still I could not get output from ike_shinnecock.atm2ww3. If you could double check in your side, that would be great. Thanks.

[uturuncoglu]

@pvelissariou1 @yunfangsun I added coastal_ike_shinnecock_atm2ww3 test case as RT and push all the changes to the ufs-coastal. The ATMMESH+WW3 (the version used by bfs-weather model) coupling working with CMEPS. You just need to create ESMF mesh file as extra input (the ww3_grid with this version of WW3 creates SCRIP grid definition file and this can be used to create ESMF mesh file). We still need to modify WW3 configuration since it is not write any output to check. Anyway, please try and let me know if you have any issue. I also updated my input directory (/work/noaa/nems/tufuk/RT - just reminder, you need to update DISKNM under rt.sh with this for Orion) on Orion to reflect changes related to WW3. Anyway, let me know if you need any other information. I am also planning to bring other WW3 related tests one-by-one starting from the standalone one.

[yunfangsun]

@uturuncoglu Thank you for the good news, I will try.

[uturuncoglu]

@pvelissariou1 I think following lines are missing in ww3 standalone test ike_shinnecock.ww3 setup scripts,

cp -fp mod_def.ww3 mod_def.inlet
cp -fp mod_def.ww3 mod_def.points

without those I am getting error from run. Any idea?

[pvelissariou1]

@Ufuk Turuncoglu @.**> I checked ike_shinnecock.ww3 in CoastalApp-testsuite and all modelsetup.job scripts contain the lines you mentioned. I guess you are using a recent version of the testsuite.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Aug 4, 2023 at 3:39 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think following lines are missing in ww3 standalone test ike_shinnecock.ww3 setup scripts,

cp -fp mod_def.ww3 mod_def.inlet cp -fp mod_def.ww3 mod_def.points

without those I am getting error from run. Any idea?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1666157791, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZRZKWMNHQITEEAAHDXTVMWLANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Okay. I see now. I am not sure why I could not see it before. Anyway, I add also coastal_ike_shinnecock_ww3 test to RTs. So, you could also try to run standalone test. That requires minor modification in the WW3 cap (NOAA-EMC one) and it is in my fork https://github.com/uturuncoglu/WW3 (feature/coastal_app branch). I think we might need to discuss about maintaining those forks. I'll write to the model level issue about it.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Great, let's talk on Monday. I will keep you updated as I finish the tasks for WW3 as we discussed.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Aug 4, 2023 at 4:02 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. I see now. I am not sure why I could not see it before. Anyway, I add also coastal_ike_shinnecock_ww3 test to RTs. So, you could also try to run standalone test. That requires minor modification in the WW3 cap (NOAA-EMC one) and it is in my fork https://github.com/uturuncoglu/WW3 (feature/coastal_app branch). I think we might need to discuss about maintaining those forks. I'll write to the model level issue about it.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/7#issuecomment-1666177018, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP665UHI77AYGPGQUETXTVPOLANCNFSM6AAAAAAZNTXHGE . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I run standalone ww3 case (coastal_ike_shinnecock_ww3) but I still don't know it write the output correctly or not. I have only following files: out_grd.inlet, restart001.inlet and fort.53. Are these output files? IF so, how can I create visualization from them. If you have a tool to create simple visualization, my run directory is /work/noaa/stmp/tufuk/stmp/tufuk/FV3_RT/rt_415550/coastal_ike_shinnecock_ww3_intel.

[uturuncoglu]

@pvelissariou1 I think ww3_outf can be used to create plots but it requires ww3_outf.inp and if I provide one I am getting following error. Do you have those files to create plot?

                  *** WAVEWATCH III Field output postp. ***   
               ===============================================

  Comment character is '$'

  Grid name : Inlet                         

 *** WAVEWATCH III ERROR IN W3IOGO : 
     ERROR IN OPENING FILE
     IOSTAT =   29

[uturuncoglu]

@pvelissariou1 @yunfangsun Since I have no experience with processing WW3 output, Is it possible to provide me ww3_outf.inp file that could be used to convert ww3 output to netcdf. It would be nice to have one handy and I could use for different configurations.

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa Okay. I made it. I could able to run the case with PDLIB enabled (used for decomposition) and using single domain configuration file. I could able to activate the gridded_netcdfout = true switch in nems.configure since multigrid = false in this case. I'll check the results and also update the WW3 RTs under UFS Coastal to use this new setup. Having output in netcdf format will make post processing easier.

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa Okay. WW3 output looks fine. I checked with following simple NCL script. I'll update WW3 regression tests to use this new configuration.

begin
  ;--- arguments ---
  tindx = 5
  var = "HS"
  size = 0.8

  ;--- get list of files ---
  lst = systemfunc("ls -al *.out_grd.ww3.nc | awk '{print $9}'")
  print(""+lst)

  ;--- open files ---
  ncs = addfiles(lst, "r")

  ;--- read variables ---
  data = ncs[tindx]->$var$(0,0,:)
  lon = ncs[tindx]->lon
  lat = ncs[tindx]->lat
  nconn = ncs[tindx]->nconn

  data@lon1d = lon(0,:)
  data@lat1d = lat(0,:)

  ;--- plot it ---
  wks_type = "pdf"
  wks = gsn_open_wks(wks_type, "plot_ww3_"+var+"_"+sprinti("%2.2d", tindx))

  res = True
  res@gsnDraw = False
  res@gsnFrame = False
  res@gsnAddCyclic = False
  res@cnFillOn = True
  res@cnLinesOn = False
  res@cnLineLabelsOn = False
  res@gsnLeftString = ""
  res@gsnRightString = ""
  res@vpWidthF = size
  res@vpHeightF = size
  res@cnInfoLabelOn = False
  res@lbLabelBarOn = False
  res@cnFillOn = True
  ;res@cnFillMode = "CellFill"
  res@sfElementNodes = nconn
  res@sfFirstNodeIndex = 1

  res@mpDataBaseVersion = "MediumRes"

  res@mpLimitMode = "Corners"
  res@mpLeftCornerLatF = min(lat)
  res@mpLeftCornerLonF = min(lon)
  res@mpRightCornerLatF = max(lat)
  res@mpRightCornerLonF = max(lon)

  plot = gsn_csm_contour_map(wks, data, res) 
  draw(plot)

  frame(wks)
end

[uturuncoglu]

@pvelissariou1 If you don't mind could you double check the ike_shinnecock.ww3 test case under CoastalApp. It seems that all input wind is zero even if netcdf file has some data (I think it is artificially generated by PHaM). So, there might be an issue related with reading netcdf file in ww3_prnc step. It might be also related with my changes but atm2ww3 seems fine.

[pvelissariou1]

@uturuncoglu PaHM is not used in this test case. I checked the input file wind_atm_fin_ch_time_vec_STR.nc in ike_shinnecock.ww3/run/ and the winds look fine. Didn't have issues before with this test case.

[uturuncoglu]

@pvelissariou1 yes netcdf file is fine but did you check the wind in the ww3 output after you run? In my case all is zero.

[pvelissariou1]

@uturuncoglu I am running the case on orion. How did you check wind.inlet for the wind fields?

[uturuncoglu]

@pvelissariou1 Since it is hard to look that binary file, I change the configuration of WW3 to write output in netcdf and I could easily look at the content. I have no experience with WW3 but I think there is a post processing tool build with WW3 to convert those files to netcdf. Who created this configuration?

[uturuncoglu]

@pvelissariou1 I would like to see if this is the issue with the original configuration that writes the binary. Again, I used new configuration with atm2ww3 and the results look fine.

[pvelissariou1]

@uturuncoglu This configuration was created by Ali Abdolali at EMC at that time.

[uturuncoglu]

@pvelissariou1 Can we reach him to get some help?

[pvelissariou1]

@uturuncoglu Can you share the new configuration?

[pvelissariou1]

@uturuncoglu He is not working for NOAA anymore. He is at ERDC. Let's see if we can resolve the issue ourselves.

[uturuncoglu]

@pvelissariou1 Okay. I tried to convert output the netcdf using ww3_outf tool but it was giving error for me as I mentioned before in this thread. The new configuration is in /work/noaa/stmp/tufuk/stmp/tufuk/FV3_RT/rt_231965/coastal_ike_shinnecock_ww3_intel on Orion. Let me know if you have permission issue.

[pvelissariou1]

@uturuncoglu Thanks, I'll check it out. I'll check ww3_outf as well. Need to attend a couple of meetings now. I'll keep tou posted.

[uturuncoglu]

@pvelissariou1 Do you have any update about this issue? Thanks.

[pvelissariou1]

@uturuncoglu Not yet. Trying to finalize the test cases for UFS-Coastal and at the same time I will check on this issue as well.