search

oceanmodeling / ufs-weather-model

This repo is forked from ufs-weather-model, and contains the model code and external links needed to build the UFS coastal model executable and model components, including the ROMS, FVCOM, ADCIRC and SCHISM plus WaveWatch III model components.
https://github.com/oceanmodeling/ufs-coastal-app
Other
2 stars 3 forks source link

SCHISM+WW3 H. Ian testing-suite: Phase 1 #92

Open yunfangsun opened 1 month ago

yunfangsun commented 1 month ago

General plan for (2D) SCHISM+WW3 coupling:

Plan is to separate these from the "official" RT system, which we'll keep light-weight and attached to the model-level. The heavier cases here will eventually live on the app-level as a separate testing-suite.

Hurricane Ian "Testing suite" plan

(from @saeed-moghimi-noaa) Here is my suggestion for steps for simulations to be completed, check-mark and start the next step:

Phase 1

Phases 2 & 3

yunfangsun commented 1 month ago

24

yunfangsun commented 1 month ago

https://github.com/oceanmodeling/ufs-coastal/issues/24#issuecomment-1906460409

https://github.com/oceanmodeling/ufs-coastal/issues/24#issuecomment-1909495726

https://github.com/oceanmodeling/ufs-coastal/issues/24#issuecomment-1909598050

https://github.com/oceanmodeling/ufs-coastal/issues/24#issuecomment-1910675435

saeed-moghimi-noaa commented 1 month ago

@yunfangsun Please add details for the meshes being used.

yunfangsun commented 1 month ago

Hi @uturuncoglu @pvelissariou1 ,

Do you have any example cases for a ufs-coastal SCHISM alone? such as ike_shinnecock.sch in CoastalApp-testsuite.

Thank you!

uturuncoglu commented 1 month ago

@yunfangsun No, we don't have that configuration as RT under ufs-coastal.

yunfangsun commented 1 month ago

@yunfangsun Please add details for the meshes being used.

The mesh here combines the following two sets of meshes:

1) STOFS mesh with 11969815 elements and 6056968 nodes within 34kt of Hurricane Ian swath

Screenshot 2024-05-17 at 12 40 05 PM

2) Coarse mesh with 557481 nodes and 1069847 elements outside the 34kt of Hurricane Ian swath

Screenshot 2024-05-17 at 12 40 23 PM

The final mesh is with 4886065 elements and 2470094 nodes:

Screenshot 2024-05-17 at 12 48 01 PM Screenshot 2024-05-17 at 12 48 18 PM
yunfangsun commented 1 month ago

The configurations of the cases are set on Hercules in the folder /work2/noaa/nos-surge/yunfangs/hurricane_ian seven case folders are inside, as follows:

stand alone SCHISM for Ian (sch_alone) stand alone WW3 for Ian (ww3_alone) ufs-coastal WW3 alone for Ian (ufs_ww3) ufs-coastal SCHISM alone for Ian (ufs_sch) ufs-coastal DATM+SCHISM alone for Ian (ufs_atm2sch) ufs-coastal DATM+WW3 alone for Ian (ufs_atm2ww3) ufs-coastal DATM+SCHISM-WW3 for Ian (ufs_atm2sch2ww3)

janahaddad commented 1 month ago

@yunfangsun, great work and thank you for documenting. A few questions for clarity:

  1. Are you using this combined mesh for all the 7 configs you have listed? If not, can you sort them by mesh being used as well as winds?
  2. What's the common name of this combined mesh? If there is not one, we should come up with one to easily refer to it. What do you think?
  3. Can you please separate your list of configs into those which are complete & running, those which are not yet complete but in progress, and those you have not yet started?
yunfangsun commented 1 month ago

@yunfangsun, great work and thank you for documenting. A few questions for clarity:

  1. Are you using this combined mesh for all the 7 configs you have listed? If not, can you sort them by mesh being used as well as winds?

All the cases are using the same mesh. all the winds are ERA5 wind

  1. What's the common name of this combined mesh? If there is not one, we should come up with one to easily refer to it. What do you think?

Maybe atltantic_ian?

  1. Can you please separate your list of configs into those which are complete & running, those which are not yet complete but in progress, and those you have not yet started?

Ok, I will update the status of each case.

janahaddad commented 1 month ago

Ok thanks @yunfangsun, since this is the mesh that was giving you problems in #24, do you think it's worth going through this exercise first for ufs_atm2sch ufs_atm2sch2ww3 ufs_atm2ww3 ufs_sch ufs_ww3 with a known working mesh e.g. 1.8mil node hsofs mesh or even a coarser one that you can test quickly?

yunfangsun commented 1 month ago

@janahaddad

Thank you very much! It is an excellent suggestion, I think I could try the Hsofs mesh for the ufs_atm2sch ufs_atm2sch2ww3 ufs_atm2ww3 ufs_sch ufs_ww3 firstly, and could also compare the results between different mesh later.

yunfangsun commented 1 month ago

I have already set up the HSOFS mesh for Hurricane Ian for the following cases:

WW3 stand-alone: /work2/noaa/nos-surge/yunfangs/hurricane_ian/ww3_alone_hsofs UFS-coastal ATM+WW3: /work2/noaa/nos-surge/yunfangs/hurricane_ian/ufs_atm2ww3_hsofs

yunfangsun commented 1 month ago

@uturuncoglu

I am trying to use the UFS-coastal_ww3 case for Hurricane Ian with HSOFS mesh, there is no error message in the out, err and ww3_shel.nml.log, however, the job is hanging without running. and the PET00.ESMF_LogFile said that

20240521 091013.496 INFO             PET00 (wav_comp_nuopc:wavinit_ufs): mod_def timesteps file      100.00    100.00    100.00    100.00
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_z0 is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuu is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuv is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsvv is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_x is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_y is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Si_ifrac is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_u is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_v is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_t is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_tbot is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_u10m is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_v10m is not connected.
20240521 091026.158 INFO             PET00 (wav_comp_nuopc):(ModelSetRunClock) called
20240521 091026.158 INFO             PET00 (wav_comp_nuopc):(ModelSetRunClock)setting alarms for WAV
20240521 091026.159 INFO             PET00 (wav_comp_nuopc):(ModelSetRunClock) done
20240521 091026.159 INFO             PET00 ------>Advancing WAV from: 2022  9 15  0  0  0   0
20240521 091026.159 INFO             PET00 --------------------------------> to: 2022  9 15  1  0  0   0

Do you have any suggestions on how to fix it? The run folder is located at /work2/noaa/nos-surge/yunfangs/hurricane_ian/ufs_ww3_hsofs on Hercules

Thank you very much!

uturuncoglu commented 1 month ago

It seems that you are using 1000 processor for this job but in ufs.configure you are setting like following, 

WAV_petlist_bounds:             0 11

So, maybe that is the reason.

uturuncoglu commented 1 month ago

I am not sure this is for RT or not but i think these are using too much resources for replacing existing RTs with realistic cases. We need to have very small domain with realistic forcing and hurricane.

yunfangsun commented 1 month ago

Hi @uturuncoglu The HSOFS mesh will be used for applications not for RT, I will develop RT cases by using shinnecock mesh.

pvelissariou1 commented 1 month ago

@uturuncoglu I full agree. On the other hand @yunfangsun needs to create and evaluate the Ian tests cases for our testing purposes (meshes, configs, w/wo WW3, etc ...). I believe at this point is trying to use the RT system. But yes, at some point we need to use these outside the main RT system or, to generate a special rt_coastal file for these long cases. @yunfang will chime in.

uturuncoglu commented 1 month ago

Okay. That is great.

yunfangsun commented 1 month ago

It seems that you are using 1000 processor for this job but in ufs.configure you are setting like following,

WAV_petlist_bounds:             0 11

So, maybe that is the reason.

Hi @uturuncoglu ,

Thank you for the suggestion, I have corrected the ufs.configure by

WAV_petlist_bounds: 0 999

However, the PET log file is still the same,


20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_z0 is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuu is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuv is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsvv is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_x is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_y is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Si_ifrac is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_u is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_v is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_t is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_tbot is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_u10m is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_v10m is not connected.

Is there any other configuration file related to this error? Thank you very much!

pvelissariou1 commented 1 month ago

@yunfangsun Maybe just a couple storms for the test cases but using different configs for the models SCHISM, WW3 [& PAHM]. Need to have a "complete" set of working tests so ufs-coastal and models can be tested properly after each update.

yunfangsun commented 1 month ago

@pvelissariou1 I plan to use shinnecock mesh to develop for a few of RT cases, SCHISM, WW3, ATM+WW3, ATM+SCHISM, ATM+SCHISM+WW3 etc. for a couple of storms

janahaddad commented 1 month ago

@yunfangsun which storm are you planning on for the Shinnecock case? I realized yesterday we said H. Ian but that wouldn't make much sense for that domain. Sandy?

pvelissariou1 commented 1 month ago

@janahaddad , @yunfangsun I do have sandy test cases locally. Sandy is the only storm that produces winds (PaHM) over that domain but for shinnecock mesh we can use atm data other than sandy/pahm to set the test cases. For comparison/evaluation Sandy might be the storm choice. We do have the "Idealized" test cases for shinnecock as well. Need to finish some tasks for NHC and I'll push/test the cases I have locally, in a few days.

saeed-moghimi-noaa commented 1 month ago

@pvelissariou1 Do you mean Sandy on hsofs mesh?

pvelissariou1 commented 1 month ago

@saeed-moghimi-noaa Sandy on shinnecock mesh. Florence on hsofs mesh.

uturuncoglu commented 1 month ago

@pvelissariou1 As we discussed yesterday. we will replace idealized cases with realistic ones. There is no need to keep multiple case with same configuration under RT system since it is just checking the infrastructure. Maybe 1-2 lightweight test for each different configuration is fine. The more realistic and high resolution cases will go the app level testing.

yunfangsun commented 1 month ago

@yunfangsun which storm are you planning on for the Shinnecock case? I realized yesterday we said H. Ian but that wouldn't make much sense for that domain. Sandy?

@janahaddad

I think ELSA 2021 and HENRI 2021 could fit the Shinnecock mesh.

https://en.wikipedia.org/wiki/Hurricane_Henri#/media/File:Henri_2021_path.png https://en.wikipedia.org/wiki/Hurricane_Elsa#/media/File:Elsa_2021_path.png

pvelissariou1 commented 1 month ago

@uturuncoglu Ufuk, the Idealized test cases will eventually go away, for the time being we need to keep them as they have been tested w/wo WW3. What we need is multiple "lightweight" cases for Shinnecock that employ different options (physics) in SCHISM, ADCIRC, WW3, etc.. (like in ADCIRC/WW3 unit tests upstream) so we can test ufs-coastal and model components quickly. This is what I meant. Also we need to evaluate model results and check if they make sense from the scientific side of things. Anyway this is just a suggestion.

pvelissariou1 commented 1 month ago

@yunfang That is great!! Is it possible to check if there are any observations (co-ops, usgs, etc.) for these storms in the area? What is the proximity of the storms to the Shinnecock domain? I'll run PaHM later today using best tracks for these storms to see if any atm. fields are generated for the domain. I'll report back.

yunfangsun commented 1 month ago

@pvelissariou1 , I am developing the applications for Hurricane Ian in the Atlantic domain as shown at the beginning of this issue, the storms to the Shinnecock domain will not be my current tasks.

janahaddad commented 1 month ago

@yunfangsun thanks, that plan is in line with recent team discussions. For clarity I'll update this current Issue to reflect H. Ian HSOFS mesh setup for your list of configurations above, and will open a future/"to do" issue for the RT-system Shinnecock cases.

saeed-moghimi-noaa commented 1 month ago

General plan for (2D) SCHISM+WW3 coupling:

Plan is to separate these from the "official" RT system, which we'll keep light-weight and attached to the model-level. The heavier cases here will eventually live on the app-level as a separate testing-suite.

Hurricane Ian "Testing suite" plan

(from @saeed-moghimi-noaa) Here is my suggestion for steps for simulations to be completed, check marked and start the next step:

Phase 1

* [ ]   stand alone SCHISM for Ian

  * native time loop (schism.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  stand alone WW3 for Ian

  * native time loop  (multi.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal SCHISM alone for Ian

  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal WW3 alone for Ian (NEMS.exe)

  * nems config with single component
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal DATM+SCHISM alone for Ian

  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal DATM+WW3 alone for Ian

  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal DATM+SCHISM-WW3 for Ian

  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

Phases 2 & 3

* Phase 2: wave BCs from Ali A. (add issue #)

* Phase 3: wave BCs from Global run (add issue #)

@yunfangsun Suggest to do HSOFS for all the cases and then come back and do the high res ones. What you think?

pvelissariou1 commented 1 month ago

@yunfangsun , @janahaddad PAHM produces the following atmospheric fields for Shinnecock domain: Henri: From 2021-08-21 23:00:00 to 2021-08-22 12:00:00 Elsa: From 2021-07-09 04:00:00 to 2021-07-09 22:00:00 Both less than a day. We do have some data. We may pick one of the two in addition to Sandy.

saeed-moghimi-noaa commented 1 month ago

Hi @pvelissariou1

We are focusing on the mechanics of the 2D ww3-schsim for realistic cases (1st for hsofs mesh, then for the Ian subset mesh).
1) without wave bou 2) with wave bou

Feel free to take the lead towards what you are thinking if you have the bandwidth.

Thanks, -Saeed

yunfangsun commented 1 month ago

@saeed-moghimi-noaa

Yes, it is my routine, I will do the HSOFS for the 6 non-obc cases first, then use the high-resolution subset mesh.

pvelissariou1 commented 1 month ago

@saeed-moghimi-noaa ok will do, thanks. All if it is ok, let's forget Elsa/Henri for shinnecock (will keep Sandy for this) and let's focus on HSOFS cases and Ian subset mesh cases. @yunfangsun If its ok, I'll schedule a meeting with you tomorrow morning to discuss things over and what we need to do to finalize the test cases.

saeed-moghimi-noaa commented 1 month ago

Hi @pvelissariou1

On a similar note, you were working on a similar issue PAHM+ww3+adcirc on hsofs mesh for AMS2024, right? 1) without wave bou 2) with wave bou

I assume the one without wave bou was ready and validated before you move to the NHC position, right? If you need a meeting I would be more than happy to talk to you about the plan and review the results / validation for the adc-ww3 coupling.

Just want to make sure we all know what we are doing and do not conflate plans / tasks.

@janahaddad Do we have a specific issue for PAHM+ww3+adcirc? Perhaps we need to move this conversation over there.

Thanks, -Saeed

janahaddad commented 1 month ago

Hi @pvelissariou1

On a similar note, you were working on a similar issue PAHM+ww3+adcirc on hsofs mesh for AMS2024, right?

  1. without wave bou
  2. with wave bou

I assume the one without wave bou was ready and validated before you move to the NHC position, right? If you need a meeting I would be more than happy to talk to you about the plan and review the results / validation for the adc-ww3 coupling.

Just want to make sure we all know what we are doing and do not conflate plans / tasks.

@janahaddad Do we have a specific issue for PAHM+ww3+adcirc? Perhaps we need to move this conversation over there.

Thanks, -Saeed

yes, we do have an issue I moved to backlog: #73 -- I believe this was from the original AMS2024 list. We can rename this issue as needed if the interest is now to focus first on Sandy winds, but let's use that issue for this configuration "category".