search

noaa-ocs-modeling / CoastalApp

CoastalApp will be deprecated. Please switch to https://github.com/oceanmodeling/ufs-coastal . CoastalApp is a NUOPC application implemented following UFS best practices to couple coastal ocean models and other domains (Sea Ice, Atmosphere, Wave, Inland Hydrology, ...)
https://github.com/oceanmodeling/ufs-coastal
Creative Commons Zero v1.0 Universal
13 stars 31 forks source link

Test ATMESH+WW3 on multi nodes #134

Closed saeed-moghimi-noaa closed 1 year ago

saeed-moghimi-noaa commented 1 year ago

@GM-Ling @pvelissariou1 @aliabdolali @FVCOM-GitHub

Guoming Ling 9:36 AM 1) follow up with UND (i.e. Guoming) and check if they can run coupled model with PDLIB activated in WW3? If the answer is yes, we should check their setup and version of WW3 in their test. Yes, I always run with PDLIB activated, but I don't have much knowledge about the PDLIB except the introductions in the user manual(https://github.com/NOAA-EMC/WW3/wiki/Manual). My setups and results for the Shinnecock test case were uploaded here: https://github.com/GM-Ling/ATMESH-ADCIRC-WW3-TestSuite.

works for Guoming

Guoming Ling 9:40 AM git log commit 216b06032ee47a10ebb7762817752843fc945fec (HEAD -> feature/pahm, origin/feature/pahm) Author: zacharyburnettNOAA zachary.burnett@noaa.gov Date: Mon Aug 9 12:36:59 2021 +0000

added aswip

commit 0154723783823b7786f3930be3787dc9757f5d53 Author: pvelissariou1 panagiotis.velissariou@noaa.gov Date: Fri Aug 6 00:50:15 2021 -0500

modified the pahm_from_file blocks

commit 21faa6238ca8d274c1d1112e1a5f530fe6e1c5a0 Author: pvelissariou1 panagiotis.velissariou@noaa.gov

pvelissariou1 commented 1 year ago

I believe that I have fixed the ATMESH issue. I'll check out what is going on.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Oct 24, 2022 at 3:41 PM Saeed Moghimi @.***> wrote:

Thanks @pvelissariou1 https://github.com/pvelissariou1

Here is from the email.

Hi Saeed,

The develop_branch has been updated as we discussed. WW3 points to the scalability branch of Andre (forked) with my changes Build scripts of CoastalApp have also been updated to accommodate the hpc-stack system.

The modulefiles folder contains the envmodules_intel.hera, envmodules_intel.hera.sys and envmodules_intel.hera.hpc-stack modulefiles. The "hpc-stack" loads the hpc-stack modules, and the "sys" loads the system's available modules. The envmodules_intel.hera file is a copy of the envmodules_intel.hera.sys file. To switch to the "hpc-stack" just copy envmodules_intel.hera.hpc-stack to envmodules_intel.hera

WW3 compiles as expected and generates the ww3_multi as well (in ALLBIN_INSTALL)

I'll continue with the tests and I'll update the issue.

Takis

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/134#issuecomment-1289589737, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6S55YY5M4BA3N4RCLWE3X5NANCNFSM6AAAAAAQLO6VTI . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

ATMESH wind fields have been renamed to follow the standards. ATMESH also updated to include the UND (Dam/Guoming) temporary modifications regarding the initial time stepping where the wind fields were set to zero. This configuration has been tested with Andre for both WW3 and ADCIRC.

pvelissariou1 commented 1 year ago

The ATMESH modifications were only tested with ADCIRC and WW3.

saeed-moghimi-noaa commented 1 year ago

Takis Email to Saeed

<html><body>
<!--StartFragment-->

CoastalApp + NUOPC wind fields + ADCIRC fixes1 message
--

<!--EndFragment-->
</body>
</html>CoastalApp + NUOPC wind fields + ADCIRC fixes
1 message
Panagiotis Velissariou - NOAA Affiliate <panagiotis.velissariou@noaa.gov>   Tue, Oct 25, 2022 at 2:13 AM
To: Saeed Moghimi - NOAA Affiliate <saeed.moghimi@noaa.gov>
Hi Saeed,

The CoastalApp with the updated wind field names in the NUOPC caps for
ATMESH, PAHM and ADCIRC is located in:
/scratch2/COASTAL/coastal/noscrub/shared/Takis/CoastalApp

The ATMESH repo already has the changes for the wind standard names (as described in the issue), but for ADCIRC and PAHM I did it manually (for the time being).

ADCIRC didn't compile out of the box because of a missing flag in ADCIRC/work/[cmplrflags.mk](http://cmplrflags.mk/)
for the intel compiler.

If you want to compile from scratch, please get the archive:
/scratch2/COASTAL/coastal/noscrub/shared/Takis/CoastalApp-modified.tar.bz2

I have compiled CoastalApp using the system environment modules and the hpc-stack system:
system compilation contains: ATMESH, PAHM, ADCIRC, WW3 and WW3DATA
hpc-stack compilation contains: ATMESH, WW3 and WW3DATA (using ParMETIS from Ali's location)

Takis
pvelissariou1 commented 1 year ago

The wind field names were never fixed as Andre was testing WW3 and he was doing the modifications on ATMESH by himself. What has been fixed and pushed upstream to ADCIRC is the name of the radiation stresses.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Oct 25, 2022 at 3:37 AM Saeed Moghimi @.***> wrote:

Takis Email to Saeed

CoastalApp + NUOPC wind fields + ADCIRC fixes1 message -- CoastalApp + NUOPC wind fields + ADCIRC fixes 1 message Panagiotis Velissariou - NOAA Affiliate ***@***.***> Tue, Oct 25, 2022 at 2:13 AM To: Saeed Moghimi - NOAA Affiliate ***@***.***> Hi Saeed, The CoastalApp with the updated wind field names in the NUOPC caps for ATMESH, PAHM and ADCIRC is located in: /scratch2/COASTAL/coastal/noscrub/shared/Takis/CoastalApp The ATMESH repo already has the changes for the wind standard names (as described in the issue), but for ADCIRC and PAHM I did it manually (for the time being). ADCIRC didn't compile out of the box because of a missing flag in ADCIRC/work/[cmplrflags.mk](http://cmplrflags.mk/) for the intel compiler. If you want to compile from scratch, please get the archive: /scratch2/COASTAL/coastal/noscrub/shared/Takis/CoastalApp-modified.tar.bz2 I have compiled CoastalApp using the system environment modules and the hpc-stack system: system compilation contains: ATMESH, PAHM, ADCIRC, WW3 and WW3DATA hpc-stack compilation contains: ATMESH, WW3 and WW3DATA (using ParMETIS from Ali's location) Takis — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned.Message ID: ***@***.***>
saeed-moghimi-noaa commented 1 year ago

@Panagiotis Velissariou - NOAA Affiliate @.***>

I think in order to have these model components working, it is required to have a unified naming for exchange fields. I was surprised because I have recollections of this becoming unified/ resolved several times now by Andre, Guoming and yourself to the best of my memory, otherwise they could not have done the fully coupled test cases, right? This includes all exchange variables including wav, atm and ocn related fields across the application. Please correct me if I am wrong.

It also puzzled me as I looked into pam-adc-ww3 codes and noticed the vars are not consistent between ww3 and pahm however it seems like the runs went through. Do you have any explanation for this!? Please see cross check, for instance here:

/scratch2/COASTAL/coastal/save/Andre.VanderWesthuysen/CoastalApp-06082022 or /scratch2/COASTAL/coastal/save/Andre.VanderWesthuysen/CoastalApp

Thanks, -Saeed

Saeed Moghimi, PhD NOS Storm Surge Modeling https://nauticalcharts.noaa.gov/learn/storm-surge-modeling.html Team Lead Coastal Marine Modeling Branch, Coast Survey Development Laboratory, Office of Coast Survey at NOAA National Ocean Service. Address: 1315 East West Hwy, Room 6607, Silver Spring, Maryland 20910

Phone: ‪(302) 827-3986

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

On Tue, Oct 25, 2022 at 8:50 AM Panagiotis Velissariou < @.***> wrote:

The wind field names were never fixed as Andre was testing WW3 and he was doing the modifications on ATMESH by himself. What has been fixed and pushed upstream to ADCIRC is the name of the radiation stresses.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Oct 25, 2022 at 3:37 AM Saeed Moghimi @.***> wrote:

Takis Email to Saeed

CoastalApp + NUOPC wind fields + ADCIRC fixes1 message -- CoastalApp + NUOPC wind fields + ADCIRC fixes 1 message Panagiotis Velissariou - NOAA Affiliate ***@***.***> Tue, Oct 25, 2022 at 2:13 AM To: Saeed Moghimi - NOAA Affiliate ***@***.***> Hi Saeed, The CoastalApp with the updated wind field names in the NUOPC caps for ATMESH, PAHM and ADCIRC is located in: /scratch2/COASTAL/coastal/noscrub/shared/Takis/CoastalApp The ATMESH repo already has the changes for the wind standard names (as described in the issue), but for ADCIRC and PAHM I did it manually (for the time being). ADCIRC didn't compile out of the box because of a missing flag in ADCIRC/work/[cmplrflags.mk](http://cmplrflags.mk/) for the intel compiler. If you want to compile from scratch, please get the archive: /scratch2/COASTAL/coastal/noscrub/shared/Takis/CoastalApp-modified.tar.bz2 I have compiled CoastalApp using the system environment modules and the hpc-stack system: system compilation contains: ATMESH, PAHM, ADCIRC, WW3 and WW3DATA hpc-stack compilation contains: ATMESH, WW3 and WW3DATA (using ParMETIS from Ali's location) Takis — Reply to this email directly, view it on GitHub < https://github.com/noaa-ocs-modeling/CoastalApp/issues/134#issuecomment-1290193633 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/APC7TP4SD76PFHU6B5QB6W3WE6L53ANCNFSM6AAAAAAQLO6VTI . You are receiving this because you were mentioned.Message ID: ***@***.***>

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/134#issuecomment-1290504081, or unsubscribe https://github.com/notifications/unsubscribe-auth/APZULD7C5WQV5PXPZ3GSAJDWE7JR3ANCNFSM6AAAAAAQLO6VTI . You are receiving this because you were mentioned.Message ID: @.***>

saeed-moghimi-noaa commented 1 year ago

@pvelissariou1 @GM-Ling

After unifying names I got this error from ATMESH. This is the version modified/updated by @GM-Ling . The issue here is that for atm-wav case we do not need pmsl and therefore that field is not connected however later it is still checking for the field. I need to compare the older atmesh with the modified one I guess this might be introduced by UND as they always had pmsl connected. Just a wild guess !!!????

20221025 103434.718 INFO             PET00  (atmesh_mod:init_atmesh_nc) --- init atmesh netcdf file  ---
20221025 103434.718 INFO             PET00  (ATMESH:SetServices) --- Read atmesh info from file ---
20221025 103434.718 INFO             PET00  (ATMESH:SetServices) --- adc SetServices completed ---
20221025 103434.732 INFO             PET00  (ATMESH:ATMESH_FieldsSetup) --- Passed---
20221025 103434.732 INFO             PET00  (ATMESH:AdvertiseFields)fldsFrATM(num)%stdname  air_pressure_at_sea_level
20221025 103434.732 INFO             PET00  (ATMESH:AdvertiseFields)fldsFrATM(num)%stdname  inst_zonal_wind_height10m
20221025 103434.732 INFO             PET00  (ATMESH:AdvertiseFields)fldsFrATM(num)%stdname  inst_merid_wind_height10m
20221025 103434.732 INFO             PET00 Advertise: air_pressure_at_sea_level
20221025 103434.764 INFO             PET00 Advertise: inst_zonal_wind_height10m
20221025 103434.765 INFO             PET00 Advertise: inst_merid_wind_height10m
20221025 103434.765 INFO             PET00  (ATMESH:AdvertiseFields) --- initialization phase 1 completed ---
20221025 103434.872 INFO             PET00 Driver is in ModifyCplLists()
20221025 103434.872 INFO             PET00  Found            5  Connectors. Modifying CplList Attribute....
20221025 103434.939 INFO             PET00 atmesh.F90:508  (ATMESH:ATMESH_RealizeFields) --- OUT---
20221025 103435.015 INFO             PET00 atmesh.F90:490 (ATMESH:ATMESH_RealizeFields)ATMESH export Field air_pressure_at_sea_level                                        is not connected.
20221025 103435.016 INFO             PET00 atmesh.F90:481 (ATMESH:ATMESH_RealizeFields)ATMESH export Field inst_zonal_wind_height10m                                        is connected.
20221025 103435.016 INFO             PET00 atmesh.F90:481 (ATMESH:ATMESH_RealizeFields)ATMESH export Field inst_merid_wind_height10m                                        is connected.
20221025 103435.016 INFO             PET00 atmesh.F90:508  (ATMESH:ATMESH_RealizeFields) --- OUT---
20221025 103435.016 INFO             PET00 atmesh.F90:432  (ATMESH:RealizeFieldsProvidingGrid) --- initialization phase 2 completed ---
20221025 103435.258 INFO             PET00 atmesh.F90:704  ATMESH currTime =         2008 /           9 /           4            12 :           0 :           0
20221025 103435.258 INFO             PET00  (atmesh_mod:read_atmesh_nc)atmesh file it index >           61
20221025 103435.260 INFO             PET00  (atmesh_mod:read_atmesh_nc) --- read ATMesh netcdf file  ---
20221025 103435.260 ERROR            PET00 ESMF_StateAPI.cppF90:2565 ESMF_StateGet Not found  -  no ESMF_Field found named: pmsl
20221025 103435.260 ERROR            PET00 atmesh.F90:813 Not found  - Passing error in return code
pvelissariou1 commented 1 year ago

CoastalApp components ATMESH and WW3 have been updated to conform to UFS/CoastalApp standards regarding the definition of the ESMF atmospheric fields. The ATMESH/atmesh.F90 contains the option "#define INITSTEPFIX_on" to allow to fill the first time step fields with proper values from the input atm. NetCDF file (UND fix). NOTE: This is a temp fix.

By default WW3 uses PDLIB. To turn it off please modify the file WW3/model/esmf/switch.

Updated branch: develop_build

saeed-moghimi-noaa commented 1 year ago

Hi Takis

Any update on atmesh-ww3 for hsofs mesh?

Best Saeed

On Saturday, October 29, 2022, Panagiotis Velissariou < @.***> wrote:

CoastalApp components ATMESH and WW3 have been updated to conform to UFS/CoastalApp standards regarding the definition of the ESMF atmospheric fields. The ATMESH/atmesh.F90 contains the option "#define INITSTEPFIX_on" to allow to fill the first time step fields with proper values from the input atm. NetCDF file (UND fix). NOTE: This is a temp fix.

By default WW3 uses PDLIB. To turn it off please modify the file WW3/model/esmf/switch.

Updated branch: develop_build

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci3.googleusercontent.com/proxy/pkuFXOVktJKI_Yig_P8Jl6ciJQ1MLK4L4MdmYcjd1MMfj0jwZPea8lJjR2dTc0HF4uFZViPwYhvLPG6Bb9nTB3tqyBzw0oC6Qp4rthzj3ROXCaTyPJNyL4khpA0ZlO1M3C5weOqfaRsAa2cTe560uEoKNLc87iFlxJdQcA3R73O3EJOMDsmflGW9wCjpaqSXR_-CdOA3FPhf7pyBb8CyoqO-KbiSP0xLHmsr1iQG_q_lawsgfw=s0-d-e1-ft#https://github.com/notifications/beacon/APZULD6UPZ263FOWAKFD6OTWFQO6TA5CNFSM6AAAAAAQLO6VTKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSNGV2DW.gif>Message ID: @.***>

--

Saeed Moghimi, PhD NOS Storm Surge Modeling https://nauticalcharts.noaa.gov/learn/storm-surge-modeling.html Team Lead Coastal Marine Modeling Branch, Coast Survey Development Laboratory, Office of Coast Survey at NOAA National Ocean Service. Address: 1315 East West Hwy, Room 6607, Silver Spring, Maryland 20910

Phone: ‪(302) 827-3986

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

pvelissariou1 commented 1 year ago

Update 1 on CoastalApp WW3: It seems the main problem all the time was that in CoastalApp, ParMETIS was being built using the 64bit flag on (per install instructions) which was producing a buggy ParMETIS built. Unfortunately no error/warning messages were ever produced when running CoastalApp with the 64bit on for ParMETIS. The fix has been pushed to CoastalApp/develop_build.

CoastalApp built WW3 (ww3_multi, NEMS) are working now using both hpc_stack or CostalApp's system modulefile for hera/intel. Test of florence hsofs/ww3_multi works now in all cases within CoastalApp. Switching to full testing of ATMESH+WW3, PAHM+WW3 configurations ...

aliabdolali commented 1 year ago

Update 1 on CoastalApp WW3: It seems the main problem all the time was that in CoastalApp, ParMETIS was being built using the 64bit flag on (per install instructions) which was producing a buggy ParMETIS built. Unfortunately no error/warning messages were ever produced when running CoastalApp with the 64bit on for ParMETIS. The fix has been pushed to CoastalApp/develop_build.

CoastalApp built WW3 (ww3_multi, NEMS) are working now using both hpc_stack or CostalApp's system modulefile for hera/intel. Test of florence hsofs/ww3_multi works now in all cases within CoastalApp. Switching to full testing of ATMESH+WW3, PAHM+WW3 configurations ...

Great news Takis!!!

pvelissariou1 commented 1 year ago

Thanks Ali, and thank you for all your help trying to fix all the issues with CoastalApp/WW3.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Nov 16, 2022 at 1:22 PM Ali.Abdolali @.***> wrote:

Update 1 on CoastalApp WW3: It seems the main problem all the time was that in CoastalApp, ParMETIS was being built using the 64bit flag on (per install instructions) which was producing a buggy ParMETIS built. Unfortunately no error/warning messages were ever produced when running CoastalApp with the 64bit on for ParMETIS. The fix has been pushed to CoastalApp/develop_build.

CoastalApp built WW3 (ww3_multi, NEMS) are working now using both hpc_stack or CostalApp's system modulefile for hera/intel. Test of florence hsofs/ww3_multi works now in all cases within CoastalApp. Switching to full testing of ATMESH+WW3, PAHM+WW3 configurations ...

Great news Takis!!!

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/134#issuecomment-1317550310, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2QRGFACQOXNBWVDJLWIUX6HANCNFSM6AAAAAAQLO6VTI . You are receiving this because you were assigned.Message ID: @.***>

saeed-moghimi-noaa commented 1 year ago

@GM-Ling @pvelissariou1 @aliabdolali @FVCOM-GitHub @platipodium @josephzhang8 @sarriegi

All,

I would like to share with you our intermediate test suit repo. I uploaded the Shinnecock inlet tests. Please see Readme.md at the repo mentioned below for HOWTO.

https://github.com/saeed-moghimi-noaa/coastal_app_test

We will include tests for a larger mesh on 100s of cores hopefully by mid-next week.

Thanks for your patience,

saeed-moghimi-noaa commented 1 year ago

@GM-Ling @pvelissariou1 @aliabdolali @FVCOM-GitHub @platipodium @josephzhang8 @sarriegi

All, Just added some hsofs (1.8M) nodes tests using multiple nodes for WW3 and ADCIRC. You will need to download ~12Gb of data.

This is an unofficial/crude test suits to support some of you with urgent need for test/develop codes. Takis and I will update this and the CostalApp code base in the following days.

See Readme.md in https://github.com/saeed-moghimi-noaa/coastal_app_test .

Thanks, -Saeed

pvelissariou1 commented 1 year ago

Regression tests on hera using PaHM as forcing engine:

  1. PaHM+ADCIRC (florence, successful)
  2. PaHM+WW3 (florence, successful)
  3. PaHM+ADCIRC+WW3 (florence, successful)
  4. PaHM+ADCIRC (shinnecock, successful)
  5. PaHM+WW3 (shinnecock, successful)
  6. PaHM+ADCIRC+WW3 (shinnecock, successful)

Scripts have been written/modified for the test cases. Will go through with Saeed to explain how things work. Need to move the full test suite to a more common/shared location for the team.

Notes:

pvelissariou1 commented 1 year ago

There is a need to have a full case that uses the latest 120m HSOFS mesh

pvelissariou1 commented 1 year ago

First update of CoastalApp-testsuite in place. Recostructed the tests, main scripts in place. Will incorporate the HSOFS tests on 12/19/2022. Workin on main scripts for the automation of tests.

saeed-moghimi-noaa commented 1 year ago

@FVCOM-GitHub

You need to follow here. @pvelissariou1 will update this today afternoon once again.

https://github.com/noaa-ocs-modeling/CoastalApp-testsuite

The safest is to you start testing this following the readme tomorrow.

-Saeed

pvelissariou1 commented 1 year ago

@saeed-moghimi-noaa It seems that WW3 requires extended memory for its large domain simulations. All ww3 test case(s) in CoastalApp-testsuite run successfuly by allowing slurm to use all available memory at each compute node. For example, the test case florence_hsofs.atm2adc2ww3 runs successfuly by using the sbatch option: --mem=0 (see the *.job slurm scripts in the run directory of the test case). All test cases in CoastalApp-testsuite were run successfuly on "hera" using the Intel compilers. It is not clear why this test case was running successfuly before and now requires to set memory requirements for slurm. Simulation results for this test case can be found at: /scratch2/COASTAL/coastal/noscrub/shared/Takis/coastal_app_test/work/florence_hsofs.atm2adc2ww3/run

saeed-moghimi-noaa commented 1 year ago

@pvelissariou1 @aliabdolali @platipodium @FVCOM-GitHub

Please notice also that we allocate twice of the nodes and use half of cores for WW3 cases to make sure WW3 is not running out of memory.

aliabdolali commented 1 year ago

@pvelissariou1 @aliabdolali @platipodium @FVCOM-GitHub

Please notice also that we allocate twice of the nodes and use half of cores for WW3 cases to make sure WW3 is not running out of memory.

This is a known requirement from WW3. I should've told you before. In the most recent version of WW3, it requires less memory but it is not yet synced in Coastal-app

FVCOM-GitHub commented 1 year ago

@saeed-moghimi-noaa @pvelissariou1

Successfully repeated the case we did before (without PDLIB). I read the discussion between Saeed and Ali here and know there are two more 'no' parameters in file ww3_multi.inp now. Will try with parameter "PDLIB". Thanks.

saeed-moghimi-noaa commented 1 year ago

@pvelissariou1 @aliabdolali @platipodium @FVCOM-GitHub @GM-Ling @josephzhang8 @sarriegi

I am going to close this issue as the issue is resolved. Please follow up and comment in this discussion board regarding regression testing of the CoastalApp.

PLEASE follow up here and do not communicate via emails

https://github.com/noaa-ocs-modeling/CoastalApp/discussions/150