ADCIRC integration

[uturuncoglu]

I am creating new issue for ADCIRC integration. As we discussed today, I'll look at option to create additional layer in the ADCIRC cap to interact with CMEPS. I'll update you about it once I have discussion with ESMF core team.

[pvelissariou1]

@Ufuk Turuncoglu @.> , @Saeed Moghimi - NOAA Federal @.> Ufuk thank you for the update. As long as this is the case, please have a meeting with Bob as soon as possible to see what can be done from the ESMF side. I will be at the ADCIRC User's meeting next week (06/08 (T) - 06/09 (F)) and I will try to talk to the developers over there. Could you please write a paragraph or two (in this thread) briefly explaining what is the problem and what we expect (want) the ADCIRC developers to help with. I understand that everyone is busy, but somehow, we need to get them more actively involved in helping, if ADCIRC is going to be one of the model components in ufs-coastal.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, May 31, 2023 at 4:48 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 i was thinking that I solved the issue related to the ADCIRC mesh to make it compatible with CMEPS but apparently not. I have still same issue. So, it might be nice to have a call with ADCIRC developers soon. I still could not talk with Bob (from ESMF team) since he is busy with some other works but I will talk with him next week. So, we might wait for him and then talk with ADCIRC developers later. Anyway, let me know what you think?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1571008139, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2J53M57MC2OGAOPUTXI64BVANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[saeed-moghimi-noaa]

@pvelissariou1 @uturuncoglu @danrosen25

Thanks Takis and Ufuk.

I totally agree.

We are into the month six of the project and I feel like we need to move faster to make sure we cover the 1st year goals. Perhaps in parallel, NCAR team may start looking into ESMX driver implementation meanwhile waiting for developers to respond. Let me know what you think?

Thanks a lot, -Saeed

[uturuncoglu]

@saeed-moghimi-noaa @danrosen25 is working on ESMX and the more stable version with new feature will be available in 8.5.0 (will be released in a month). Then, we could start to implement ESMX under ufs-coastal. Probably he has more information about it. In the meantime, I am plaining to keep working on ADCRIC cap to solve the issue. Since FVCOM and ADCIRC are very similar interns of mesh generation, I think FVCOM integration will be fast.

[dwirasae]

@dwirasae As I see there are some elements with negative id (means ghost) but all nodes are resident. I am expecting to have positive element id for those elements. Do you have any explanation about it?

No, I do not. I expect those elements to have a postive id. Let me double check and get back to you.

[uturuncoglu]

@dwirasae Thanks for your help. Maybe there is a bug in the code. I'll double check in my side.

[uturuncoglu]

@dwirasae I am not sure I am following correctly or not but one example of having negative element id with all resident nodes from florence_hsofs.atm2adc test.

In the PE0144/fort.18 file, I have following entry

FileFmt      1050624           0           0
NELG         3564104        7707        6972
       -6971
       -6972
       -6981
       -6982
       -6991
       -6992
       -7001
       -7002
       -7011
       -7012
       -7019
       -7020
       -7021
       -7022
       -7023
...
...

So, if you look at element with global id -7023 it is in 25th row (23th element locally since 2 rows are header). Then, if I look at PE0144/fort.14 file to find its nodes, you just need to look at 23th entry and it is something like following,

23 3 4118 4119 4124

So, local id of the element -7023 is 23 and seems that 4118, 4119 and 4124 are the nodes (I think these are global but all the entry in the element connection list positive). Again from the same file their entry like following,

    4118 -0.904350820000E+02  0.299931520000E+02  0.221426280000E+02 <--- 18th entry in NOMAD section
    4119 -0.904363980000E+02  0.299907540000E+02 -0.275970200000E+01 <--- 19th entry
    4124 -0.904338730000E+02  0.299905870000E+02 -0.298566510000E+01 <--- 24th entry

This does not say anything about the node is ghost or not since all ids are local and positive in PE0144/fort.14. Right? But, could find the global ids of those from PE0144/fort.18 file by looking NNODG section. If I look at entry 18, 19 and 24 after NNODG section, you can find the global ids of those nodes and they are all positive. So, it seems that they are resident but if I look at RES NODE for other SEND/RECV PE section i could not find those nodes (maybe they are listed local ids). Anyway, let me know what you think?

[dwirasae]

@uturuncoglu. I look into the decomposition code. This could happen is a scenario where there are internal 'weir' boundaries (to handle levees, jetties). The hsof grid may use these boundaries in southern Louisiana. I will take a look at the test and get back to you.

[uturuncoglu]

@dwirasae Thanks for your help. It is good to know, it is possible. Is there any way to identify those elements/nodes? BTW, following is the example for most elements (from PET144 and PET143). The pink region is exiting in the both PE and I am trying get rid of that. If I could identify those 'weir' boundaries, then maybe I could fix the issue. The subroutine to eliminate the ghost elements is ready but since I could not identify which element will be stay and which one will be eliminated, I could not make it work. Since this case uses 499 core, it is also hard to debug and follow the connection between elements and nodes. So, maybe it would be setup a realistic case with less number of cores and relatively low resolution mesh but that is another approach that we could take if I could not solve the issue. I will be out of office tomorrow but I'll check my mails and this issue page for any update.

[uturuncoglu]

@dwirasae I look at it little bit more and I think that it is hard to access weir boundaries with the current version of code since this information is not written to any file like fort.14 or fort.18 and it is just available in adcprep tool. I am trying to include all the elements that at least two nodes are resident but in this case some nodes will be outside because those elements has only one resident nodes and ESMF mesh creation is complaining about it. At this point I am not sure those nodes related with weir boundaries or not until put some debug code to adcprep and try to write id of those nodes. Anyway, do you have any suggestion?

[uturuncoglu]

@dwirasae @saeed-moghimi-noaa @pvelissariou1

As we discussed today we have two issues:

1. We need to eliminate ghost elements from the mesh. I tried different appraoches without any luck and just by looking element ids >0 is not working at this point. I think we need to have new data structures (simply mask) that indicates the element is ghost or not.

2. The import and export fields are need to be defined on ESMF_MESHLOC_ELEMENT (the default one is node). I think this is secondary issue and we need to solve the previous issue first. Then, we could look at this further.

Anyway, here is the details of my initial development to solve the (1) issue.

I have just push the initial development in ADCIRC cap to resolve the issues to make it work with CMEPS mediator. It is not working at this point since we have no enough information to eliminate ghost elements and just looking to positive element ids is not solving. Here is the details of the code,

• I am calling a special routine called like eliminate_ghosts() just after reading mesh information. As you can see this depends on some configuration option called meshloc to isolate this development. The main goal is in here to remove the ghost elements from the every PEs and the mesh.

https://github.com/uturuncoglu/adcirc/blob/509cf9ad9551a4bfd7ca592a166e27ef5bd6420c/thirdparty/nuopc/adc_cap.F90#L686

• The routine is in the following link. The ElMask variable is just looking for the elements that the_data%ElIDs(i) > 0 at this point and not working with CMEPS. So, we need to find a way in here to filter out the ghost elements. https://github.com/uturuncoglu/adcirc/blob/509cf9ad9551a4bfd7ca592a166e27ef5bd6420c/thirdparty/nuopc/adc_mod.F90#L424

[dwirasae]

@uturuncoglu Sorry for not responding promptly. Do you have by any chance PEXXXX directories from running the prep?

[uturuncoglu]

@dwirasae No worries. I just switched another component - FVCOM to use my time more efficiently. Since that one shared most of the mesh generation code with ADCIRC, probably I'll have similar issue. Do you have access to NCAR's Cheyenne system. If so, you could find the files in the /glade/scratch/turuncu/COASTAL_APP/florence_hsofs.atm2adc_med directory. A this point I have only access Cheyenne and Orion.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Ufuk, is it possible to deploy all these files on orion. ADCIRC folks have access to orion.

Thanks

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jun 6, 2023 at 5:08 PM Ufuk Turunçoğlu @.***> wrote:

@dwirasae https://github.com/dwirasae No worries. I just switched another component - FVCOM to use my time more efficiently. Since that one shared most of the mesh generation code with ADCIRC, probably I'll have similar issue. Do you have access to NCAR's Cheyenne system. If so, you could find the files in the /glade/scratch/turuncu/COASTAL_APP/florence_hsofs.atm2adc_med directory. A this point I have only access Cheyenne and Orion.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1579518317, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2GVUYHOBIZKAHRYE3XJ6S55ANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@dwirasae @pvelissariou1 I put the zipped PE files to /work/noaa/nems/tufuk/PE.tar.gz on Orion. Let me know if you have access issue.

[uturuncoglu]

@pvelissariou1 JFYI, I am getting following errors from ADCIRC,

141: TIME STEP =  302402     ITERATIONS =    0     TIME =  0.60480400E+06
141:  ELMAX =  7.1438E+035 AT NODE     1267  SPEEDMAX =  3.3298E+034 AT NODE     1275  ON MYPROC =  125   ** WARNING: Elevation.gt.WarnElev **
141: TIME STEP =  302402     ITERATIONS =    0     TIME =  0.60480400E+06
141:  ELMAX =  7.1438E+035 AT NODE     1267  SPEEDMAX =  3.3298E+034 AT NODE     1275  ON MYPROC =  125
141:  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **
156: TIME STEP =  302402     ITERATIONS =    0     TIME =  0.60480400E+06
156:  ELMAX =  4.2046E+276 AT NODE   199608  SPEEDMAX =     Infinity AT NODE   202232  ON MYPROC =  140   ** WARNING: Elevation.gt.WarnElev **
156: TIME STEP =  302402     ITERATIONS =    0     TIME =  0.60480400E+06
160: TIME STEP =  302402     ITERATIONS =    0     TIME =  0.60480400E+06
160:  ELMAX =  4.2046E+276 AT NODE  -199608  SPEEDMAX =     Infinity AT NODE  -202232  ON MYPROC =  144   ** WARNING: Elevation.gt.WarnElev **
160: TIME STEP =  302402     ITERATIONS =    0     TIME =  0.60480400E+06
160:  ELMAX =  4.2046E+276 AT NODE  -199608  SPEEDMAX =     Infinity AT NODE  -202232  ON MYPROC =  144
160:  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **
156:  ELMAX =  4.2046E+276 AT NODE   199608  SPEEDMAX =     Infinity AT NODE   202232  ON MYPROC =  140
156:  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **

This is without CMEPS and I already updated my fork with ADCIRC/development. Since we are using different dataset to force ADCIRC under ufs-coastal, it would create issue. anyway, I think you are working on CoastalApp testsuite to make input files consistent and adjust configuration to write output regular intervals. So, once this is done, I think I need to update all the RTs defined under ufs-coastal and again compare them with the CoastalApp testsuite ones.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Ufuk, I was getting these warnings/errors when running PaHM+ADCIRC due to a bug in PaHM. What test case are you running?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jul 5, 2023 at 3:39 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 JFYI, I am getting following errors from ADCIRC,

141: TIME STEP = 302402 ITERATIONS = 0 TIME = 0.60480400E+06 141: ELMAX = 7.1438E+035 AT NODE 1267 SPEEDMAX = 3.3298E+034 AT NODE 1275 ON MYPROC = 125 WARNING: Elevation.gt.WarnElev 141: TIME STEP = 302402 ITERATIONS = 0 TIME = 0.60480400E+06 141: ELMAX = 7.1438E+035 AT NODE 1267 SPEEDMAX = 3.3298E+034 AT NODE 1275 ON MYPROC = 125 141: ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. 156: TIME STEP = 302402 ITERATIONS = 0 TIME = 0.60480400E+06 156: ELMAX = 4.2046E+276 AT NODE 199608 SPEEDMAX = Infinity AT NODE 202232 ON MYPROC = 140 WARNING: Elevation.gt.WarnElev 156: TIME STEP = 302402 ITERATIONS = 0 TIME = 0.60480400E+06 160: TIME STEP = 302402 ITERATIONS = 0 TIME = 0.60480400E+06 160: ELMAX = 4.2046E+276 AT NODE -199608 SPEEDMAX = Infinity AT NODE -202232 ON MYPROC = 144 WARNING: Elevation.gt.WarnElev 160: TIME STEP = 302402 ITERATIONS = 0 TIME = 0.60480400E+06 160: ELMAX = 4.2046E+276 AT NODE -199608 SPEEDMAX = Infinity AT NODE -202232 ON MYPROC = 144 160: ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. 156: ELMAX = 4.2046E+276 AT NODE 199608 SPEEDMAX = Infinity AT NODE 202232 ON MYPROC = 140 156: ERROR: Elevation.gt.ErrorElev, ADCIRC stopping.

This is without CMEPS and I already updated my fork with ADCIRC/development. Since we are using different dataset to force ADCIRC under ufs-coastal, it would create issue. anyway, I think you are working on CoastalApp testsuite to make input files consistent and adjust configuration to write output regular intervals. So, once this is done, I think I need to update all the RTs defined under ufs-coastal and again compare them with the CoastalApp testsuite ones.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1622466633, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4CEBYO57SL5VHT3DTXOXGHJANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 This is just ADCIRC coupled with CDEPS.

[pvelissariou1]

ok, thanks. I'll check it out.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Jul 5, 2023 at 3:59 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 This is just ADCIRC coupled with CDEPS.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1622496340, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPYIJ43JSJ2MCDPPQ6TXOXITVANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I might need input for CDEPS to use under ike_shinnecock.atm2adc case. I checked ike_shinnecock.atm2sch but that one is using different start date for the simulation and it does not cover this configuration. Anyway, I'll try to force it with ERA5 and let you know.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Please Let me know, I can create an atm. input file if needed.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Jul 7, 2023 at 3:40 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I might need input for CDEPS to use under ike_shinnecock.atm2adc case. I checked ike_shinnecock.atm2sch but that one is using different start date for the simulation and it does not cover this configuration. Anyway, I'll try to force it with ERA5 and let you know.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1626065342, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6K4WSQILEJX4BOALLXPBXZXANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Okay. Downloading now. It would be very coarse in terms of spatial resolution but at least it will give me chance to run the case. We could update the input later once we have consistent inputs for all Ike and Florence. I'll let you know if I need help about input.

[uturuncoglu]

@pvelissariou1 I am getting following error again from this case,

2: TIME STEP =  540535     ITERATIONS =   20     TIME =  0.10810700E+07
2:  ELMAX =  1.3245E+003 AT NODE      -98  SPEEDMAX =  1.2296E+003 AT NODE      -25  ON MYPROC =    2
9: TIME STEP =  540535     ITERATIONS =   20     TIME =  0.10810700E+07
9:  ELMAX =  1.3245E+003 AT NODE       98  SPEEDMAX =  1.6949E+003 AT NODE       26  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **
9: TIME STEP =  540535     ITERATIONS =   20     TIME =  0.10810700E+07
9:  ELMAX =  1.3245E+003 AT NODE       98  SPEEDMAX =  1.6949E+003 AT NODE       26  ON MYPROC =    9
9:  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **
2:  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **

Maybe it is because of the input. Not sure. Any idea?

[pvelissariou1]

@Ufuk Turuncoglu @.***> It is strange. Let me check this out. Please share the atmospheric data file. Need to go to get my car from the shop. I'll test this later tonight. I'll keep you posted.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Jul 7, 2023 at 3:59 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I am getting following error again from this case,

2: TIME STEP = 540535 ITERATIONS = 20 TIME = 0.10810700E+07 2: ELMAX = 1.3245E+003 AT NODE -98 SPEEDMAX = 1.2296E+003 AT NODE -25 ON MYPROC = 2 9: TIME STEP = 540535 ITERATIONS = 20 TIME = 0.10810700E+07 9: ELMAX = 1.3245E+003 AT NODE 98 SPEEDMAX = 1.6949E+003 AT NODE 26 ON MYPROC = 9 WARNING: Elevation.gt.WarnElev 9: TIME STEP = 540535 ITERATIONS = 20 TIME = 0.10810700E+07 9: ELMAX = 1.3245E+003 AT NODE 98 SPEEDMAX = 1.6949E+003 AT NODE 26 ON MYPROC = 9 9: ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. 2: ERROR: Elevation.gt.ErrorElev, ADCIRC stopping.

Maybe it is because of the input. Not sure. Any idea?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1626104186, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPYPNWQOBNODIWUQZDLXPB2DFANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 It is in following directory on Cheyenne.

/glade/scratch/turuncu/COASTAL_APP/ike_shinnecock.atm2adc/INPUT/download_200809_rlat.nc

Maybe I could try to write out field in ADCIRC side in VTK format using ESMF calls to check them.

[uturuncoglu]

@pvelissariou1 I think the interpolation is not working properly. The ADCRIC that I used in here is using connectors and no modification in the cap. The following image is the decomposition blocks,

and the next one is pressure. As you can see that we have issue in the interpolation in the boundaries between decomposition elements. This could be caused by ghost elements. I just wonder that did you check the results under CoastalApp-test suite?

In the CoastalApp-testsuite, the run sequence is defined like following,

# Run Sequence #
runSeq::
  @3600
    ATM -> OCN   :remapMethod=redist
    ATM
    OCN
  @
::

but in my case, I am using following to force bit-to-bit reproducibility and in my case I am not using data that is in the same resolution with the model and using bilinear interpolation (in the CoastalApp case it is redist),

runSeq::
  @3600
    ATM -> OCN :remapMethod=bilinear:unmappedaction=ignore:zeroregion=select:srcTermProcessing=0:termOrder:srcseq
    ATM
    OCN
  @
::

Anyway, I'll do further investigation and let you know.

[uturuncoglu]

@pvelissariou1 I am trying to run CoastalApp-testsuite on Orion to see I could reproduce the issue in there. I'll update you.

[uturuncoglu]

@pvelissariou1 Okay. I am getting following error. I think there is an issue in the system or related with the nems account that I am using since I think there was a disk quota issue recently. Is it possible to add me to the coastal group on Orion? I could also try on Cheyenne again but I am not sure CoastalApp-testsuite is working in there or not.

[1688769215.647710] [Orion-01-09:440986:0]        ib_verbs.h:88   UCX  ERROR ibv_exp_query_device(mlx5_0) returned 38: No space left on device
[1688769215.647753] [Orion-01-09:440984:0]        ib_verbs.h:88   UCX  ERROR ibv_exp_query_device(mlx5_0) returned 38: No space left on device

[uturuncoglu]

@pvelissariou1 Just for checking, I run SCHISM for the same domain (ike_shinnecock.atm2sch configuration) and forced with the CDEPS (again ERA5) but I am not seeing same issue. So, it seems that ghost elements are creating issue in the interpolation (at this point I am not sure why we are seeing issue also inside of the elements but those could be side effects). I still could not run same case (ike_shinnecock.atm2adc) under CoastalApp and still getting same error. I'll try again on Cheyenne to see what happens. If the all cases with ADCIRC uses same grid for all the components maybe this is not matched before but not sure.

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa I could able to run out-of-box ike_shinnecock.atm2adc ADCIRC case on Orion from CoastalApp-test suite and it seems that it has also same issue. See following plot for surface pressure. Also note that this uses redist since it uses ATMMESH. I am not sure why this error is not cached before under CoastalApp or their applications. Any idea? Definitely ADCIRC cap has issue due to ghost elements. This is the connector only and I think I could add it to the presentation as issue 3. Let me know what you think? If you could reproduce the same in your end that would be great.

[uturuncoglu]

@pvelissariou1 Based on my experience it is not likely but this could be an initialization issue of the configuration too. It worths to get opinion from ADCIRC developers. But, still strange to see this pattern in the input forcing like pressure field after interpolation. The mode does not involve that step. So, this seems related with the interpolation artifact due to the ghost element. I'll check it with Bob from ESMF team and get his feeling too. In any case, it would be great that someone else could confirm this finding in CoastalApp side.

[uturuncoglu]

@pvelissariou1 I think the issue that I am seeing with original code could be related to the halo update. Since I am outputting fields just before/after run call with esmf field write, I might miss that halo update call that is inside of the adcirc run. So this might explain the structure seen in the connector only (and using redist) run under CoastalApp test suite. I'll check the model output to be sure. I am also planing to test same case by changing redist to bilinear to see any change in the result.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Thank you for the updates. I ran the test case on Orion and I didn't see the issues you described in the output file (fort.73.nc) or the input file. Also, in CoastalApp I haven't updated ADCIRC to its very latest version.

NOTE: FYI, on Orion using intel/2019.5 sometimes the simulation goes through and sometimes produces MPI errors as you mentioned before. intel/2020.2 does not work due to issues with stdc++ library for the CXX api. intel/2022.1.2 works flawlessly (I am switching CostalApp to use this compiler on Orion). Different HPC clusters have different configurations and things might break due to misconfigurations, for example intel (19/2019) works very well on hera and Frontera but it doesn't work as expected on Orion and ParallelWorks.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Jul 9, 2023 at 9:33 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think the issue that I am seeing with original code could be related to the halo update. Since I am outputting fields just before/after run call with esmf field write, I might miss that halo update call that is inside of the adcirc run. So this might explain the structure seen in the connector only (and using redist) run under CoastalApp test suite. I'll check the model output to be sure. I am also planing to test same case by changing redist to bilinear to see any change in the result.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1627732542, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPYFO6Y43MNP5CZOZUTXPK6LDANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Okay. At this point, I am assuming CoastalApp is working fine even without using latest version of ADCIRC. I also tried to run ike_shinnecock.atm2adc case under CoastalApp but changing redist option to bilinear to see any issue in the interpolation but it fails with 11: NetCDF Error: Variable col in weightmatrix_ATM-TO-OCN_inst_merid_wind_height10m.nc : NetCDF: Name contains illegal characters. So, maybe ATMMESH is not designed to work like that. Anyway, the issue that I am seeing under UFS-Coastal is maybe a specific issue. It is hard to create same configuration under CoastalApp since we replaced ATMMESH with CDEPS. I'll look at more to find the source of the interpolation error. BTW, even if you have correct fort.73.nc file, it does not mean the coupling is working fine since the import and export states are showing zero along the ghost regions and those are transferred to other components. We are using ATMMESH and there is nob two way coupling in here but in two-way coupled application, it would be nice to check the receiver about those zeros along the boundary to be sure those are not transferred.

[uturuncoglu]

@pvelissariou1 Thanks for your help. I think it would be great to test ufs-coastal with the same input used by ike_shinnecock.atm2adc configuration under CoastalApp testsuite. Is there any progress to make input files consistent across the CoastalApp testsuite and having those files in their original grid for CDEPS?

[uturuncoglu]

@pvelissariou1 I added new test to RTs. It is coastal_ike_shinnecock.atm2adc and working with CMEPS. Please try it under Orion and let me know if you have any issue. Same RT folder can be used.

[pvelissariou1]

@uturuncoglu Thank you, I will try it. Tomorrow I'll update the nuopc cap for FVCOM as well.

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa We were discussing with @dwirasae last couple of days related to the issue in the ADCIRC. We might have a call with him today. Let me know if you want to join.

[pvelissariou1]

Ufuk, thank you, I will join any time today. FYI, Saeed is extremely busy and I don't know if he will have free time to join.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Aug 24, 2023 at 11:44 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 @saeed-moghimi-noaa https://github.com/saeed-moghimi-noaa We were discussing with @dwirasae https://github.com/dwirasae lats couple of days related with the issue in the ADCIRC. We might have a call with him today. Let me know if you want to join.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1692053474, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZVSMTCCHKG6JP5RRDXW6AE5ANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I am trying to update florence_hsofs.atm2adc case from Coastal App test suite but some files are missing in the following directory such as fort.14. Is this case running under test suite? I could keep RTs under ufs-cosatal same and not update. Let me know your preference.

https://github.com/noaa-ocs-modeling/CoastalApp-testsuite/tree/develop/florence_hsofs.atm2adc/run

[pvelissariou1]

@uturuncoglu Ufuk, those "missing files" are contained in a separate archive accessible via the command: wget https://tacc-nos-coastalapp-testsuit.s3.amazonaws.com/hsofs-data-v3.tgz Over there you will find the files for adcirc. The files are rather large, that is why I don't have them in the github repo. You might want to extract the files from the archive and put them in the RT case in UFS-Coastal.

[uturuncoglu]

@pvelissariou1 Okay. Let me check it. Thanks.

[uturuncoglu]

@pvelissariou1 I think the file is under CoastalApp-testsuite/comm/mesh/. Right? Since there is no any subfolder dedicated to the component, I assuming that these files belongs to ADCIRC.

[pvelissariou1]

Correct

[uturuncoglu]

@pvelissariou1 Do we have 2d version of Florence_HWRF_HRRR_HSOFS.nc that can be used under CDEPS? Is it possible to create it?

[pvelissariou1]

@uturuncoglu I think I had created that file before. Let me check and if not I will create it.

[uturuncoglu]

@dwirasae @pvelissariou1 @saeed-moghimi-noaa There is a bug in ADCIRC side that prevents me to use different set of PEs. If I set it the nodes coordinates goes wrong and creates issue. If I align processors with other components it works as expected. Anyway, I am trying to debug and fix at this point. Another aspect is that ADCIC has capability to use dedicated I/O tasks but this feature can nat be used with the existing implementation of the NUOPC cap. We have same issue with SCHSIM new I/O feature. Anyway, I think they are not high priority at this point but it would be nice to open a ticket in component repository side to track.

[uturuncoglu]

@dwirasae @pvelissariou1 @saeed-moghimi-noaa In addition to the previous issue which I am still trying to fix. I think ADCIRC cap also needs to be fixed like FVCOM. I check the ADCIRC output with adcircpy tool and did not see any issue in the ADCIRC output but I believe that the way to get the data from input states and fill ADCIRC internal data structures is wrong. It still assumes that the data coming from ESMF mesh is on nodes. In fact, the ESMf fields on elements at this point.

[uturuncoglu]

@pvelissariou1 I am testing to use 1d data (on model mesh) through the CMEPS. data/atm/Florence_HWRF_HRRR_HSOFS.nc is used in atm2adc Florence case. This file covers from 2018-09-09 06:00:00 to 2018-09-30 23:00:00 but if you check following file https://github.com/noaa-ocs-modeling/CoastalApp-testsuite/blob/develop/florence_hsofs.atm2adc/run/model_configure, it seems that simulation is starting from 2018-09-07 00:00:00. So, CDEPS is failing since it could not find the required data in the file. If you don't mind could you double check. We might need to extend the data to cover simulation period. Maybe I need to download the data from scratch if you already updated. Anyway, let me know what you think.

[uturuncoglu]

@pvelissariou1 I am not sure but there might be an issue with Florence_HWRF_HRRR_HSOFS.nc input file. Here is the snapshot from ncdump command. At this point, number of element is greater than number of nodes. How this could happen. Is this file also includes ghost elements? Even if that I think it is way more than the number of nodes. I am not sure what is wrong with it.

netcdf Florence_HWRF_HRRR_HSOFS {
dimensions:
    level = 1 ;
    node = 1813443 ;
    element = 3564104 ;
    time = 522 ;
    noel = 3 ;

I have also case of unstructured data used by CESM and in their case the mesh file has following,

netcdf ne30pg3_ESMFmesh_cdf5_c20211018 {
dimensions:
    nodeCount = 48602 ;
    elementCount = 48600 ;
    maxNodePElement = 4 ;
    coordDim = 2 ;

So, as I expected the number of element is less than number of nodes. Anyway, maybe ATMMESH has some feature and required data like this but I think it would be hard to use it under CDEPS since there is inconsistency between node and element counts. We have case under CESM that uses unstructured data and it is working without any issue. I am modifying data file in a same way but in our case CDEPS fails. Anyway, it would be nice to check the data file and see why elements are greater than the nodes.

[uturuncoglu]

@pvelissariou1 I updated ADCIRC under ufs-coastal. Now, it points our official fork: https://github.com/oceanmodeling/adcirc and we are using feature/coastal_app branch for development.