ADCIRC integration

[uturuncoglu]

I am creating new issue for ADCIRC integration. As we discussed today, I'll look at option to create additional layer in the ADCIRC cap to interact with CMEPS. I'll update you about it once I have discussion with ESMF core team.

[uturuncoglu]

@pvelissariou1 I wonder if there is any progress about the issue that we faced during writing output? I think you were plaining to fix the configuration files and maybe you have some update about it.

[pvelissariou1]

@Ufuk Turuncoglu @.***> I modified the fort.15 files in the run folders for the ADCIRC cases to print out the results every hour (checked this on hera). fort.63.nc : Elevation Time Series at All Nodes in the Model Grid fort.64.nc : Depth-averaged Velocity Time Series at All Nodes in the Model Grid fort.73.nc : Atmospheric Pressure Time Series at All Nodes in the Model Grid fort.74.nc : Wind Stress or Velocity Time Series at All Nodes in the Model Grid

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Apr 5, 2023 at 12:26 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I wonder if there is any progress about the issue that we faced during writing output? I think you were plaining to fix the configuration files and maybe you have some update about it.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1496936257, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7F6B4C3IQOYC2ZR3LW7T7BPANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Thanks for your help. I will get new file form test suite repo. Do I need to do anything else? You know, i am using CFS data for this case.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Just get the fort.15 file from the run directory. You don't have to modify fort.15, it is designed to output data every hour for 7 days. If you just want to run the model for just one day, just modify the model_configure file accordingly. Also, I guess you have modified the config.rc file to point to the CFS data.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 1:00 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Thanks for your help. I will get new file form test suite repo. Do I need to do anything else? You know, i am using CFS data for this case.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1498533591, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4E5WH7XE2LEHBA4XLW7ZLWJANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 In my case model fails in ADCIRC_Init. I'll run the prep from scratch and test again to see what happens.

[uturuncoglu]

@pvelissariou1 prep is also giving following error,

0:forrtl: severe (59): list-directed I/O syntax error, unit 15, file /glade/scratch/turuncu/COASTAL_APP/florence_hsofs.atm2adc/fort.15

Do I need to update the model? I think this is working for you. Right?

[uturuncoglu]

@pvelissariou1 I think I am using wrong file. Let me copy again.

[uturuncoglu]

@pvelissariou1 It seems that it is writing to fort.* files but I have still zero time dimension in following files,

-rw-r--r-- 1 turuncu ncar  45505303 Mar 24 12:56 maxele.63.nc
-rw-r--r-- 1 turuncu ncar  45505232 Mar 24 12:57 maxrs.63.nc
-rw-r--r-- 1 turuncu ncar  45510050 Mar 24 12:57 maxvel.63.nc
-rw-r--r-- 1 turuncu ncar  45505182 Mar 24 12:57 maxwvel.63.nc
-rw-r--r-- 1 turuncu ncar  45505295 Mar 24 12:57 minpr.63.nc

Is it same also for you? Anyway, I think I need to check the model results from fort* files. If you don't mind could you remind me about the content of those files?

[pvelissariou1]

@Ufuk Turuncoglu @.***> Sorry for the late reply, I was working on something else. You need to copy the fort.15 files in the run folders of the hsofs test cases. They are working. You don't need to update/recompile anything. The files maxele.63.nc maxvel.63.nc maxwvel.63.nc minpr.63.nc should only have 1 time record in them (the last time record) as they contain the max/or min values of the results for the simulation period. The file maxrs.63.nc is for the wave radiation stresses (no waves in this simulation).

Please use fort.63.nc and fort.64.nc when comparing the results.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 1:06 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 It seems that it is writing to fort.* files but I have still zero time dimension in following files,

-rw-r--r-- 1 turuncu ncar 45505303 Mar 24 12:56 maxele.63.nc -rw-r--r-- 1 turuncu ncar 45505232 Mar 24 12:57 maxrs.63.nc -rw-r--r-- 1 turuncu ncar 45510050 Mar 24 12:57 maxvel.63.nc -rw-r--r-- 1 turuncu ncar 45505182 Mar 24 12:57 maxwvel.63.nc -rw-r--r-- 1 turuncu ncar 45505295 Mar 24 12:57 minpr.63.nc

Is it same also for you? Anyway, I think I need to check the model results from fort* files. If you don't mind could you remind me about the content of those files?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1499435312, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP347GGDARN4LY3YC6TW74A2VANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I have no data in maxele.63.nc, maxvel.63.nc, maxwvel.63.nc and minpr.63.nc. Their time dimension is 0. I think this is because we set the simulation length 7 days and I am running model 1 day. Maybe I could run 7 days to see what happens. What do I need to change to adjust fort.15 to run mode 1 day? fort.63 and 64 seems fine at this point.

[pvelissariou1]

@Ufuk Turuncoglu @.**> In the fort.15 file there is the RNDAY line for the total simulation time. RNDAY value should be 7 + run_simulation_days. The 7* represents the spinup period used for the simulation. I have set RNDAY to 14 in the fort.15 files. For shorter simulation periods you may only adjust the times in the model_configure file.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 2:57 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I have no data in maxele.63.nc, maxvel.63.nc, maxwvel.63.nc and minpr.63.nc. Their time dimension is 0. I think this is because we set the simulation length 7 days and I am running model 1 day. Maybe I could run 7 days to see what happens. What do I need to change to adjust fort.15 to run mode 1 day? fort.63 and 64 seems fine at this point.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1499544967, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6EVE5VWNFOXMSN7G3W74N2HANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Okay. I understand. I'll try to adjust it for my case. Thanks for your great help. Sorry about lots of message.

[uturuncoglu]

I am not sure this is expected or not but I plot fort.63.nc file using adcircpy tool. Here is the plot for first time step and last one.

The last time step seems suspicious to me. In this case I am running model 1 day (let me know, if I need to run longer). Since we are forcing the model with CFS data, there might be some issue over there. I'll try to create visualization for the fields that goes to ADCIRC to be sure that interpolation is okay.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Please let it run for a week Can you plot the data from frot.73/fort.74?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 3:51 PM Ufuk Turunçoğlu @.***> wrote:

I am not sure this is expected or not but I plot fort.63.nc file using adcircpy tool. Here is the plot for first time step and last one.

[image: fort63_0] https://user-images.githubusercontent.com/1228999/230491112-5d3f45be-3647-45e9-8ede-82edabca3828.png [image: fort63_23] https://user-images.githubusercontent.com/1228999/230491123-04a5a151-eafd-49f8-acbb-2e8ea592b29a.png

The last time step seems suspicious to me. In this case I am running model 1 day (let me know, if I need to run longer). Since we are forcing the model with CFS data, there might be some issue over there. I'll try to create visualization for the fields that goes to ADCIRC to be sure that interpolation is okay.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1499609757, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP3BNJJHPR6W2OHBWG3W74UFZANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Okay. I run it 7 days and now I have data on those files. I also changed WTIMINC parameter and change the temporal resolution of the data from 3600 to 21600 (only first number) to fit to my CFS data. Let me know if you think that that is wrong. I also checked the results after 7 days and it looks similar to after 1 days. There might be an issue in the CFS forcing. Does ADCIRC also write forcing to the file? Is it possible to change the configuration to write those?

[pvelissariou1]

@Ufuk Turuncoglu @.***> Yes it does for the interpolated wind data to ADCIRC's mesh. These are the files fort.73.nc and fort.74.nc. Explanation: fort.73.nc : Atmospheric Pressure Time Series at All Nodes in the Model Grid fort.74.nc : Wind Stress or Velocity Time Series at All Nodes in the Model Grid

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 11:19 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. I run it 7 days and now I have data on those files. I also changed WTIMINC parameter and change the temporal resolution of the data from 3600 to 21600 (only first number) to fit to my CFS data. Let me know if you think that that is wrong. I also checked the results after 7 days and it looks similar to after 1 days. There might be an issue in the CFS forcing. Does ADCIRC also write forcing to the file? Is it possible to change the configuration to write those?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1499917667, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6XCW4SLZGTGGKBIU3W76IWHANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 I think I found the issue related with the results. The CFS data that you provide me does not cover the small part in south edge of the domain. Please, look at following plot. The white is the CFS mesh and the red one is ADCIRC. As you can see small part of ADCIRC mesh is in the outside of the CFS mesh. So, if you don't mind could you regenerate the CFS data again but in this time please extend it in little bit more south. I think that will solve the issue that we are seeing at this point.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Ok, I will recreate the data. Thank you for checking.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Apr 9, 2023 at 1:04 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think I found the issue related with the results. The CFS data that you provide me does not cover the small part in south edge of the domain. Please, look at following plot. The white is the CFS mesh and the red one is ADCIRC. As you can see small part of ADCIRC mesh is in the outside of the CFS mesh. So, if you don't mind could you regenerate the CFS data again but in this time please extend it in little bit more south. I think that will solve the issue that we are seeing at this point.

[image: Screen Shot 2023-04-09 at 12 00 42 AM] https://user-images.githubusercontent.com/1228999/230757221-be957edf-cb59-421c-bd2e-8137948b3cb7.png

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1501049923, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4IB2NKTDKL2IDXNSLXAJGPNANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[pvelissariou1]

@Ufuk Turuncoglu @.***> Please get the new Florence CFS data file on orion at: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/Florence_CFS.nc

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Apr 10, 2023 at 8:48 AM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

@Ufuk Turuncoglu @.***> Ok, I will recreate the data. Thank you for checking.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Apr 9, 2023 at 1:04 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think I found the issue related with the results. The CFS data that you provide me does not cover the small part in south edge of the domain. Please, look at following plot. The white is the CFS mesh and the red one is ADCIRC. As you can see small part of ADCIRC mesh is in the outside of the CFS mesh. So, if you don't mind could you regenerate the CFS data again but in this time please extend it in little bit more south. I think that will solve the issue that we are seeing at this point.

[image: Screen Shot 2023-04-09 at 12 00 42 AM] https://user-images.githubusercontent.com/1228999/230757221-be957edf-cb59-421c-bd2e-8137948b3cb7.png

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1501049923, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4IB2NKTDKL2IDXNSLXAJGPNANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Okay. Thanks. I'll try with this file. BTW, I noticed that this file is smaller then the previous one. The previous dimension was 264x201 but this one has 161x109. Are you sure that the file covers the bigger area than the previous one.

[uturuncoglu]

@pvelissariou1 I think there is a resolution difference but I am not sure why. The new data resolution is 0.5 deg. but previous was 0.2 deg. Do you know why?

[pvelissariou1]

@Ufuk Turuncoglu @.***> I downloaded the 0.5 degree data. Let me check the availability of the 0.2 degree data.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Apr 10, 2023 at 2:58 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think there is a resolution difference but I am not sure why. The new data resolution is 0.5 deg. but previous was 0.2 deg. Do you know why?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1502244983, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2YKYCDM6DQOP5A7YTXARQ6ZANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[pvelissariou1]

@Ufuk Turuncoglu @.**> Please get the data: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/Florence_CFS-0.2x0.2.nc The spatial resolution is 0.2 degrees and the temporal resolution is 1 hr* (don't forget to modify fort.15 to reflect this).

Does the spatial resolution affect how CMEPS does the interpolation?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Apr 10, 2023 at 8:15 PM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

@Ufuk Turuncoglu @.***> I downloaded the 0.5 degree data. Let me check the availability of the 0.2 degree data.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Apr 10, 2023 at 2:58 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think there is a resolution difference but I am not sure why. The new data resolution is 0.5 deg. but previous was 0.2 deg. Do you know why?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1502244983, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2YKYCDM6DQOP5A7YTXARQ6ZANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Could you let me know the contact person/s for the ADCIRC cap development. If you know their GitHub name that would be great and we could interact through the issue page. Thanks.

[uturuncoglu]

@pvelissariou1 BTW, 0.2 deg. data solve the issue. Thanks for your help.

[pvelissariou1]

@Ufuk Turuncoglu @.***> I sent an email to ADCIRC folks to supply us with the info you asked for. Waiting for their response.

For the 0.2 deg. data I made sure that the atm. grid overlaps the ADCIRC mesh at least by 2.0 degrees at each side. For your information, the atm. pressure in the 0.2 deg. data is not the mean sea level converted pressure (the models expect) but rather the surface pressure (sea level or ground surface) which, in our case is ok as we go inland up to 10-m above mean sea level.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Apr 21, 2023 at 12:49 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 BTW, 0.2 deg. data solve the issue. Thanks for your help.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1517294959, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6B3O6BANASUXXWOS3XCINVXANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[pvelissariou1]

@Ufuk Turuncoglu @.***> The atmospheric data file is in orion at: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/Wind_Pressure_HWRF_Florence_ExtendedSmooth.nc This is a high resolution product, file size is ~23GB. Please focus your attention over ocean (not land/rivers), HWRF does not produce good results over land.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Apr 21, 2023 at 1:08 PM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

@Ufuk Turuncoglu @.***> I sent an email to ADCIRC folks to supply us with the info you asked for. Waiting for their response.

For the 0.2 deg. data I made sure that the atm. grid overlaps the ADCIRC mesh at least by 2.0 degrees at each side. For your information, the atm. pressure in the 0.2 deg. data is not the mean sea level converted pressure (the models expect) but rather the surface pressure (sea level or ground surface) which, in our case is ok as we go inland up to 10-m above mean sea level.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Apr 21, 2023 at 12:49 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 BTW, 0.2 deg. data solve the issue. Thanks for your help.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1517294959, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6B3O6BANASUXXWOS3XCINVXANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 This data does not cover entire simulation for florence_hsofs.atm2adc configuration. The configuration starts on 2018-09-07 but data starts from 2018-09-09 06:00:00. So, simulation fails. Any idea how this works in ATMMESH. Maybe it is marked with another dataset.

[uturuncoglu]

@dwirasae @pvelissariou1 @saeed-moghimi-noaa As I see from the code, the ADCRIC cap uses following subroutine - extract_parallel_data_from_mesh_orig() to collect required information for mesh creation,

https://github.com/uturuncoglu/adcirc/blob/7cf14e22b08a978c7530828cc74436a573a6f3bf/thirdparty/nuopc/adc_mod.F90#L256

This seems to read files in PE directories. Anyway, I'll follow the same approach at this point and once we have working cap with CMEPS. We could modify it to use new method.

[pvelissariou1]

@Ufuk Turuncoglu @.***> I uploaded to orion in /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data the file files_for_HWRF_Florence_ExtendedSmooth.tar.gz The archive contains the proper fort.67.nc file for the HWRF atm. data you are using. I included the updated fort.15 and model_configure files as well. Just replace the old files with the new ones and re-run the case. Hopefully, this will work. Please, let me know how it goes.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, May 1, 2023 at 12:22 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 This data does not cover entire simulation for florence_hsofs.atm2adc configuration. The configuration starts on 2018-09-07 but data starts from 2018-09-09 06:00:00. So, simulation fails. Any idea how this works in ATMMESH. Maybe it is marked with another dataset.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1529976563, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7HJ7ULYTFYWXML7GLXD7WLTANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

@pvelissariou1 Are we changing the dates in this case, rather then changing input files? Are you also plaining to update CoastalApp test suite to reflect this. I am asking because the test under ufs-coastal will be different from the test that is available under test suite.

[pvelissariou1]

@Ufuk Turuncoglu @.***> Please change the files in your copy of the testsuite so both have the same forcing and dates. Don't forget to edit the config.rc file with the new atmospheric input file.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, May 1, 2023 at 2:51 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Are we changing the dates in this case, rather then changing input files? Are you also plaining to update CoastalApp test suite to reflect this. I am asking because the test under ufs-coastal will be different from the test that is available under test suite.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/4#issuecomment-1530129462, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP22OJNMMNVXMREBL7LXEAH43ANCNFSM6AAAAAAWTS2PN4 . You are receiving this because you were mentioned.Message ID: @.***>

[uturuncoglu]

I have no config.rc since I am using cdeps for forcing not atmmesh. Anyway, let we run the case with those updated files. Thanks for your help.

[uturuncoglu]

@dwirasae It seems that fort.14 and fort.18 are not consistent in terms of used element ids. The node coordinates and element connection table is coming from fort.14 but rest of the information is from fort.18. When I try to find the element id in element connection to find the associated nodes to calculate element coordinates I could not since the element ids are not matching between fort.14 and fort.18 for the same PE. Maybe those files are consistent internally but not each other. If this is correct maybe we need to go with new method which can be seen in extract_parallel_data_from_mesh() call. Maybe I am missing something in here. Maybe one holds global and other have local ids and we need to do some conversation. Anyway, let me know what you think?

[dwirasae]

Element id in a local fort.14 files (i.e. PEXXXX/fort.14) are actually a local element id (but the node ids are global ). Ids listed in fort.18 (lines after NELG) correspond to a global element id. exact_parallel_data_from_mesh() should work fine; however, it is expensive since, if not mistaken, it search through a global partmesh.txt file.

[uturuncoglu]

@dwirasae Anyway, it seems that there is no way to map local ids to global ones. Right? If so, the only option is to go with exact_parallel_data_from_mesh(). I think it is not so important to be slow since it will be run only once in the initialization.

[uturuncoglu]

@dwirasae I am not sure but still there are some inconsistencies in this subroutine. Some variables such as node coordinates and element connections are coming from ADCIRC internal data structures and others from files. Again, some of them local and others are global. So, I think we need to find a place that we could find all those information in a consistent manner. Is there any documentation that have detailed information about structure of fort.14 (I think I could find it in the wiki), fort.18 and also global parmesh.txt files? Also having some information about internal ADCIRC variables that are related with the mesh would be great.

[uturuncoglu]

@dwirasae I think also node ids returned from fort.14 are local and used to construct ElConnect. In the read14femesh() call n1, n2, and n3 (actual read values) are converted to local using the call like find(node_dict_tmp, n1). So, they are not the actual values read from the file. Maybe I am misinterpreting in here. Anyway, I think I started to understand the issue. Let me work on it and see what happens.

[dwirasae]

@uturuncoglu. A link below contains information of (a global) fort.14 :

https://adcirc.org/home/documentation/users-manual-v52/input-file-descriptions/adcirc-grid-and-boundary-information-file-fort-14

To my knowledge, there is no document for fort.18 and partmesh.txt.

• In a nutshell, a number in the i-th line of partmesh.txt corresponds to a PE to which the i-th node belong.

• fort.18 basically contain a local-to-global map and massage tables. Information below might be useful.

  NELG NELG NELL NELL # NELG - total no elements in the mesh # NELL - total no eleement in the local subdomain DO I = 1, NELL GELID(I) # Global element ID of the i-th local element. A negative value indicate a ghost element. END DO NBODG NNG NNLT NLLR # NNG - total no nodes in the mesh #NNLT - total no of node in the local subdomain (residet + ghost nodes) DO I = 1, NNLT GNID(I) # Global node ID of the i-th local nodes END DO

  NFLUX # Flux boundary NETA # OPEN boundatry NSTAE # No. elevation station (to which the time series of elevation are recored) NSTAV # No. current statiom NSTAM # No. meteorological station RES NODE # Massage table ... ... .... ....

[dwirasae]

@uturuncoglu. Yes, the node ids return from read14femesh() are local. read14femsh() use a function find( , ) to decode a hash key (this feature was introduced in v54). To find the coordinates of the elements, one could do something like

do i1 = 1, the_data%NumEl, 1

             xc(1,i1)  = xc(1,i1) +  sum( vk(1, etov(:, i1) ) )/3.0  ;
             xc(2,i1) =  xc(2.i1) +  sum( vk(2, etov(:, i1) ))/3.0   ;

end do

[uturuncoglu]

@dwirasae Thanks for your suggestion and help. The element coordinates are fine now but I have issue with filtering nodes that is associated with the non-ghost elements. Is there any easy way to access non-ghost elements and their associated nodes. At this point I am developing a new subroutine that takes the the_data user defined data structure and modify it by eliminating ghost elements and keeping only associated nodes but maybe there is an easy way that I don't know. Anyway, please let me know if you have any additional information about it.

[dwirasae]

@uturuncoglu. The way you did is probably the best way to extract the nodes associated with non-ghost elements. If you just want the non-ghost nodes, you can get them from the_data%owned_to_present_nodes.

[uturuncoglu]

@dwirasae I think having non-ghost nodes are fine since edge elements could have ghost nodes. I think I have issue with mapping global node ids found in the element connection array to the local node ids. Is there any way to access that mapping? Can I assume the the global node ids in the element connections are in a same order in the local node ids?

[saeed-moghimi-noaa]

Hi @uturuncoglu

I am using fort 80 to plot domain decomposition using this code. I am not sure if this could be helpful to you

def ReadFort80(dir):
    """
    Read fort.80 file for domain decomposition information

    return a dictionary including:
    IMAP_EL_LG
    IMAP_NOD_LG
    IMAP_NOD_GL (negative values for not owned elements)

    """
    fdata = open( dir + '/fort.80' ,  'r')
    while True:
    #for  line in fdata.readlines():
        line = fdata.readline()
        if 'Number of processors'     in line: nproc = int(line.split()[0]) 
        if 'Total # elements & nodes' in line: 
            nelem = int(line.split()[0])
            nnode = int(line.split()[1])          
        if 'NWLON, NWLAT'             in line:
            print (line)
            break

    #allocate
    IMAP_NOD_LG  = []
    IMAP_NOD_GL   = np.zeros((nnode,3),dtype=np.int)
    IMAP_EL_LG   = []
    #IMAP_STAE_LG = []
    #IMAP_STAV_LG = []
    #IMAP_STAC_LG = []
    #IMAP_STAM_LG = []
    pe_all        = []
    print ('[info:] read nodes local2global')
    for inp in  range( nproc ):
        line1       = fdata.readline()
        #print line1
        pe          = int(line1.split()[0])
        nnodp       = int(line1.split()[1])                
        nod_res_tot = int(line1.split()[2])                
        pe_all.append(pe)
        tmpa = np.array([])
        proc_read = True
        while proc_read:
           line1 = fdata.readline()
           tmp = np.array([int(v) for v in line1.split()])
           tmpa = np.r_[tmpa,tmp]
           if len(tmpa) == nnodp:
               IMAP_NOD_LG.append(tmpa)
               proc_read = False

    print ('[info:] read nodes local2global')
    line1       = fdata.readline()
    #print line1
    for il in range(nnode):
        line1       = fdata.readline()
        node_globa = int(line1.split()[0])
        pe         = int(line1.split()[1])                
        node_local = int(line1.split()[2]) 
        IMAP_NOD_GL[il,:] = node_globa , pe ,  node_local

    print ('[info:] read elements local2global')
    for inp in  range( nproc ):
        line1       = fdata.readline()
        #print line1
        pe          = int(line1.split()[0])   # pe number
        nelmp       = int(line1.split()[1])   # element on pe             

        pe_all.append(pe)
        tmpa = np.array([])
        proc_read = True
        while proc_read:
           line1 = fdata.readline()
           tmp = np.array([int(v) for v in line1.split()])
           tmpa = np.r_[tmpa,tmp]
           if len(tmpa) == nelmp:
               IMAP_EL_LG.append(tmpa)
               proc_read = False

    fdata.close() 
    return (dict (nproc = nproc, nelem = nelem, nnode = nnode ,
                  IMAP_EL_LG  = np.array(IMAP_EL_LG) ,
                  IMAP_NOD_LG = np.array(IMAP_NOD_LG),
                  IMAP_NOD_GL = np.array(IMAP_NOD_GL) )) 

def plot_domain_decomp(run_dir):
    import pynmd.models.adcirc.post as adcp 
    import pynmd.plotting.plot_settings as ps
    import matplotlib.pyplot as plt
    decomps = adcp.ReadFort80(run_dir+'/')
    nproc = decomps['nproc']
    IMAP_NOD_LG = decomps['IMAP_NOD_LG'] 
    IMAP_EL_LG  = decomps['IMAP_EL_LG'] 
    IMAP_NOD_GL = decomps['IMAP_NOD_GL']
    x,y,tri = adcp.ReadTri  (run_dir+'/')

    fig = plt.figure(figsize=(5,5))
    ax  =  fig.add_subplot(111)
    xp  = []
    yp  = []
    cc = 100 * ps.colors
    mm = 100 * ps.marker
    for ip in range(nproc):
        ind = np.where(IMAP_NOD_GL[:,1] == ip)
        ind = ind[::10]
        ax.scatter(tri.x[ind],tri.y[ind], c = cc[ip],
            s = 0.2 , edgecolors = 'None', marker = mm[ip])
    fig.savefig(run_dir + '/0decomp.png',dpi=450)
    plt.close()
    #print ' > Finished plotting domain decomposition'
    txt1 = ' > Finished plotting domain decomposition ..'
    logf(txt1,log_file)          

[saeed-moghimi-noaa]

See here some of my very old tools:

https://github.com/moghimis/pynmd/blob/master/models/adcirc/post/adcirc_post.py

[dwirasae]

@uturuncoglu. Sorry for a late response. The node ids in the_data%Elconnect(3(i-1):3i) are the local node ids of the ith local element.

[uturuncoglu]

@dwirasae @saeed-moghimi-noaa Thanks both of you. I am trying to build custom ESMF version to see which node or element causing the issue in the ESMF side. So, this might help me to find the issue in my code. I'll update you about it.

[uturuncoglu]

@dwirasae Do you have any information about having negative element ids in the fort.18 files? I was thinking the negative element ids indicate the ghost elements but I think it is something different. Could you clarify it for me? I create element mask by looking its nodes are local or not and that mask also marked some elements with negative id as local element. Any idea?

[uturuncoglu]

@dwirasae As I see there are some elements with negative id (means ghost) but all nodes are resident. I am expecting to have positive element id for those elements. Do you have any explanation about it?

[uturuncoglu]

@pvelissariou1 i was thinking that I solved the issue related to the ADCIRC mesh to make it compatible with CMEPS but apparently not. I have still same issue. So, it might be nice to have a call with ADCIRC developers soon. I still could not talk with Bob (from ESMF team) since he is busy with some other works but I will talk with him next week. So, we might wait for him and then talk with ADCIRC developers later. Anyway, let me know what you think?