search

oceanmodeling / ufs-weather-model

This repo is forked from ufs-weather-model, and contains the model code and external links needed to build the UFS coastal model executable and model components, including the ROMS, FVCOM, ADCIRC and SCHISM plus WaveWatch III model components.
https://github.com/oceanmodeling/ufs-coastal-app
Other
5 stars 5 forks source link

SCHISM integration #2

Closed uturuncoglu closed 1 week ago

uturuncoglu commented 1 year ago

@platipodium @saeed-moghimi-noaa @pvelissariou1 I am creating this issue to communicate the SCHISM related developments. I am planing to do the same for each component. So, we could focus specific component in each issue. Anyway here is my update about the ongoing development regarding to SCHISM:

uturuncoglu commented 1 year ago

@josephzhang8 Okay. I am not sure what was the issue that force you to get rid of element based partitioning but If we need to discuss it more we could also invite Bob.

josephzhang8 commented 1 year ago

Sounds good. Bob sent us the following ESMF online manual that showed an example of the partitioning approach. We might misunderstand but we got rid of the errors once we changed the code to be node based. Happy to correct that.

http://earthsystemmodeling.org/docs/nightly/develop/ESMF_refdoc/node5.html#SECTION050103400000000000000

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Tuesday, April 4, 2023 5:14 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 Okay. I am not sure what was the issue that force you to get rid of element based partitioning but If we need to discuss it more we could also invite Bob.

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1496614393, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ6PNP5CEU6MSOBR5LLW7SFK3ANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 Okay. Let me check with Bob and get back to you. Probably I could ask him in tomorrow's ESMF core meeting.

josephzhang8 commented 1 year ago

Besides that web site we studied ADCIRC cap and found that they used node based partitioning and it worked. So we were under the impression that ESMF only accepts this type. Time to revisit

Sent from my iPhone

On Apr 4, 2023, at 5:27 PM, Ufuk Turunçoğlu @.***> wrote:

 [EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 Okay. Let me check with Bob and get back to you. Probably I could ask him in tomorrow's ESMF core meeting.

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1496626850, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ4IRPWLYVFSHL3L2FDW7SG2VANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

New test for coupled case is added to https://github.com/oceanmodeling/ufs-coastal/tree/feature/coastal_app). It is coastal_ike_shinnecock_atm2sch and uses CFSR data. I have also configuration that uses hourly ERA5 dataset but I did not add it as RT.

josephzhang8 commented 1 year ago

Dear Joseph and Ufuk

Carsten can resurrect the code using elem based partitioning. As I explained, this is the native partitioning used by SCHISM, and we had to change it at the request of Robert O. to get the cap

1) As far as I can remember, we started the elem based partitioning (as that one is native to SCHISM) in the cap, but abandoned it (never got it to work) after consulting Bob. @Ufuk where can we find an example of a cap doing node-based partitioning?

2) Let’s think this through. Why should a model be forced to be based on either node/element-based partitioning? Wouldn’t it be the job of the coupler to just deal with whatever partitioning comes?

Helmholtz-Zentrum hereon GmbH Max-Planck-Straße 1 I 21502 Geesthacht I Deutschland/Germany

Geschäftsführung I Board of Management: Prof. Dr. Matthias Rehahn Vorsitzender des Aufsichtsrates I Chairman of the Supervisory Board: Ministerialdirigent Engelbert Beyer Amtsgericht Lübeck HRB 285 GE (Register Court) Internet: www.hereon.de https://www.hereon.de

uturuncoglu commented 1 year ago

@josephzhang8 @platipodium

josephzhang8 commented 1 year ago

@Ufuk:

I found a copy of the old code that attempts to use elem based partitioning; see the routine 'addSchismMesh' inside: http://ccrm.vims.edu/yinglong/TMP/schism_esmf_util.F90http://ccrm.vims.edu/yinglong/TMP/schism_esmf_util.F90

We did not get this to work at all, so we contacted Bob and others and finally changed it to node based. I hope u can compere the logic there with the new code ( the routine was renamed as 'SCHISM_MeshCreate'). Thx.

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Wednesday, April 5, 2023 12:53 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 @platipodiumhttps://github.com/platipodium

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1497822419, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ5Q45UYPPZ77GNDGG3W7WPORANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 Thanks. I'll look at it.

josephzhang8 commented 1 year ago

As I said the old code did not work and ESMF gave us errors. Carsten and I would be happy to have a work session with you to sort these issues out.

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Thursday, April 6, 2023 5:04 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 Thanks. I'll look at it.

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499629314, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ3FJHEM5OIE3UQXOBDW74VVBANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 Okay. I think I need to first put this code (maybe slightly modified version) and make a new option in the cap for node vs. element. Then, I could test it with CMEPS under recent version of UFS with recent version of ESMF. If I have any issue, I think it would be nice to discuss it. We could also add Bob to the call if it is required.

pvelissariou1 commented 1 year ago

Dear Joseph and Carsten

Let me find a time for that next week. Possible dates: M (04/10), T(04/11), W(04/12, before 1:00pm ET), Please let me know your availability.

Thank you Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 4:10 PM Joseph Zhang @.***> wrote:

As I said the old code did not work and ESMF gave us errors. Carsten and I would be happy to have a work session with you to sort these issues out.

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Thursday, April 6, 2023 5:04 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 Thanks. I'll look at it.

- Reply to this email directly, view it on GitHub< https://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499629314>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AFBKNZ3FJHEM5OIE3UQXOBDW74VVBANCNFSM6AAAAAAWNY76MU

. You are receiving this because you were mentioned.Message ID: @.**@.>>

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499635229, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP3AFDRXROHKX3H4LTDW74WLLANCNFSM6AAAAAAWNY76MU . You are receiving this because you were mentioned.Message ID: @.***>

josephzhang8 commented 1 year ago

We really appreciate your Ufuk! Standing by...

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Thursday, April 6, 2023 5:15 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 Okay. I think I need to first put this code (maybe slightly modified version) and make a new option in the cap for node vs. element. Then, I could test it with CMEPS under recent version of UFS with recent version of ESMF. If I have any issue, I think it would be nice to discuss it. We could also add Bob to the call if it is required.

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499639039, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ6PUPZRENJLG7ZS45TW74W4DANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 It is impossible without your help. The unstructured models makes them hard to debug since you can see the output easily. I still need to find a way to make simple plots since the tools that I tried did not work well. At this point, I am working on multiple projects (one another will come soon) and I am not sure I have an initial implementation until next week but I'll try. So, T(04/11) morning (around 9:30am MT) works for me if you want to discuss.

josephzhang8 commented 1 year ago

9:30am (MT) on 4/11 works for me. I'll cut short another meeting.

Carsten, Takis: how about you?

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Thursday, April 6, 2023 5:48 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 It is impossible without your help. The unstructured models makes them hard to debug since you can see the output easily. I still need to find a way to make simple plots since the tools that I tried did not work well. At this point, I am working on multiple projects (one another will come soon) and I am not sure I have an initial implementation until next week but I'll try. So, T(04/11) morning (around 9:30am MT) works for me if you want to discuss.

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499666159, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ7DC2C5BJJ3JKYC2T3W7422HANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

pvelissariou1 commented 1 year ago

Dear Carsten, sorry I didn't consider Easter, ours is the week after yours.

From our side, I see the following open time blocks: W (04/12) 9:00 am -- 10:00 am ET, 10:45 am -- 11:15 am (ET) Th (04/13) 10:45 am -- 12:00 pm ET, 3:30 pm -- 4:30 pm (ET) M (04/16) 10:45 am -- 4:00 pm (ET) T (04/18) 9:00 am -- 11:00 am (ET) 2:05 pm -- 4:00pm (ET) We just need to reserve a 1-hr for the discussion.

Thank you Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 6, 2023 at 5:14 PM Carsten Lemmen @.***> wrote:

9:30am (MT) on 4/11 works for me. I'll cut short another meeting. Carsten, Takis: how about you? … <#m-3680091301875784608>

I can make it work. But cannot prepare as I am on an Easter holiday including Monday.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499689581, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2MEYT2VPPDNQKLRJLW7454ZANCNFSM6AAAAAAWNY76MU . You are receiving this because you were mentioned.Message ID: @.***>

josephzhang8 commented 1 year ago

I’ll defer to Ufuk to accommodate his availability.

Regards,

Joseph Zhang (804)684 7595 (office) SCHISM web: http://ccrm.vims.edu/schism/ From: Panagiotis Velissariou @.> Sent: Thursday, April 6, 2023 8:02 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

Dear Carsten, sorry I didn't consider Easter, ours is the week after yours.

From our side, I see the following open time blocks: W (04/12) 9:00 am -- 10:00 am ET, 10:45 am -- 11:15 am (ET) Th (04/13) 10:45 am -- 12:00 pm ET, 3:30 pm -- 4:30 pm (ET) M (04/16) 10:45 am -- 4:00 pm (ET) T (04/18) 9:00 am -- 11:00 am (ET) 2:05 pm -- 4:00pm (ET) We just need to reserve a 1-hr for the discussion.

Thank you Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.<mailto:@.>

On Thu, Apr 6, 2023 at 5:14 PM Carsten Lemmen @.<mailto:@.>> wrote:

9:30am (MT) on 4/11 works for me. I'll cut short another meeting. Carsten, Takis: how about you? … <#m-3680091301875784608>

I can make it work. But cannot prepare as I am on an Easter holiday including Monday.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499689581, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2MEYT2VPPDNQKLRJLW7454ZANCNFSM6AAAAAAWNY76MU . You are receiving this because you were mentioned.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1499775109, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZYHULNZKRXOFN2VLADW75KORANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

Hi All, following works for me,

uturuncoglu commented 1 year ago

@josephzhang8 @platipodium I made couple of changes in the the cap to reflect element based partitioning and I think it is working under CMEPS now. I create mesh (and also distgrids) without ghost nodes because it was creating issue when weight matrix is calculated in the ESMF side for the redistribution and interpolation. I still need to check the results to be sure that it is working but my initial comparison with the connector only configurations show identical results except couple of variables which might be expected since CMEPS is trying to use different interpolation types. I'll check this again. I'll also push the changes to my branch soon and let you know. So, you could review the new version of cap and maybe test in your side.

I have one minor question,

josephzhang8 commented 1 year ago

This is great news; thank you s much Ufuk! I can add you to the schism repos if it's more convenient for you this way. Let me know.

RE: oceanmask, I guess you meant areas with water? Yes there will be wetting and drying at the water-land interface. The flags for node/side/elem are idry(:), and idrys,e

Regards,

Joseph Zhang (804)684 7595 (office) SCHISM web: http://ccrm.vims.edu/schism/ From: Ufuk Turunçoğlu @.> Sent: Tuesday, April 11, 2023 6:10 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 @platipodiumhttps://github.com/platipodium I made couple of changes in the the cap to reflect element based partitioning and I think it is working under CMEPS now. I create mesh (and also distgrids) without ghost nodes because it was creating issue when weight matrix is calculated in the ESMF side for the redistribution and interpolation. I still need to check the results to be sure that it is working but my initial comparison with the connector only configurations show identical results except couple of variables which might be expected since CMEPS is trying to use different interpolation types. I'll check this again. I'll also push the changes to my branch soon and let you know. So, you could review the new version of cap and maybe test in your side.

I have one minor question,

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1504191713, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ2BPD3VDXXSHBPFKTLXAXJFJANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 I am seeing 0 and 3 in the mask values. In the code I found that 0 wet and 1 dry. If you don't mind could you clarify it. I need to send mask information to CMEPS in a standard way.

uturuncoglu commented 1 year ago

@josephzhang8 @platipodium Initial implementation is in here,

https://github.com/schism-dev/schism-esmf/compare/master...uturuncoglu:schism-esmf:feature/coastal_app?expand=1

I renamed old mesh create routine as SCHISM_MeshCreate2 in the code. We could discuss about it on the call. I could not be sure about keeping it or just using new routine.

PS: I also moved the work related with new I/O support to a new branch (feature/newio). We see this isolated for now since new is not the priority at this point and might require more development.

uturuncoglu commented 1 year ago

BTW, I also did not add fields related to ugrid from the old code but we could add them if you want.

platipodium commented 1 year ago

BTW, I also did not add fields related to ugrid from the old code but we could add them if you want.

I would like this very much! Metadata is key. Please create an issue on schism-esmf

platipodium commented 1 year ago

I renamed old mesh create routine as SCHISM_MeshCreate2 in the code. We could discuss about it on the call. I could not be sure about keeping it or just using new routine.

I'd prefer semantic naming, so SCHISM_MeshCreateNode and SCHISM_MeshCreateElement. And let's keep (and test) both routines. In the end, this should be a build-time configuration argument to SCHISM-ESMF

platipodium commented 1 year ago
  • Is ocean mask is changing in time. CMEPS requires ocean_mask or (So_mask) to do some internal calculation etc. and used in interpolation routines I think.

Thanks for adding the ocean_mask field. I have not seen yet how this field is updated in the component's mesh, please create an issue on SCHISM_ESMF

josephzhang8 commented 1 year ago

@Ufuk: I don't see how idry can be 3 as you mentioned. It'd be only 1 or 0. The check for dry in the code is usually if(idry==1). We can discuss this at tomorrow's meeting. Thx.

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Wednesday, April 12, 2023 2:22 AM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

BTW, I also did not add fields related to ugrid from the old code but we could add them if you want.

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1504727258, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ7TJKRLWKXIHQAJWGTXAZCY3ANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 Does model output idry? If so maybe we could check it. I could also put some print statement to code to print out min and max values.

josephzhang8 commented 1 year ago

Yes check out2d*.nc ('dryFlagNode')

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Ufuk Turunçoğlu @.> Sent: Wednesday, April 12, 2023 4:26 PM To: oceanmodeling/ufs-coastal @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] SCHISM integration (Issue #2)

[EXTERNAL to VIMS received message]

@josephzhang8https://github.com/josephzhang8 Is model outputs idry? If so maybe we could check it. I could also put some print statement to code to print out min and max values.

- Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1505881011, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFBKNZ6GTGTYS7TJMDD4KPTXA4FUVANCNFSM6AAAAAAWNY76MU. You are receiving this because you were mentioned.Message ID: @.**@.>>

uturuncoglu commented 1 year ago

@josephzhang8 @platipodium I modified to code to be able to use both mesh generation routine. I introduced a new config option like meshloc and it could be set as node (default) or element (for coupling with CMEPS). I also compare the results of mediator configuration with connector only one and I have some answer changes (at this point just checking with NCAR's cprnctool). Still I can't visualize the results. What is the best way for old output configuration? I have still build issue with Util folder, so I have no tools to combine outputs but I'll look at it. Maybe the best way to use VisIT but I am not sure how hard to install the plugin and VisIT under Mac OS and I am not sure if it will work with uncombined file or not. I also updated my branch, so you could see the last version of code. I suppose it still need work.

uturuncoglu commented 1 year ago

@josephzhang8 Okay. I fixed the mask issue. Now I could see only 0-1. Thanks for your help.

uturuncoglu commented 1 year ago

@platipodium @josephzhang8 ii think I know why answers are different. The atmospheric forcings coming from CDEPS are all zero when mediator sends them to SCHISM. CDEPS export is fine. I am still investigating it but I think it is related with the masking defined in CMEPS side and it zero out all the information coming from CDEPS. I need to fix those masks for coastal coupling mode (I introduced for ufs-coastal) in CMEPS.

platipodium commented 1 year ago

See visualisation tools at https://codebase.helmholtz.cloud/mussel/mossco-scripts/-/tree/main/schism

platipodium commented 1 year ago

VIsIT package for linux available from @saeed-moghimi-noaa (please send around)

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.**> The "ViSiT" package on orion* is at: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/visit-saeed.tar

@Carsten Lemmen @.> , @Y. Joseph Zhang @.> The "ViSiT" package on frontera is at: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/visit-saeed.tar

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Apr 13, 2023 at 10:05 AM Carsten Lemmen @.***> wrote:

VIsIT package for linux available from Saeed (please send around)

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1507135780, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6K6QC6RWYL24DF5W3XBAI3JANCNFSM6AAAAAAWNY76MU . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@saeed-moghimi-noaa I am trying to use thalassa. Do I need to combine files for it? Is there any example that I could with old I/O and also multiple file for each PE?

saeed-moghimi-noaa commented 1 year ago

Please open an issue with your question at the thalassa github repo. The lead developer is on vacation, he or his colleague may get to you soon. Thanks, -Saeed

uturuncoglu commented 1 year ago

@saeed-moghimi-noaa thanks.

platipodium commented 1 year ago

@saeed-moghimi-noaa I am trying to use thalassa. Do I need to combine files for it? Is there any example that I could with old I/O and also multiple file for each PE?

i also tried Thalassa yesterday. Got it to work on my M1 Mac almost seamlessly (had to replace cartopy with a different version). Simple OLDIO unprocessed output from SCHISM did not work.... tell me about progress you make.

uturuncoglu commented 1 year ago

@pvelissariou1 Just for testing, I am trying to run ike_shinnecock.atm2sch case from CoastalApp-testsuite but when I try to build ATMMESH+SCHISM on Orion using following command,

./build.sh --compiler intel --platform orion --component "atmesh schism" -y

I am getting following error from build

make[1]: Entering directory `/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/SCHISM/schism-esmf'
src/include/Rules.mk:28: *** Your ESMF version 8.0 is too old. At least ESMF 8.1 is required.  Stop.

I think SCHISM required at list ESMF 8.1. I wonder if you try to run this test on Orion before. I could easily build ESMF in here but I just want to ask you first.

uturuncoglu commented 1 year ago

@platipodium I have just opened an issue in their repo. In parallel, I am also trying to build VisIT under Mac but no luck yet. I'll update you if I find anything.

platipodium commented 1 year ago

make[1]: Entering directory `/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/SCHISM/schism-esmf' src/include/Rules.mk:28: *** Your ESMF version 8.0 is too old. At least ESMF 8.1 is required. Stop.



I think SCHISM required at list ESMF 8.1. I wonder if you try to run this test on Orion before. I could easily build ESMF in here but I just want to ask you first.

Yes, this is correct. There are breaking API changes between 8.0 and 8.1, I believe most of them with ESMF_Info vs ESMF_Attribute (deprecated), but also others.

saeed-moghimi-noaa commented 1 year ago

Hi @platipodium

I tried this notebook. Worked for their example. The code is very simple. it may complain for specific dimension name or size. You need to skip it when xarray wants to read header some where in normalization function. They try to normalize var and dim names to have a uniform interface with various unstructured models.

See this one:

https://github.com/ec-jrc/Thalassa/blob/master/notebooks/API.ipynb

#with

mkdir ./data
wget -O data/dataset.nc https://static.techrad.eu/thalassa/dataset.nc
pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> On orion please modify the envmodules_intel.orion file as: from: module load esmf/8.0.0 to: module load esmf/8.1.1

esmf 8.1.1 is already installed, I had requested that for SCHISM at that time.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Apr 14, 2023 at 8:12 AM Saeed Moghimi @.***> wrote:

Hi @platipodium https://github.com/platipodium

I tried this notebook. Worked for their example. The code is very simple. it may complain for specific dimension name or size. You need to skip it some where in normalization function. They try to normalize var and dim names to have a uniform interface with various unstructured models.

See this one:

https://github.com/ec-jrc/Thalassa/blob/master/notebooks/API.ipynb

with

mkdir ./data wget -O data/dataset.nc https://static.techrad.eu/thalassa/dataset.nc

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/ufs-coastal/issues/2#issuecomment-1508486523, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7OGIMER3RFSNJGA23XBFEMTANCNFSM6AAAAAAWNY76MU . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. I changed it but I am still getting error like following,

make[1]: Entering directory `/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/SCHISM/schism-esmf'
src/include/Rules.mk:142: *** ESMF Makefile snippet variable not defined.  Stop.

I'll check what is missing.

uturuncoglu commented 1 year ago

Okay. I am getting following when I try to load esmf/8.1.1

Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "esmf/8.1.1"
   Try: "module spider esmf/8.1.1" to see how to load the module(s).

I think it is available through hpc-stack but not sure. Any idea?

uturuncoglu commented 1 year ago

@pvelissariou1 I think I need to update compiler to load that module.

uturuncoglu commented 1 year ago

@pvelissariou1 I am not sure but I have still issue when I try to build CoastalApp. I used following modules,

module purge
module load cmake
module load intel/2019.5 impi/2019.6
module load hdf5/1.10.5-parallel pnetcdf/1.12.0 netcdf/4.7.2-parallel
module load esmf/8.1.1

but I am getting following. Maybe there are some interface change in ESMF side that leads to error.

ifort: command line warning #10121: overriding '-mkl=sequential' with '-mkl'
MAIN_NEMS.o: In function `MAIN__':
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/MAIN_NEMS.tmp.f90:377: undefined reference to `esmf_timemod_mp_esmf_timeset_'
module_MEDIATOR.o: In function `module_mediator_mp_initializep0_':
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:920: undefined reference to `esmf_attributemod_mp_esmf_gridcompattgetchar_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:945: undefined reference to `esmf_attributemod_mp_esmf_gridcompattgetchar_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:956: undefined reference to `esmf_attributemod_mp_esmf_gridcompattgetchar_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:964: undefined reference to `esmf_attributemod_mp_esmf_gridcompattgetchar_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:972: undefined reference to `esmf_attributemod_mp_esmf_gridcompattgetchar_'
module_MEDIATOR.o:/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:980: more undefined references to `esmf_attributemod_mp_esmf_gridcompattgetchar_' follow
module_MEDIATOR.o: In function `module_mediator_mp_initializep0_':
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:1005: undefined reference to `esmf_timemod_mp_esmf_timeset_'
module_MEDIATOR.o: In function `module_mediator_mp_nuopcplus_updatetimestampf_':
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:9201: undefined reference to `esmf_attributemod_mp_esmf_fieldattset1di4_'
module_EARTH_GRID_COMP.o: In function `module_earth_grid_comp_mp_setmodelservices_':
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3332: undefined reference to `esmf_attributemod_mp_esmf_gridcompattgetchar_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3520: undefined reference to `wmesmfmd_mp_setservices_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3603: undefined reference to `adc_cap_mp_setservices_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3631: undefined reference to `pahm_cap_mp_setservices_'
module_EARTH_GRID_COMP.o: In function `module_earth_grid_comp_mp_modifycpllists_':
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3835: undefined reference to `esmf_attributemod_mp_esmf_cplcompattgetchar_'
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/ATMESH_INSTALL/libatmesh_cap.a(atmesh_mod.o): In function `atmesh_mod_mp_read_atmesh_nc_':
atmesh_mod.F90:(.text+0x473): undefined reference to `esmf_timemod_mp_esmf_timeset_'
make[1]: *** [nems] Error 1
make[1]: Leaving directory `/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src'
make: *** [/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/exe/NEMS.x] Error 2

Anyway, I was plaining to test the existing model to see the atmmesh+schisim case is working or not but maybe it is not good idea. Somehow the data atmosphere fields are not passing in ufs-coastal. It could be related with configuration or maybe I am missing something else.

pvelissariou1 commented 1 year ago

Is ESMFMKFILE environment variable defined and is it pointing to correct esmf.mk file? We use esmf 8.1.1 on hera and other platforms without an issue.

On Friday, April 14, 2023, Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 I am not sure but I have still issue when I try to build CoastalApp. I used following modules,

module purge module load cmake module load intel/2019.5 impi/2019.6 module load hdf5/1.10.5-parallel pnetcdf/1.12.0 netcdf/4.7.2-parallel module load esmf/8.1.1

but I am getting following. Maybe there are some interface change in ESMF side that leads to error.

ifort: command line warning #10121: overriding '-mkl=sequential' with '-mkl' MAIN_NEMS.o: In function `MAIN__':

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/MAIN_NEMS.tmp.f90:377: undefined reference to esmf_timemod_mp_esmf_timeset_' module_MEDIATOR.o: In functionmodule_mediator_mpinitializep0':

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:920: undefined reference to `esmf_attributemod_mp_esmfgridcompattgetchar'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:945: undefined reference to `esmf_attributemod_mp_esmfgridcompattgetchar'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:956: undefined reference to `esmf_attributemod_mp_esmfgridcompattgetchar'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:964: undefined reference to `esmf_attributemod_mp_esmfgridcompattgetchar'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:972: undefined reference to `esmf_attributemod_mp_esmfgridcompattgetchar'

module_MEDIATOR.o:/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:980: more undefined references to esmf_attributemod_mp_esmf_gridcompattgetchar_' follow module_MEDIATOR.o: In functionmodule_mediator_mpinitializep0':

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:1005: undefined reference to esmf_timemod_mp_esmf_timeset_' module_MEDIATOR.o: In function module_mediator_mp_nuopcplusupdatetimestampf':

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_MEDIATOR.tmp.f90:9201: undefined reference to esmf_attributemod_mp_esmf_fieldattset1di4_' module_EARTH_GRID_COMP.o: In function module_earth_grid_comp_mpsetmodelservices':

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3332: undefined reference to `esmf_attributemod_mp_esmfgridcompattgetchar'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3520: undefined reference to `wmesmfmd_mpsetservices'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3603: undefined reference to `adc_cap_mpsetservices'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3631: undefined reference to pahm_cap_mp_setservices_' module_EARTH_GRID_COMP.o: In function module_earth_grid_comp_mpmodifycpllists':

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src/module_EARTH_GRID_COMP.tmp.f90:3835: undefined reference to `esmf_attributemod_mp_esmfcplcompattgetchar'

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/ATMESH_INSTALL/libatmesh_cap.a(atmesh_mod.o): In function atmesh_mod_mp_read_atmesh_nc_': atmesh_mod.F90:(.text+0x473): undefined reference to esmf_timemod_mp_esmftimeset' make[1]: [nems] Error 1 make[1]: Leaving directory `/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/src' make: [/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/NEMS/exe/NEMS.x] Error 2

Anyway, I was plaining to test the existing model to see the atmmesh+schisim case is working or not but maybe it is not good idea. Somehow the data atmosphere fields are not passing in ufs-coastal. It could be related with configuration or maybe I am missing something else.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci4.googleusercontent.com/proxy/vYo41RE_o_LCvIq18OJ_R1UkVySyWem1CaT_t9Zzq2h5kFB97dy5jZ1Jcoj6SQwJTPBLZGMEAcfORHp4m_MnmcxD2z4pr5s38u0HHKB5cbjt1fQQX4PMtKNC0dG_0w2jSW0XXo6s453sQRvTtXuQkyziRpopmc9K16GPvZD3rt4j3JH2SGRJUkt6VINAwEhT0tClNyiztCBiqyfVXGRnjdstfi_ussDaOd0CjJ2Qkzs0oXHqQA=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TP4IN2HD4247ZG3LGV3XBGL2PA5CNFSM6AAAAAAWNY76MWWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSZ6K3AG.gif>Message ID: @.***>

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. I could compile with ESMF 8.1.1 after cleaning the old build. am using following set of modules with new version of ESMF,

module purge
module load cmake
module load intel/2019.5 impi/2019.6
module load hdf5/1.10.6 netcdf/4.7.4 pnetcdf/1.12.1
module load esmf/8.1.1

I also need to add following environment variable to the model_run_slurm.job

export FI_PROVIDER=tcp

because this MPI version was giving error like unable to create a business card intel mpi. Anyway, with these ike_shinnecock.atm2sch configuration runs.