Error during compilation

[sarriegi]

Hello, so far I have been running CoastalApp in the MIT-supercloud cluster. Recently they changed the environment, so I compiled again the esmf library in the new environment. After that, I tried to compile again the CoastalApp, but I got this error, right at the beginning of the compilation process:

Do you know what the problem might be?

Angelos @pvelissariou1 @saeed-moghimi-noaa

[pvelissariou1]

Angelos, Are you sure you cloned CoastalApp recursively, maybe your ATMESH,WW3,ADCIRC are empty. Try to clone the application as usual: git clone --recurse-submodules https://github.com/noaa-ocs-modeling/CoastalApp.git

I built the application from scratch using: build.sh --component "atmesh ww3 adcirc" --compiler gnu --tp parmetis and everything compiled just fine.

[sarriegi]

Thank you for your response Takis! I am trying to compile a code that has worked in the past, so the folders are all there. Also, the compilation of ADCIRC and WW3 as standalone models inside the CoastalApp works fine. The only thing that changed between the time that I was able to compile and run the CoastalApp, was that they changed the system's compilers. And somehow this change has made the CoastalApp compilation crushing at the very beginning of the process.

Angelos

[sarriegi]

Note also, that I was trying with a previous version of CoastalApp. When I tried to compile the latest version, the compilation got through this initial stage, ADCIRC and WW3 were compiled successfully, but then just before the end of the compilation process, I got this error: Do you know what could be the reason for this error? @pvelissariou1

Angelos

[pvelissariou1]

Angelos,

What version of ESMF are you using to compile the application. Make sure before each compilation that you clean the system completely.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sat, Aug 26, 2023 at 12:28 AM sarriegi @.***> wrote:

Note also, that I was trying with a previous version of CoastalApp. When I tried to compile the latest version, the compilation got through this initial stage, ADCIRC and WW3 were compiled successfully, but then just before the end of the compilation process, I got this error: [image: image] https://user-images.githubusercontent.com/77630268/263441550-8ff3285b-c207-4b3f-a27c-0e4ce53c4960.png Do you know what could be the reason for this error? @pvelissariou1 https://github.com/pvelissariou1

Angelos

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1694173711, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2B4DBL53E6S7QRH5DXXGCPLANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[sarriegi]

Hello Takis, I am using version 8.0.1, the same version I was always using, which was working fine before they changed the compilers.

Angelos

[pvelissariou1]

What is the version of the compiler you are using? Make sure that the ESMFMKFILE variable points to the correct installation of ESMF. I have tested CoastalApp/ESMF with GNU and Intel compilers (up to 2022) without any issue.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sat, Aug 26, 2023 at 10:44 AM sarriegi @.***> wrote:

Hello Takis, I am using version 8.0.1, the same version I was always using, which was working fine before they changed the compilers.

Angelos

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1694391702, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6IK3FSASNWQMOPSY3XXIKVDANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[sarriegi]

I am using the new compilers that they installed in MIT Supercloud, intel 2023.1. With the previous compiler which was intel 2019.5, it was compiling fine.

Angelos

[pvelissariou1]

Could you please send me the full compile log from CoastalApp and possibly from ESMF? I cannot determine what is wrong with your compilations.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Aug 27, 2023 at 10:43 PM sarriegi @.***> wrote:

I am using the new compilers that they installed in MIT Supercloud, intel 2023.1. With the previous compiler which was intel 2019.5, it was compiling fine.

Angelos

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1694958095, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7VH7XKLEPRE3GOKGTXXQHURANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[sarriegi]

Hello Takis, I am attaching the coastalapp compilation and the esmf build log files. The error was this (for some reason was not printed on the log file: coastalapp_compile.log esmf_make.log

Thank you for looking into this!

Angelos

[pvelissariou1]

Angelos,

I'll try to look into this as soon as possible. I'll keep you posted

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Aug 28, 2023 at 12:34 PM sarriegi @.***> wrote:

Hello Takis, I am attaching the coastalapp compilation and the esmf build log files. The error was this (for some reason was not printed on the log file: [image: image] https://user-images.githubusercontent.com/77630268/263777704-2ae368d1-22d4-4380-8936-e1665d3b1b58.png coastalapp_compile.log https://github.com/noaa-ocs-modeling/CoastalApp/files/12456883/coastalapp_compile.log esmf_make.log https://github.com/noaa-ocs-modeling/CoastalApp/files/12456884/esmf_make.log

Thank you for looking into this!

Angelos

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1696080382, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4O4NLE65FFGUAQUUTXXTJDLANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[pvelissariou1]

Hi Angelos,

I didn't find any apparent issues in the compilations of ESMF/CoastalApp. (I am wondering though why the ESMF compilation is stored in an openmpi folder: /home/gridsan/apapandreou/esmf-ESMF_8_0_1_openmpi) So in the CoastalApp/NEMS/src/module_MEDIATOR.F90 we have the two code blocks:

!---------------------------

! validate all data by default
!---------------------------

if ESMF_VERSION_MAJOR >= 8

call NUOPC_SetTimestamp(NState_AtmExp, clock, rc=rc)

else

call NUOPC_UpdateTimestamp(NState_AtmExp, clock, rc=rc)

endif

if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
  line=__LINE__, file=__FILE__)) return  ! bail out

!---------------------------
! COLDSTART:
! invalidate all data on timestep 1
! invalidate SST on all timesteps
!---------------------------

if (coldstart) then

  if (is_local%wrap%fastcntr == 1) then

if ESMF_VERSION_MAJOR >= 8

    call NUOPC_SetTimestamp(NState_AtmExp, clock_invalidTimeStamp,

rc=rc)

else

    call NUOPC_UpdateTimestamp(NState_AtmExp, clock_invalidTimeStamp,

rc=rc)

endif

which means that NUOPC_SetTimestamp is called despite the fact that your ESMF version is greater than 8. Can you check the esmf.mk pointing to by the ESMFMKFILE env. variable for the variables: ESMF_VERSION_MAJOR, ESMF_VERSION_MINOR, ESMF_VERSION_REVISION to see if these variables are set correctly? Can you share this esmf.mk file?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Aug 28, 2023 at 3:38 PM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

Angelos,

I'll try to look into this as soon as possible. I'll keep you posted

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Aug 28, 2023 at 12:34 PM sarriegi @.***> wrote:

Hello Takis, I am attaching the coastalapp compilation and the esmf build log files. The error was this (for some reason was not printed on the log file: [image: image] https://user-images.githubusercontent.com/77630268/263777704-2ae368d1-22d4-4380-8936-e1665d3b1b58.png coastalapp_compile.log https://github.com/noaa-ocs-modeling/CoastalApp/files/12456883/coastalapp_compile.log esmf_make.log https://github.com/noaa-ocs-modeling/CoastalApp/files/12456884/esmf_make.log

Thank you for looking into this!

Angelos

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1696080382, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4O4NLE65FFGUAQUUTXXTJDLANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[sarriegi]

Thank you for looking into this Takis, I think that they are set correctly. I am attaching the esmf.mk file

Angelos esmf.txt

[pvelissariou1]

Angelos,

I see no issues with the esmf.mk file you shared. So, to reiterate: 1) you load the correct esmf module OR, you set the ESMFMKFILE variable before compiling CoastalApp. I see that in your CoastalApp.log you have: ESMFMKFILE = /home/gridsan/apapandreou/ESMFlib_openmpi_ang_intel_mitsupercloud_1/lib/libg/Linux.intel.64.intelmpi.default/ esmf.mk I assume that this is the file you shared with me. 2) You load all other required modules properly.

I am sure you do the above but I needed to verify this.

You may recompile your CoastalApp by cleaning first the app: 1) ./build.sh --compiler intel --component "atmesh adcirc ww3" --platform hatteras --clean 2 2) ./build.sh --compiler intel --component "atmesh adcirc ww3" --platform hatteras

I really do not see any issue in the compilation process and I don't understand why you are getting this error.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Aug 29, 2023 at 10:38 AM sarriegi @.***> wrote:

Thank you for looking into this Takis, I think that they are set correctly. I am attaching the esmf.mk file

Angelos esmf.txt https://github.com/noaa-ocs-modeling/CoastalApp/files/12466407/esmf.txt

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1697694314, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6PCVVEJKF4TUP24QTXXYEGJANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[sarriegi]

I don't understand either. I think I will move on by trying to run CoastalApp in a different cluster. Thank you again!

Angelos

[pvelissariou1]

Let me know how it goes.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Aug 29, 2023 at 12:22 PM sarriegi @.***> wrote:

I don't understand either. I think I will move on by trying to run CoastalApp in a different cluster. Thank you again!

Angelos

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1697852485, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2472D6APFNR4LATMLXXYQNNANCNFSM6AAAAAA37C63SI . You are receiving this because you were mentioned.Message ID: @.***>

[sarriegi]

Hi Takis, I was able to compile it after in MIT supercloud (by adding -DESMF_VERSION_MAJOR=8 to CPPFLAGS in NEMS/src/GNUmakefile) after all. However, when I try to run I get these errors: Have you ever seen errors like that? Do you have any idea of what the problem might be?

Angelos

[pvelissariou1]

@sarriegi , @saeed-moghimi-noaa Angelos it seems to me that in your cloud cluster the environment variables are not propagating properly. CoastalApp and its components during the compilation stage source and/or, include the file pointed to by the ESMFMKFILE variable which, contains the definitions of all the ESMF related variables. We never had any of these issues with ESMF. May be your bash login is not configured properly. You should never have to modify the NEMS/src/GNUmakefile for this.

Regarding the errors, I think these are coming from the library libucs which is part of the ucx package in linux. In the cluster run the command: rpm -q ucx ucx-devel (I guess the cluster uses some version of RedHat/CentOS) to see what version of the ucx package is installed in your system. I believe, you should have ucx >= 1.12 installed for things to work. I had this issue before and it was all due to the ucx library. I compiled a newer version of ucx in one of our HPC clusters and all these issues were resolved. See what you can find and let us know.

[sarriegi]

Thank you for this information Takis, this is the related information about Supercloud: Since it is 1.12 I assume this is not the problem. Also everything was working fine with the old version of the compiler. It's the new compiler that has messed things up. I think I will move on and try to run CoastalApp in the Unity cluster (UMass).

Angelos

[sarriegi]

Hello Takis, I compile CoastalApp in a new cluster. However, when I tried to run it I received these errors in on of the PET* files: Do you have any idea about what I could be doing wrong?

Angelos @pvelissariou1 @saeed-moghimi-noaa

[pvelissariou1]

Have you tried this configuration before?

[pvelissariou1]

@sarriegi are the atm. fields given on triangular mesh? Same mesh as the ocean model? Can you try to use: remapMethod=bilinear:extrapMethod=nearest_stod to see what happens?

[sarriegi]

Thank you for the suggestion, Takis! I have used this configuration successfully before. I am using atmesh.F90 and atmesh_mod.F90 files that are not the original ones in CoastalApp. These files allow for the wind forcing file to be given on a regular (structured) grid. I am attaching these files here: atmesh.txt atmesh_mod.txt

Angelos

[pvelissariou1]

@sarriegi I guess the atmospheric grid overlaps the ocean mesh completely.