Initial testing of UFS-Coastal on Frontera

[pvelissariou1]

NOTES (1): Compile UFS-Coastal on Frontera

• Using BUILD_TOOLS to build the SCHISM tools via the call: add_subdirectory(SCHISM/src/Utility) in SCHISM-interface/CMakeLists.txt breaks the compilation (the Utility folder is added in the SCHISM/src/CMakeLists.txt) . The SCHISM tools are always built by default (they are absolutely necessary for pre- and post-processing tasks). The following code block needs to be removed (not needed):

  if(BUILD_TOOLS)
  add_subdirectory(SCHISM/src/Utility)
  endif()

• The BUILDTOOLS flag is also used by WW3. For WW3, this flag needs to be defined to compile the ww3* tools. If we need a BUILD_TOOLS flag for SCHISM, then we need to use another name to avoid conflicts.
• Spack-stack on Frontera is using Intel 19.1.1; had bad experiences with this version of the Intel compiler. How about using intel/23.1.0 + impi/21.9.0? I will check the current spack-stack config and I will report back.

[pvelissariou1]

@uturuncoglu Ufuk, please check NOTES (1) in this issue

[uturuncoglu]

@pvelissariou1 Thanks for testing. I'll check BUILD_TOOLS but as I know SCHSIM only build Utils when it is compiled standalone with latest update. So, BUILD_TOOLS is working with latest update without any issue but since we have issue with their recent code change I revert the SCISM pointer to the version before the latest update. So, maybe it is not working now because of it. So, I think once the wave output issue is solved, we are good with the current implementation and there is no need to change it. For now, I could comment out that part and once they fix the wave output issue we could reactivate it. For the compilers, these are the current supported compilers. Let's go with them for now. I have plan to update them but it could take time in terms of testing and building spack. Anyway, I think this is not the priority for the first tutorial and we could make it for the second one.

[pvelissariou1]

@uturuncoglu Actually all the SCHISM tools were built as exptected without using the BUILD_TOOLS flag in: /work2/07380/panvel/ufs-coastal/tests/build_fv3_coastalPSW/SCHISM-interface/SCHISM/src/Utility/* folders

[uturuncoglu]

@pvelissariou1 Yes, I know but it won't work if we update the SCHSIM to its head of develop since the SCHSIM internal build system is updated and it will only build tools in standalone mode. So, the line that I add previously enables us to build tools with the most recent version of SCHSIM. At this point we could not use it since there is an issue related with wave fields. Again, for now we could comment that part in the CMake interface and enable it once the model is fixed and updated.

[pvelissariou1]

Ok, I see. In this case let's use a flag other than BUILD_TOOLS maybe something BUILD_SCHISM_TOOLS, so there is no conflict with WW3

[uturuncoglu]

@pvelissariou1 I agree. I was thinking to use same with WW3 by this way you could pass only one CPP option to build all the tools but maybe I was wrong and it could be confusing. Let me change it along with other staff.

[yunfangsun]

@uturuncoglu , I have tested the case of coastal_ike_shinnecock_atm2sch on Frontera, it went through, and no error messages were shown.

[pvelissariou1]

@yunfangsun Yunfang thank you very much for testing.

[uturuncoglu]

@yunfangsun That is great. Thanks for testing. If you don't mind could you also try to run coastal_ike_shinnecock_atm2sch2ww3 case. I have just update the baseline since we had a fix in SCHSIM side.

[yunfangsun]

Hi @uturuncoglu and @pvelissariou1 ,

I have run the coastal_ike_shinnecock_atm2sch2ww3 case on Frontera, it did run and finish the simulation at /work2/09283/yfsun001/frontera/yfsun001/FV3_RT/rt_137409/coastal_ike_shinnecock_atm2sch2ww3_intel, however, in the log folder "/work2/09283/yfsun001/frontera/ufs-coastal/tests/logs/log_frontera" the files are compile_12_intel.log compile_12_intel_time.log job_001_timestamp.txt job_002_timestamp.txt run_coastal_ike_shinnecock_atm2sch2ww3_intel.log

There is no rt_001_coastal_ike_shinnecock_atm2sch2ww3_intel.log file existing, it seems it didn't do the comparison on Frontera.

[uturuncoglu]

@yunfangsun Let me try to run in my side to see what happens.

[uturuncoglu]

@yunfangsun I could able to run coastal_ike_shinnecock_atm2sch2ww3 without any error. Maybe there are some permission issue. If you could share your run directory and open permissions. I could check.

+ ./run_test.sh /work2/01118/tg803972/frontera/ufs-coastal/tests /scratch1/01118/tg803972/frontera/tg803972/FV3_RT/rt_4645 coastal_ike_shinnecock_atm2sch2ww3 001 12_intel
+ continue
+ read -r line
+ '[' '' ']'
+ [[ false == true ]]
+ [[ false == true ]]
+ set +e
+ cat /work2/01118/tg803972/frontera/ufs-coastal/tests/logs/log_frontera/compile_12_intel_time.log
+ cat /work2/01118/tg803972/frontera/ufs-coastal/tests/logs/log_frontera/rt_001_coastal_ike_shinnecock_atm2sch2ww3_intel.log
+ FILES='fail_test_* fail_compile_*'
+ for f in '$FILES'
+ [[ -f fail_test_* ]]
+ for f in '$FILES'
+ [[ -f fail_compile_* ]]
+ [[ -e fail_test ]]
+ echo

+ echo REGRESSION TEST WAS SUCCESSFUL
REGRESSION TEST WAS SUCCESSFUL
+ echo
+ echo REGRESSION TEST WAS SUCCESSFUL
+ rm -f 'fv3_*.x' fv3_12_intel.exe modules.fv3_12_intel.lua 'modulefiles/modules.fv3_*' keep_tests.tmp
+ [[ true == false ]]
+ [[ false == true ]]
+ [[ false == true ]]
+ [[ coastal_ike_shinnecock_atm2sch2ww3 != '' ]]
+ rm -f rt_single.conf
+ date
++ printf '%02dh:%02dm:%02ds\n' 0 34 34
+ elapsed_time=00h:34m:34s
+ echo 'Elapsed time: 00h:34m:34s. Have a nice day!'
+ echo 'Elapsed time: 00h:34m:34s. Have a nice day!'
Elapsed time: 00h:34m:34s. Have a nice day!
+ echo 'rt.sh finished'
rt.sh finished
+ cleanup
++ awk '{print $2}'
+ [[ 4645 == 4645 ]]
+ rm -rf /work2/01118/tg803972/frontera/ufs-coastal/tests/lock
+ [[ false == true ]]
+ trap 0
+ exit

[yunfangsun]

Hi @uturuncoglu ,

I changed the permission of my Frontera folder, you can check the log file at /work2/09283/yfsun001/frontera/ufs-coastal/tests/logs/log_frontera and the run folder at /work2/09283/yfsun001/frontera/yfsun001/FV3_RT/rt_323275/coastal_ike_shinnecock_atm2sch2ww3_intel

Thank you for helping to check it.

[uturuncoglu]

@yunfangsun That is great. Thanks for testing.

[janahaddad]

@pvelissariou1 will test oceanmodeling/ufs-coastal#58 on Frontera after the current Orion tests are done