search

oceanmodeling / ufs-weather-model

This repo is forked from ufs-weather-model, and contains the model code and external links needed to build the UFS coastal model executable and model components, including the ROMS, FVCOM, ADCIRC and SCHISM plus WaveWatch III model components.
https://github.com/oceanmodeling/ufs-coastal-app
Other
2 stars 3 forks source link

Install dependencies/spack-stack on Frontera #33

Closed janahaddad closed 3 months ago

janahaddad commented 4 months ago

Description

This is the first step in preparing the upcoming UFS Coastal/NUOPC Walkthrough. Spack-stack and dependencies need to be installed on Frontera system.

Solution

We met Tues 2/13 and agreed on this initial step, @pvelissariou1 will work on installing.

See https://spack-stack.readthedocs.io/en/latest/CreatingEnvironments.html

uturuncoglu commented 4 months ago

@janahaddad I contacted with Benjamin A. Cash from George Mason University about Frontera. It seems that they have an old version of spack-stack in there and they are running fully coupled P8 test from UFS Weather Model. It needs to be installed again with recent one and I am not sure they could able to run the regression tests at this point. I am trying to get some details from them that I could share with you. If I could, maybe we don't need to start from scratch. Anyway, I'll update you about it.

uturuncoglu commented 4 months ago

@janahaddad @pvelissariou1 I could able to login Frontera. I am waiting some response from spack-stack side. I think that are plaining to add Frontera as Tear-2 system and we need to be the maintainers of it. I'll update you when I have more information.

pvelissariou1 commented 4 months ago

@uturuncoglu This is great news. Thank you for taking the lead on this.

uturuncoglu commented 4 months ago

@janahaddad @pvelissariou1 It turns out that they have site definition for Frontera for spack-stack 1.5.1 (https://github.com/JCSDA/spack-stack/tree/release/1.5.1/configs/sites/frontera) but this is not in the develop. I am not sure about their plan for now but we could use this as a base. I am working on it and let you know about progress.

pvelissariou1 commented 4 months ago

@uturuncoglu On Frontera there is no a NOAA specific setup. For CoastalApp and other related simulations I developed and installed a full module system and libraries in /work2/07380/panvel/apps and this is what we were using. I believe spack-stack need to be installed in some OS related folder/filesystem, say in one of the /scratch* folders. Let's see what the spack-side will say.

uturuncoglu commented 4 months ago

@pvelissariou1 Okay. I could able to install spack-stack 1.5.1. This is the version that spack-spack developers are used. We could build our own with never intel compiler etc. Anyway, at this point the main item is to build one of the costal configuration against it. If that works we could start working on RT system to support running them via rt.sh. Anyway, I'll keep you updated.

uturuncoglu commented 4 months ago

@janahaddad @pvelissariou1 @saeed-moghimi-noaa This is almost done. I still need to fix couple of minor issues.

uturuncoglu commented 4 months ago

@janahaddad @pvelissariou1 @saeed-moghimi-noaa @yunfangsun I think Frontera support is ready for external testing. Here are the details,

https://github.com/oceanmodeling/ufs-coastal/commit/3b5ba93812bffc58cde8683d25ef65f5ec9f8685 (brings frontera support) https://github.com/oceanmodeling/ufs-coastal/commit/f108a0478d699f6396a95f019f99c3c7c7a724a9 (adjust coastal specific tests)

We could extend it to cover gnu later. I think this is not urgent. Let me know if you want to make this a priority.

cd $WORK
git clone --recursive -b feature/coastal_app https://github.com/oceanmodeling/ufs-coastal.git
cd tests
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_ike_shinnecock_atm2adc intel
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_scituateharbor_atm2fvc intel
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_ike_shinnecock_atm2sch intel
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_ike_shinnecock_atm2sch2ww3 intel
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_irene_atm2roms intel
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_ike_shinnecock_ww3 intel
./rt.sh -a [ACCOUNT] -l rt_coastal.conf -k -n coastal_ike_shinnecock_atm2ww3 intel

At this point we are just supporting those tests (bold in wiki page) and be aware that we have reproducibility issue with coastal_scituateharbor_atm2fvc and coastal_ike_shinnecock_atm2sch2ww3 tests. Those are documented in the component side. I also updated the wiki page (https://github.com/oceanmodeling/ufs-coastal/wiki/Current-Status-of-UFS%E2%80%90Coastal-Implementation) and you could see the issues and also other information over there.

Anyway, please test and let me know if you have any issue (permission etc.). then we could fix them. Also note that I tried to install spack-stack 1.6.0 but it failed. I'll look at it but it is not urgent a this point.

I am planing to spend most of my time next week for the land project so I could only spend minimal time if we have any issue since I need to make some progress also in that side. Once I back to coastal work, I strongly suggest that priority needs to be in the reproducibility issue of the two RT configurations but let me know if it is not.

uturuncoglu commented 4 months ago

BTW, we might need to modify Frontera part of tests/default_vars.sh little bit more (basically sets some number some cores, threading etc.) to run non coastal tests but I'll look at it later since our intention is limited with the ufs costal and also I don't have details all the variables used in there.