search

oceanmodeling / ufs-weather-model

This repo is forked from ufs-weather-model, and contains the model code and external links needed to build the UFS coastal model executable and model components, including the ROMS, FVCOM, ADCIRC and SCHISM plus WaveWatch III model components.
https://github.com/oceanmodeling/ufs-coastal-app
Other
5 stars 5 forks source link

Porting UFS Coastal to custom platform #14

Open uturuncoglu opened 1 year ago

uturuncoglu commented 1 year ago

@pvelissariou1 @saeed-moghimi-noaa @hga007 I am creating this to continue conversation about porting ufs-coastal to new platform and also create documentation about it for the community.

saeed-moghimi-noaa commented 1 year ago

This sounds very good. We need more of https://github.com/oceanmodeling/ufs-coastal/wiki/Current-Status-of-UFS%E2%80%90Coastal-Implementation regular updates and adding more wiki's and howto's.

Thanks,

uturuncoglu commented 1 year ago

@saeed-moghimi-noaa I think we could use wiki for some quick update and maybe instructions but we need to go with Sphinx for more detailed documentation since it is commonly used by UFS. I could create the initial version of it and put instruction here for compiling and pushing documentation updates.

uturuncoglu commented 1 year ago

@pvelissariou1 I started looking to stack-stack and define the additional dependencies that is required for the ufs-coastal. There is my list,

Anyway, we need to make libraries consistent along the components. So, everybody could use same stack. @saeed-moghimi-noaa @pvelissariou1 Please let me know if you know any limitations in terms of used library in the components. I think that Metis is the common one (I am not sure about parmetis). We need to specify single version of each additional library.

uturuncoglu commented 1 year ago

Maybe we need to install Proj4-fortran and Julian externally outside of the spack environment and use as externals. Since they are not actively development and outside of the spack. This could be done in app level. Any suggestion?

pvelissariou1 commented 1 year ago

@uturuncoglu ParMETIS has licensing issues and cannot be officially ported in NOAA applications. Newer versions of WW3 will use the scotch library. I don't know what the SCHISM developers will do. Julian code should be part of FVCOM source code or, they need to update their sources to account for the functionality of Julian. It is a very old library and will not be part of spack.

pvelissariou1 commented 1 year ago

@uturuncoglu Proj4-fortran can also be compiled within the FVCOM source code (as is METIS in ADCIRC for example). It cannot be an external library. For both issues I have already talked with the FVCOM developers. They need to start working on their source code as soon as possible. I'll talk to them this week.

pvelissariou1 commented 1 year ago

@uturuncoglu For the proj library, FVCOM only supports version 4* at this point (and proj4-fortran is not tested with versions of proj > 4). We might need to talk with the FVCOM developers about this as well.

pvelissariou1 commented 1 year ago

@uturuncoglu , @saeed-moghimi-noaa As a first iteration we might need to stick with spack-stack but eventually we need to have a UFS-Coastal to be more portable than this.

pvelissariou1 commented 1 year ago

@uturuncoglu Regarding the documentation, let's start with sphinx to create the basic documentation, but in the end we would like to have an extended documentation on NUOPC caps within UFS-Coastal with fortran calls, code trees etc. I am not sure if Sphinx can do that for Fortran (and in what level). We might need to use multiple documenters to create the complete end documentation (user and developer). Let's both of us work on this.

uturuncoglu commented 1 year ago

@pvelissariou1 I think we could go with spack-stack. It seems it is flexible enough. As I mentioned before we could install couple of libraries externally through the app. For parmetis, we could go with spack installation for now until models start to use scotch library. Since we are not plaining to use ufs-coastal in operational machine in short term going with the exiting libraries for now could be the solution. Switching scotch could take time for the components. Anyway, I am teasing stack-stack on Orion now and I'll update you about it. Once I have successful installation and also tested UFS Costal with it we could start populate the documentation with Sphinx.

pvelissariou1 commented 1 year ago

@uturuncoglu Ok, then do your testing on Orion with spack-stack. On Hercules they are using spack. Soon we will check both implementations I guess.

uturuncoglu commented 1 year ago

@pvelissariou1 spack-stack is additional layer top of spack to make build easy. Even if you are seeing spack, the modules are built with spack-stack. So, spack-stack in the top level to install everything rather than installing through spack itself.

pvelissariou1 commented 1 year ago

@uturuncoglu , @saeed-moghimi-noaa Porting issues on Hercules/Paraller Works/Hera:

uturuncoglu commented 1 year ago

@pvelissariou1 Hercules support with spack-stack was completed for both Intel and GNU last week (https://github.com/ufs-community/ufs-weather-model/pull/1733). We need to sync UFS-Coastal to bring those changes. Last week I tried to install spack-stack on Orion and try to run ufs wether model with the newly compiled stack but it was failing. I'll look at it more to find the issue. Once that works I'll try to add additional libraries like Metis/parmetis using chaining feature. In any case, the component needs to get rid of Metis to used under operational machine. I know this is a big transition but we might need to inform the component developers in advance.

pvelissariou1 commented 1 year ago

@uturuncoglu Thank you. Let's work together on this to port UFS-Coastal to Hercules as soon as possible. It is going to be a usefule learning curve for me.

uturuncoglu commented 1 year ago

@pvelissariou1 Sure. I think once we sync the model, probably it will be run without any issue. Let's discuss it tomorrow.

pvelissariou1 commented 1 year ago

@tufuk Yes, that is why I put my thoughts here

uturuncoglu commented 1 year ago

@pvelissariou1 @saeed-moghimi-noaa I sync ufs-coastal with ufs-weather-model as of today. Now it has Hercules support. At this point, I only tested ROMS RT with the synced model and everything is working as expected. ROMS now points its official repository (develop branch) too. I'll keep continue to test other components and sync the configurations with CoastalApp test suite and fix the issues if I have. In the meantime, please let me know if you have any issues. BTW, after sync now ufs-coastal is using ESMF 8.4.2 (the previous one was using 8.3.0b09).

hga007 commented 1 year ago

It worked well using ESMF 8.4.2 on my computer with JEDI Spack-Stack. Gosh, you guys have so many computers to run. Is Hercules at MSU, too?

pvelissariou1 commented 1 year ago

@hga007 Hercules is at MSU as well. Still not in full production mode though.

uturuncoglu commented 1 year ago

I was testing custom installation on Orion and hit minor problem. Here is the details of the issue: https://github.com/JCSDA/spack-stack/issues/807. My plan is to install first to a known platform. Then, if everything goes fine, I am planning to chain extra dependencies like parmetis, Metis, proj etc. Some of the extra dependencies can be installed manually (it would be part of functionality provided by the ufs-coastal-app) since they don't have package under spack or spack-stack. After that we could create set of instructions for the user to port UFS Coastal to custom platforms. Eventually, these steps will be part of app.

uturuncoglu commented 1 year ago

@pvelissariou1 @saeed-moghimi-noaa I could able to run ufs-coastal with custom spack-stack build under Orion. At this point, I am trying to run it under Docker container and populate the instructions for it. When I have successful build and run (the model might need some minor fixes in terms of CMake interfaces of the ufs-coastal components but I think I could fix them), I'll push those information to app level documentation. So, you could also try to test.

uturuncoglu commented 1 year ago

@pvelissariou1 @saeed-moghimi-noaa I was working with spack-stack and about porting ufs-coastal. Since I have successful build on Orion, I decided to work on a Docker container (Ubuntu Linux) to see what happens if I go with totally unsupported platform. The good news is that I could able to install all the dependencies and build at leas SCHSIM case under container using gcc. I'll create a section about in the app level documentation for the users. I think this effort could be also extrapolated to any other linux cluster. Still following needs to be done,

saeed-moghimi-noaa commented 1 year ago

@uturuncoglu @pvelissariou1

Thanks for the update. Good progress. I want to ask @SorooshMani-NOAA to chime in here in regards to using Singularity instead of Docker for UFS-Coastal, Also formed a list of a possible action items to check off:

SorooshMani-NOAA commented 1 year ago

@saeed-moghimi-noaa reading the comments above my understanding is that Docker was tested just to see if spack-stack is successful or not in that environment. In that case it's a valid test, however if one wants to run tests regularly using container, especially on HPC platforms, then singularity (i.e. apptainer) containers is the way to go as far as I understand.

Also note that even if one wants to use Singularity on HPC there's a valid usecase for Docker, and that is to create a image with the dependencies to start from when building the Singularity image. This is similar to what @pvelissariou1 is doing for compilation tests of CoastalApp, but then instead of running GitHub it's going to run Singularity to start from the Docker image of interest.

All in all, both Docker and Singularity have their own merit depending on what is the use-case. That's my 2 cents.

saeed-moghimi-noaa commented 1 year ago

@saeed-moghimi-noaa reading the comments above my understanding is that Docker was tested just to see if spack-stack is successful or not in that environment. In that case it's a valid test, however if one wants to run tests regularly using container, especially on HPC platforms, then singularity (i.e. apptainer) containers is the way to go as far as I understand.

Also note that even if one wants to use Singularity on HPC there's a valid usecase for Docker, and that is to create a image with the dependencies to start from when building the Singularity image. This is similar to what @pvelissariou1 is doing for compilation tests of CoastalApp, but then instead of running GitHub it's going to run Singularity to start from the Docker image of interest.

All in all, both Docker and Singularity have their own merit depending on what is the use-case. That's my 2 cents.

Thanks @SorooshMani-NOAA .

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. I know proj needs to be version 4. The latest version is 4.9.2 for version 4 (btw, the last version available with spack is 8.2.1). Does it work fine with the FVCOM? Do we have any restriction in terms of used Metis/parmetis libraries. What is the common version that can be used by all the components?

pvelissariou1 commented 1 year ago

@uturuncoglu No it doesn't work with FVCOM. The model requires version 4 (4.9.2) is what I use in CoastalApp. Now, proj is not exactly required by FVCOM except special cases. I will again talk to them. Regarding Julian and proj-fortran libraries, these should be part of the FVCOM source tree and be compiled internally along with the FVCOM. These libraries can be in spack. They asked me to do this for them among other things but I don't have time at this point. I will again talk with them.

uturuncoglu commented 1 year ago

@pvelissariou1 Thanks. I think Julian and proj-fortran libraries needs to be handled internally since we have no packages in the spack side and I think we don't want to maintain those packages under spack. So, at this point just proj and Metis libraries can be handled by spack.

pvelissariou1 commented 1 year ago

@uturuncoglu Correct, please let's focus on proj and METIS in spack.

uturuncoglu commented 1 year ago

@pvelissariou1 I could able to build SCHSIM against external parmetis installation. When I try to install ADCIRC I am getting error like following undefined reference tometisestimatememory'. The main reason is that ADCIRC is not compatible with Metis 5 and requires Metis 4.0.3 but parmetis spack-stack package can be only installed with Metis 5. At this point I am not sure what we need to do. I think that we need to update ADCIRC to use Metis 5 rater than Metis 4. The only function ismetisestimatememory'` and it is used in the following sources,

../ADCIRC-interface/ADCIRC/prep/metis.F:        EXTERNAL metis_partgraphkway,metis_estimatememory
../ADCIRC-interface/ADCIRC/prep/metis.F:      CALL metis_estimatememory(MNP,XADJ,ADJNCY,NUMFLAG,
../ADCIRC-interface/ADCIRC/prep/metis.F:        EXTERNAL metis_estimatememory

I think this can be removed but not sure how it will affect. @dwirasae do you have any idea?

uturuncoglu commented 1 year ago

@pvelissariou1 BTW, I already push the changes for SCHSIM and now support both internal build and also use external parmetis. I have new version of ADCIRC build interface that supports both way (not working yet due to conflict in the Metis version but once we solve it I think it will build without any issue). I'll push that one also to repo.

uturuncoglu commented 1 year ago

@pvelissariou1 @dwirasae If i remove metis_estimatememory references from ADCIRC/prep/metis.F it builds but I did not try to run. I am not sure prep will run with Metis 5 or not.

pvelissariou1 commented 1 year ago

@uturuncoglu @saeed-moghimi-noaa Ufuk, I think ParMETIS should not be part of spack as this will raise issues with NOAA. It is good to have the capability to build and use ParMETIS though in case we (or the user) need to do so. Regarding ADCIRC and METIS, my understanding is that @dwirasae is working on bringing METIS 5 into ADCIRC. The issue is that it will be a modified METIS to support Adaptive Message Passing Interface (AMPI). I don't know what kind of modifications @dwirasae or others are implementing and how to cope with these. @dwirasae Dam, could you please chime in?

uturuncoglu commented 1 year ago

@pvelissariou1 in the way that I used is not part of the spack-stack or model itself. We are just using low level spack commands to install it. So, it is not different from installing through the component CMakelist interface. Once we get rid of Métis dependency we could remove those extra commands.

uturuncoglu commented 1 year ago

@pvelissariou1 I found an issue in spack proj package and open an issue: https://github.com/spack/spack/issues/40456 it will be fixed soon. I have a workaround for it until it appears in the spack-stack which allows me to modify FVCOM Cmake interface.

pvelissariou1 commented 1 year ago

Great, thank you. What was the issue?

uturuncoglu commented 1 year ago

@pvelissariou1 @saeed-moghimi-noaa @yunfangsun I updated the documentation to include information about installing spack-stack and also using it under ufs-coastal. I still need to test the RT etc. but it is a good starting point. Anyway, if you have time please try those instructions and let me know if you have any issues/questions. The updated documentation is in following link: https://oceanmodeling.github.io/ufs-coastal-app/versions/main/html/porting.html#docker-container

pvelissariou1 commented 1 year ago

@uturuncoglu Thank you Ufuk, I will follow on this as well.

pvelissariou1 commented 8 months ago

Finished & presented tutorial # 1. Preparations started for tutorial # 2 on Frontera (SCHISM & possibly an atm2fvc case)

janahaddad commented 7 months ago

Moving documentation issue to #57 so that this issue can be focused on porting to custom platform

uturuncoglu commented 7 months ago

@janahaddad @pvelissariou1 @saeed-moghimi-noaa I have just updated spack-stack installation on Frontera. spack-stack 1.6.0 version is installed without any issue and we have minor issue with develop. The draft PR is in here: https://github.com/JCSDA/spack-stack/pull/1075 and the MAPL library issue is documented in https://github.com/JCSDA/spack-stack/issues/1076. I still need to test in UFS Coastal side and update the module file.

saeed-moghimi-noaa commented 7 months ago

@uturuncoglu @pvelissariou1

Would it be possible to develop a how to on how to stand up a spack-stack for ufs-coastal?

Thanks

uturuncoglu commented 7 months ago

@saeed-moghimi-noaa We have a script in the app level but it is just tested on container. It requires more work. Also we have app level basic documentation. In Spack-stack side there is a plan to simplify the build process for university platforms etc. spack-stack has a script for Cloud based installation. So, there are planing to do the same for the clusters. I hope it would be easier in the near future since there are lots of pressure coming from community.