Model level issues/discussions

uturuncoglu commented 11 months ago

I am creating this issue to keep track model level issues, discussions and improvements.

uturuncoglu commented 11 months ago

Here is the recent issue (https://github.com/ufs-community/ufs-weather-model/issues/1837) that I opened in UFS Weather Model side about improvement in the rt.sh script.

uturuncoglu commented 10 months ago

@pvelissariou1 @saeed-moghimi-noaa JFYI, I created following wiki page to keep track of current status of integration of ufs-coastal.

https://github.com/oceanmodeling/ufs-coastal/wiki/Current-Status-of-UFS%E2%80%90Coastal-Implementation

saeed-moghimi-noaa commented 10 months ago

Hi @uturuncoglu This looks great many thanks! @saeed-moghimi-noaa

uturuncoglu commented 10 months ago

@pvelissariou1 @saeed-moghimi-noaa @yunfangsun Since we are working on multiple components in the sometime it is hard to track which one is working and which one has issue. So, I would like to summarize the current situation in here then I propose a plan to go forward.

The current situation:

ADCIRC: As we tested the prep is fixed and now we could run both cases coastal_florence_hsofs_atm2adc and coastal_ike_shinnecock_atm2adcwith CMEPS. At this point we are waiting fix from @dwirasae. He will push it soon to ADCIRC repo and we will update the ufs-coastal and test again.
FVCOM: The cap fixed and now we could run with CMEPS mediator. We have currently one RT under ufs-coastal (coastal_scituateharbor_atm2fvc).
SCHISM: The cap was working before updating the model. There were some confusion to set nws parameter and also some issue in the cap. As I know we are waiting for the SCHSIM developers (@josephzhang8 @platipodium) to fix the issues and then we will retest in our side (under ufs-coastal).
WW3: Last time that I checked it was working. We have two tests under ufs-coastal, coastal_ike_shinnecock_ww3 and coastal_ike_shinnecock_atm2ww3. It would be nice to test them again. BTW, there was some configuration issue in WW3 side that was preventing to write output from WW3. I am not sure this is fixed or not but if it is fixed and working, I could update the cases under ufs-coastal.
ROMS: The build sistem is working and we could compile ROMS under ufs-coastal but having some difficulty to run atm2roms configuration. @hga007 provided new configuration file and fixed the model. I'll test it and update you.

What is Next:

My suggestion is to finalize the ROMS and test all the available RTs under ufs-coastal and confirm that they all working. Then, we could sync ufs-coastal with ufs-weather-model and retest again. If everything works without any issue. I am plaining to create baseline on Orion for all RTs specific to ufs-coastal. I know Takis is working with parametric model and I think that we have that one too soon. So, before adding more tests to exiting ones and bringing other configurations, I think it is better to wait little bit and make all the tests stable. BTW, we could start testing it on different platforms.

We also need to think about strategy og having consistent dependencies among the model components. For example, Metis is one of them and it is used by multiple components. As I know, ufs-weather model is switching spack-stack individually in each platform. I know that spack has upstream capability (more information in here: https://spack.readthedocs.io/en/latest/chain.html) to use editing spack installation and extend it with additional libraries. So maybe that is the way that we need to go and this could be available in the app level. Maybe after having stable model with existing configurations, we might need to focus little bit to app level. Anyway, I think it would be nice to arrange a meeting for it and discuss it little bit more.

I am not sure but if we need to bring any urgent configuration required for the AMS etc. Please let me know. If you think that the existing ones are fine then that is great.

uturuncoglu commented 10 months ago

@pvelissariou1 @saeed-moghimi-noaa @yunfangsun The other item that we need to do is to have a app that compiles all the components. So, user will able to run any application without recompiling again. That could introduce more naming clashes that we had recently with SCHSIM and WW3 (that is another test that we want to define as RT).

hga007 commented 10 months ago

Yes, the Spack-Stack is still evolving and requires expertise to install on our computers. We use Spack-Stack exclusively in the ROMS-JEDI interface, which requires lots of packages:

Currently Loaded Modules:
  1) MATLAB/R2022b           23) sqlite/3.34.1           45) gsl-lite/0.37.0              67) py-bottleneck/1.3.5
  2) totalview/2023.1.6      24) util-linux-uuid/2.38.1  46) libjpeg/2.1.0                68) py-pyparsing/3.0.9
  3) intel/2021.8.0          25) python/3.10.8           47) krb5/1.20.1                  69) py-packaging/21.3
  4) numactl/2.0.14          26) py-pip/22.2.2           48) libtirpc/1.2.6               70) py-numexpr/2.8.3
  5) pmix/4.1.2              27) wget/1.21.1             49) hdf/4.2.15                   71) py-six/1.16.0
  6) zlib/1.2.13             28) base-env/1.0.0          50) jedi-cmake/1.4.0             72) py-python-dateutil/2.8.2
  7) openmpi/4.1.4           29) boost/1.78.0            51) ncview/2.1.8                 73) py-pytz/2022.2.1
  8) cmake/3.23.1            30) bufr/11.7.1             52) netcdf-cxx4/4.3.1            74) py-pandas/1.4.0
  9) curl/7.76.1             31) git-lfs/2.13.3          53) json/3.10.5                  75) py-pybind11/2.8.1
 10) gettext/0.21            32) crtm/v2.4.1-jedi        54) json-schema-validator/2.1.0  76) py-pycodestyle/2.8.0
 11) libunistring/0.9.10     33) ecbuild/3.7.2           55) odc/1.4.5                    77) py-pyhdf/0.10.4
 12) libidn2/2.3.0           34) openjpeg/2.3.1          56) py-attrs/22.1.0              78) py-pyyaml/6.0
 13) pcre2/10.42             35) eccodes/2.27.0          57) py-pycparser/2.21            79) py-gast/0.5.3
 14) git/2.38.1              36) eigen/3.4.0             58) py-cffi/1.15.1               80) py-beniget/0.4.1
 15) hdf5/1.14.0             37) openblas/0.3.19         59) py-findlibs/0.0.2            81) py-ply/3.11
 16) zstd/1.5.2              38) eckit/1.23.0            60) py-setuptools/59.4.0         82) py-pythran/0.11.0
 17) netcdf-c/4.9.2          39) fftw/3.3.10             61) py-numpy/1.22.3              83) py-scipy/1.9.3
 18) nccmp/1.9.0.1           40) fckit/0.10.1            62) py-eccodes/1.4.2             84) py-xarray/2022.3.0
 19) netcdf-fortran/4.6.0    41) fiat/1.1.0              63) py-f90nml/1.4.3              85) sp/2.3.3
 20) parallel-netcdf/1.12.2  42) ectrans/1.2.0           64) py-h5py/3.6.0                86) udunits/2.2.28
 21) parallelio/2.5.9        43) atlas/0.33.0            65) py-cftime/1.0.3.4            87) jedi-base-env/1.0.0
 22) libxcrypt/4.4.33        44) gsibec/1.0.7            66) py-netcdf4/1.5.3             88) stack-intel/2021.8.0

However, it doesn't have ESMF yet. We could added it, but we didn't want to screw-up JEDI interfaces.

Yes, we would like to compile and run ufs-coastal and add test cases to the roms_test repository. However, I don't know how complicated it will be to compile with Spack-Stack. Luckily, Dave is very good with Spacl-Stack.

hga007 commented 9 months ago

Yes, the Spack-Stack is still evolving and requires expertise to install on our computers. We use Spack-Stack exclusively in the ROMS-JEDI interface, which requires lots of packages:

Currently Loaded Modules:
  1) MATLAB/R2022b           23) sqlite/3.34.1           45) gsl-lite/0.37.0              67) py-bottleneck/1.3.5
  2) totalview/2023.1.6      24) util-linux-uuid/2.38.1  46) libjpeg/2.1.0                68) py-pyparsing/3.0.9
  3) intel/2021.8.0          25) python/3.10.8           47) krb5/1.20.1                  69) py-packaging/21.3
  4) numactl/2.0.14          26) py-pip/22.2.2           48) libtirpc/1.2.6               70) py-numexpr/2.8.3
  5) pmix/4.1.2              27) wget/1.21.1             49) hdf/4.2.15                   71) py-six/1.16.0
  6) zlib/1.2.13             28) base-env/1.0.0          50) jedi-cmake/1.4.0             72) py-python-dateutil/2.8.2
  7) openmpi/4.1.4           29) boost/1.78.0            51) ncview/2.1.8                 73) py-pytz/2022.2.1
  8) cmake/3.23.1            30) bufr/11.7.1             52) netcdf-cxx4/4.3.1            74) py-pandas/1.4.0
  9) curl/7.76.1             31) git-lfs/2.13.3          53) json/3.10.5                  75) py-pybind11/2.8.1
 10) gettext/0.21            32) crtm/v2.4.1-jedi        54) json-schema-validator/2.1.0  76) py-pycodestyle/2.8.0
 11) libunistring/0.9.10     33) ecbuild/3.7.2           55) odc/1.4.5                    77) py-pyhdf/0.10.4
 12) libidn2/2.3.0           34) openjpeg/2.3.1          56) py-attrs/22.1.0              78) py-pyyaml/6.0
 13) pcre2/10.42             35) eccodes/2.27.0          57) py-pycparser/2.21            79) py-gast/0.5.3
 14) git/2.38.1              36) eigen/3.4.0             58) py-cffi/1.15.1               80) py-beniget/0.4.1
 15) hdf5/1.14.0             37) openblas/0.3.19         59) py-findlibs/0.0.2            81) py-ply/3.11
 16) zstd/1.5.2              38) eckit/1.23.0            60) py-setuptools/59.4.0         82) py-pythran/0.11.0
 17) netcdf-c/4.9.2          39) fftw/3.3.10             61) py-numpy/1.22.3              83) py-scipy/1.9.3
 18) nccmp/1.9.0.1           40) fckit/0.10.1            62) py-eccodes/1.4.2             84) py-xarray/2022.3.0
 19) netcdf-fortran/4.6.0    41) fiat/1.1.0              63) py-f90nml/1.4.3              85) sp/2.3.3
 20) parallel-netcdf/1.12.2  42) ectrans/1.2.0           64) py-h5py/3.6.0                86) udunits/2.2.28
 21) parallelio/2.5.9        43) atlas/0.33.0            65) py-cftime/1.0.3.4            87) jedi-base-env/1.0.0
 22) libxcrypt/4.4.33        44) gsibec/1.0.7            66) py-netcdf4/1.5.3             88) stack-intel/2021.8.0

However, it doesn't have ESMF yet. We could added it, but we didn't want to screw-up JEDI interfaces.

Yes, we would like to compile and run ufs-coastal and add test cases to the roms_test repository. However, I don't know how complicated it will be to compile with Spack-Stack. Luckily, Dave is very good with Spacl-Stack.

hga007 commented 9 months ago

I built a grid for the Lake Erie and Lake St. Clair systems for coupling CICE6-ROMS in the UFS system. I have 1.0x1.0 km (396x150) and 1.2x1.2 km (330x124) resolution rotated grids. The number of vertical levels is to be determined, probably around 10. I also plan to use it as a standalone test for ROMS with ECMWF forcing. The ROMS native coupled system also has a NUOCP cap for CICE6, which I need to test with this application. It is a good application for testing. I need to work a little on the land/water mask. Also, I can use this application to test the ROMS native sea-ice model.

LakeErie_1000m_c LakeErie_1200m

uturuncoglu commented 9 months ago

@hga007 It looks great. BTW, UFS has CICE6 component along with its NUOPC cap too. I am not sure about the difference between those caps but once we have working case with CMEPS we could start experiment it.

uturuncoglu commented 9 months ago

@pvelissariou1 @saeed-moghimi-noaa I think we need to fork some components like CMEPS, CDEPS, WW3 etc. under oceanmodeling. At this point, the development related with those components are in my personal GitHub account. It would be nice to fork them from NOAA-EMC under oceanmodeling (or any other organization) and move the UFS Coastal specific development to the forks under oceanmodeling. BTW, we could keep our development in there without any dependency to any developers. It will also make easy to use same branch across coastal development team. Anyway, let mw know what you think?

saeed-moghimi-noaa commented 9 months ago

Hi @uturuncoglu

I agree. Do you have the privilege to add those forks? if not let me have the list of github repos so I can help with that.

Thanks.

uturuncoglu commented 9 months ago

@saeed-moghimi-noaa I could able to fork. Let me fork the ones that are in my personal account. I'll update UFS coastal soon to point new repositories.

saeed-moghimi-noaa commented 9 months ago

@uturuncoglu

If those are your fork. Giving up ownership might be cleaner.

uturuncoglu commented 9 months ago

@saeed-moghimi-noaa @pvelissariou1 Okay. I created forks for CMEPS, CDEPS and also WW3 (has very minor modification in the build) based on NOAA-EMC repositories. This will make pushing any changes to UFS Weather Model easy. At this point, UFS Coastal points to feature/coastalapp branch for all of them. In case of updating UFS Coastal and sync with UFS Weather Model, we will use those forks to bring new developments. I'll create others if I need it.

uturuncoglu commented 8 months ago

@saeed-moghimi-noaa @pvelissariou1 @yunfangsun I updated the wiki page and also included list of tests and their status. You could see it in the following link,

https://github.com/oceanmodeling/ufs-coastal/wiki/Current-Status-of-UFS%E2%80%90Coastal-Implementation

Let's try to keep this up-to-date. BTW, we could see the progress in the development. Let me know if you have any other idea or information that can be put in to the wiki page.

uturuncoglu commented 8 months ago

@saeed-moghimi-noaa @pvelissariou1 JFYI, I created an issue in UFS Weather Model level about ESMX implementation. This can be found in here https://github.com/ufs-community/ufs-weather-model/issues/1961

oceanmodeling / ufs-weather-model

Model level issues/discussions #8