search

JCSDA / ufs-bundle

Bundle for interfacing UFS models with JEDI interfaces
GNU Lesser General Public License v2.1
0 stars 3 forks source link

Build, compile and execute a version UFS with aerosol and chemistry #8

Closed jeromebarre closed 1 week ago

jeromebarre commented 1 year ago

Build, compile and execute a version UFS with aerosol (GOCART or GOCART2) and chemistry (RAQMS or else).

Steps:

since I cannot put assignees right now I tag people here: @SarahLu-NOAA @jeromebarre @climbfuji

climbfuji commented 1 year ago

@jeromebarre Try again please.

climbfuji commented 1 year ago

@jeromebarre I don't know how realistic this is until the Skylab v4 release. Our current plan is to be able to build and run unit tests for ATM, NG-GODAS and S2S, in addition to running the UFS ATM in Skylab v4 via EWOK. This is already a stretch, and given the complexity of getting all the input data in place and compiling the UFS with the aerosol options turned on, I'd say this is more realistic for Skylab v5.

jeromebarre commented 1 year ago

How can we help you?

climbfuji commented 1 year ago

The current set of PRs compiles the UFS ATM, NG-GODAS and S2S applications on macOS. It also compiles the ATM on the AWS Parallelcluster, but NG-GODAS and S2S fail with a linker error (issues with fPIC, static libraries etc). This needs to be fixed first.

Second, we have working ATM ctests, which Clementine is using as blueprint for setting up the EWOK experiments. The existing NG-GODAS ctests are broken since we updated MOM6, CICE6 etc. to stay in sync with the ufs-weather-model. These need to be fixed.

Third, we need a set of input data (a run directory will do) for the S2S configuration and get those into the ctests.

Fourth, we need to add the S2SA or S2SWA or whatever the exact aerosol configuration (application in ufs-weather-model lingo) is that you need, and add a ctest for those. This requires fixing a few build errors related to GOCART-MAPL-ESMF. See https://github.com/ufs-community/ufs-weather-model/issues/1498 for more information.

climbfuji commented 1 year ago

And this is the issue that needs to be addressed ASAP, before/in parallel with items 1-3 above: https://github.com/jcsda/ufs-bundle/issues/19

jeromebarre commented 1 year ago

Just to recap on our recent conversations @climbfuji . ATMAERO is needed not S2SWA. However S2SWA would be needed in the future if we plan to run aerosol with the coupled model. For now ATMAERO is the priority for composition.

climbfuji commented 1 year ago

@jeromebarre I have a draft PR that I am testing with the other UFS configurations on my Mac: https://github.com/JCSDA/ufs-bundle/pull/21. This will address step one of this issue, building the UFS APP ATMAERO in ufs-bundle.

jeromebarre commented 1 year ago

Great! Thx for looking into this.

climbfuji commented 1 year ago

https://github.com/JCSDA/ufs-bundle/pull/21 was merged. Next step is to add a ctest for atmaero (ideally using the existing UFS c48 warmstart ICs, just adding the additional data needed for the aerosols? Is this something that @jeromebarre and/or @mer-a-o can do?

jeromebarre commented 1 year ago

I can do that. If a warmstart and then IC files are needed, I need to get the info from @SarahLu-NOAA that managed to run a UFS run with the NOAA workflow. I will try to get this info.

climbfuji commented 1 year ago

I maybe be able to help with getting those files if needed.

jeromebarre commented 1 year ago

Yes that would be great. We'd need emission files and IC files. Let me check with @SarahLu-NOAA first.

SarahLu-NOAA commented 1 year ago

At Orion, CEDS emissions arch covers 2011-2020 and QFED archive covers 2011-2019.
I test run (UFSAero, cold started from met-only GDAS) is for 2019-08-01. I'm in the process of reactivating my orion account, so can't access my note at Orion at this moment.

SarahLu-NOAA commented 1 year ago

@jeromebarre @climbfuji GOCART emissions files can be found at: HERA: /scratch1/NCEPDEV/global/glopara/data/gocart_emissions ORION: /work2/noaa/global/wkolczyn/noscrub/global-workflow/gocart_emissions S4: /data/prod/glopara/gocart_emissions

The path to the input data (AERO_INPUTS_DIR) is specified in global-workflow/parm/config/config.aero

NOTE QFED emissions cover 2011-2019 and CEDS emissions cover 2011-2020.
I did one ATMAERO cold-start test run (SDATE=20190801) at Orion mid Feb and Hera this week.

The ICs are fetched from HPSS: /NCEPPROD/hpssprod/runhistory/rh2019/201908/20190801/gpfs_dell1_nco_ops_com_gfs_prod_gfs.20190801_00.gfs_nemsioa.tar

You can copy gfs.t00z.atmanl.nemsio and gfs.t00z.sfcanl.nemsio from HERA: /scratch1/NCEPDEV/global/Sarah.Lu/save/UFS/ICs/gfs.20190801. This is v15 ICs, so global_workflow call this routine to prepare ICs: global-workflow/sorc/ufs_utils.fd/util/gdas_init/run_v15.chgres.gfs.sh

SarahLu-NOAA commented 1 year ago

My test run uses QFED emissions. AERO_EMIS_FIRE=QFED, specified in global-workflow/parm/config/config.aero

Once QFED is selected, UFS/global-workflow/parm/chem/ExtData.qfed will specify the emissions files for SU, OC, BC, and NH3 global-workflow/parm/chem directory contain resource parameters for running GOCART

SarahLu-NOAA commented 1 year ago

The global_workflow link input data and copy config files for GOCART during run-time. @climbfuji Ping me if you need additional information or have any questions. I can test your workflow at HERA, S4, or Cheyenne. I can't access ORION at this moment.

jeromebarre commented 1 year ago

@SarahLu-NOAA thanks! with that I think I will be able to set up a test. Likely next week.

SarahLu-NOAA commented 1 year ago

@climbfuji @mer-a-o Dom, For your EWOK, where are fixed files directories (fix/am, fix/orog ... etc) are located at S4, Orion, and Cheyenne? Thanks.

climbfuji commented 1 year ago

See https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/using/building_and_running/running_skylab.html#create-and-source-jedi-root-activate-sh please. In the script activate.sh, an environment variable EWOK_STATIC_DATA is set, dependent on the machine.

SarahLu-NOAA commented 1 year ago

Thanks, Dom. @mer-a-o From activate.sh, the fixed files at s4 can be found at /data/prod/jedi/static/skylab-4.0.0/ufs For instance, /data/prod/jedi/static/skylab-4.0.0/ufs/c384/global_snoclim.1.875.grb is identical to /data/users/clu/UFS/fix/am/global_snoclim.1.875.grb

SarahLu-NOAA commented 1 year ago

@mer-a-o To run aerosols, ExtData directory needs to be copied to the run directory ExtData can be found at S4: /data/prod/glopara/gocart_emissions

SarahLu-NOAA commented 1 year ago

@mer-a-o @jeromebarre will start with reviewing Maryam's note on her efforts

jeromebarre commented 1 year ago

@gibbsp @symoore90 please move this to aop23