Meteorology for GCHP advection

[lizziel]

Name and Institution

Name: Lizzie Lundgren Institution: Harvard University

New GCHP feature or discussion

This issue is to discuss current work related to meteorology used in GCHP advection. There are several things that I hope to get into version 14.3.0.

1. Validation of GCHP runs using hourly mass fluxes. All official benchmarks use 3-hourly winds instead. Hourly mass fluxes are available for GEOS-FP at C720 (limited time range) and GEOS-IT at C180 (met option to be available in 14.3.0). Mass fluxes are not available for MERRA2.
2. Implement mass flux regridding update in MAPL. This update from @sdeastham is currently a MAPL PR pending review. The same update needs to be put into our MAPL fork which is an older version of MAPL than what the PR is based on.
3. Document resource constraints when using mass fluxes. See https://github.com/GEOS-ESM/MAPL/issues/2118.
4. The algorithm for computing dry pressure level edges for advection in the GCHPctmEnv gridded component needs an overhaul. We currently (1) sum moisture-corrected total pressure delta across all levels to get surface dry pressure and then (2) construct the 3D dry pressures from the surface dry pressure using Ap and Bp. This method should be compared with a direct computation of 3D dry pressure from 3D total pressure (no reconstruction from surface pressure using Ap/Bp).
5. Add pressure diagnostics in advection. These will appear in HISTORY.rc for gridded component DYNAMICS instead of GCHPchem.
6. Add budget transport diagnostics and/or vertical flux diagnostics per species.

Pinging @sdeastham and @1Dandan who will help with this work.

[1Dandan]

Thanks @lizziel for initiating the discussion. Just to confirm that I can check out the version of 14.2.0-rc.1to use dry pressure instead of total pressure for mass-flux simulations, right?

[lizziel]

Hi @1Dandan. Yes, you can checkout 14.2.0-rc.1 and use the transport tracer simulation. 14.2.0 is still in development because of an issue with full chemistry but the transport tracer simulation should work fine. We are seeing some wonky results for certain tracers in the stratosphere, e.g. SF6, but I think that is okay for the mass flux validation. @sdeastham, correct me if I am wrong on that.

[sdeastham]

That's correct! At this point we're on a fact-finding mission - let's at least start with 14.2.0-rc.1 and see what happens. We know there are issues no matter what we do, so we may as well try and quantify them.

[1Dandan]

Thanks for confirming. Then I'll go head to set up a simulation (intend to run for the year of 2022 at C24) and will let you know when it is finished.

[lizziel]

Sounds good. All of the runs we will do should be the following:

• Transport Tracer simulation
• 1 year duration (Jan 1 2022 - Jan 1 2023)
• Default diagnostics
• Same initial restart file as used for 14.2.0 transport tracer benchmark (GEOSChem.Restart.20190101_0000z.c24.nc4 located here)
• C24 grid resolution
• GEOS-Chem version 14.2.0-rc.1

I will do the wind runs at Harvard and @1Dandan will do the mass flux runs at WashU. Here is a summary of the first runs we will do.

1. @lizziel: "GEOS-FP" option when creating run directory. This uses 3-hourly winds in advection.
2. @1Dandan: "GEOS-FP native data -> Use mass fluxes? yes" option when creating run directory. This uses hourly C720 mass fluxes in advection.

Both of these runs use dry pressure in advection. The run directory differences are all in ExtData.rc and GCHP.rc, with the primary ExtData.rc differences being (1) fields used for advection (see below), and (2) raw files from GMAO versus processed files. The raw files and the processed files contain the same data but the processed files have the vertical level flipped and data concatenated into daily files. Raw files also handle optical depth differently. Here is a breakdown of the primary differences in ExtData.rc.

GEOS-FP processed and winds in advection

 --- Surface pressure, 3-hr instantaneous ---
PS1 hPa N Y 0        none 1.0 PS ./MetDir/%y4/%m2/GEOSFP.%y4%m2%d2.I3.025x03125.nc
PS2 hPa N Y 0;001000 none 1.0 PS ./MetDir/%y4/%m2/GEOSFP.%y4%m2%d2.I3.025x03125.nc

 --- 3D variables, 3-hr instantaneous ---
SPHU1 kg_kg-1 N Y 0        none none QV ./MetDir/%y4/%m2/GEOSFP.%y4%m2%d2.I3.025x03125.nc
SPHU2 kg_kg-1 N Y 0;001000 none none QV ./MetDir/%y4/%m2/GEOSFP.%y4%m2%d2.I3.025x03125.nc

 --- 3D variables, 3-hr averaged ---
UA;VA m_s-1 N Y F0;013000 none none U;V ./MetDir/%y4/%m2/GEOSFP.%y4%m2%d2.A3dyn.025x03125.nc

GEOS-FP native and mass fluxes in advection

MFXC;MFYC Pa_m+2_s-1    N H F0;003000 none  0.6666666 MFXC;MFYC  ./MetDir/../../GEOS_C720/GEOS_FP_Native/Y%y4/M%m2/D%d2/GEOS.fp.asm.tavg_1hr_ctm_c0720_v72.%y4%m2%d2_%h2%n2.V01.nc4 2021-03-11T00:30:00P01:00
CXC;CYC   1             N H F0;003000 none  none CX;CY           ./MetDir/../../GEOS_C720/GEOS_FP_Native/Y%y4/M%m2/D%d2/GEOS.fp.asm.tavg_1hr_ctm_c0720_v72.%y4%m2%d2_%h2%n2.V01.nc4 2021-03-11T00:30:00P01:00
PS1       Pa            N Y  0        none  0.01 PS              ./MetDir/../../GEOS_C720/GEOS_FP_Native/Y%y4/M%m2/D%d2/GEOS.fp.asm.inst_1hr_ctm_c0720_v72.%y4%m2%d2_%h2%n2.V01.nc4 2021-03-11T00:00:00P01:00
PS2       Pa            N Y  0;001000 none  0.01 PS              ./MetDir/../../GEOS_C720/GEOS_FP_Native/Y%y4/M%m2/D%d2/GEOS.fp.asm.inst_1hr_ctm_c0720_v72.%y4%m2%d2_%h2%n2.V01.nc4 2021-03-11T00:00:00P01:00
SPHU1     kg_kg-1       N Y  0        none  none QV              ./MetDir/../../GEOS_C720/GEOS_FP_Native/Y%y4/M%m2/D%d2/GEOS.fp.asm.inst_1hr_ctm_c0720_v72.%y4%m2%d2_%h2%n2.V01.nc4 2021-03-11T00:00:00P01:00
SPHU2     kg_kg-1       N Y  0;001000 none  none QV              ./MetDir/../../GEOS_C720/GEOS_FP_Native/Y%y4/M%m2/D%d2/GEOS.fp.asm.inst_1hr_ctm_c0720_v72.%y4%m2%d2_%h2%n2.V01.nc4 2021-03-11T00:00:00P01:00
UA;VA     m_s-1         N Y F0;013000 none  none U;V             ./MetDir/Y%y4/M%m2/D%d2/GEOS.fp.asm.tavg3_3d_asm_Nv.%y4%m2%d2_%h2%n2.V01.nc4 2014-02-11T01:30:00P03:00

OPTDEP      TAUCLI+TAUCLW 0

Here are the differences in GCHP.rc:

< METEOROLOGY_VERTICAL_INDEX_IS_TOP_DOWN: .false.
< IMPORT_MASS_FLUX_FROM_EXTDATA: .false.
---
> METEOROLOGY_VERTICAL_INDEX_IS_TOP_DOWN: .true.
> IMPORT_MASS_FLUX_FROM_EXTDATA: .true.

I have a few questions for @sdeastham:

1. Are these two runs sufficient to start?
2. Do you see any problems in the config file entries above ?
3. Should we also do a mass flux run with total pressure in advection?
4. Any other comments?

@1Dandan, do you have any questions?

[1Dandan]

Hi @lizziel, yes, I do have some questions. 1) Is there an option I need to turn on to use moisture-corrected mass flux or is it default? 2) I usually spin up for one month, it may not be sufficient if talking about concentrations in stratosphere. For the simulation period, do you want me to start just at Jan-01-2022 and see the evolving of mass conservation?

[lizziel]

Is there an option I need to turn on to use moisture-corrected mass flux or is it default?

Good question. The default is moisture-corrected mass flux, to go along with the default of using dry pressure. Is using moisture-corrected mass flux the best way? I'm not sure. @sdeastham, do you think we should try different permutations of moisture-corrected mass flux and dry/total pressure? If yes, which combinations?

I usually spin up for one month, it may not be sufficient if talking about concentrations in stratosphere. For the simulation period, do you want me to start just at Jan-01-2022 and see the evolving of mass conservation?

The 14.2.0 transport tracer restart file for 2019 was created from a 10-year run using 14.2.0. The GC-Classic restart file at the end was regridded to C24. I think this is sufficient for what we are trying to do (@sdeastham, tell us if you disagree).

[sdeastham]

Assuming that the moisture-corrected mass flux means the mass flux of dry air only (i.e. multiplying the mass flux by 1.0/(1-QV) where QV is taken from the upwind grid cell), then you would want to use the moisture-corrected flux for dry pressure advection and the original (total) flux for total pressure advection. I don't think that any other combination makes sense but happy to discuss!

As for the restart file - I think that's fine. Using a consistent set of initial conditions is all that really matters until we can get to the point where we can do something rigorous in comparison to GEOS. Starting from a high-res file and degrading to the target resolution would be better than starting from a GC-Classic file or low-res file, but I think the differences will be small.

[sdeastham]

As for the questions you posted @lizziel:

1. Are these two runs sufficient to start? Yes, although see 3
2. Do you see any problems in the config file entries above ? Nothing obvious
3. Should we also do a mass flux run with total pressure in advection? I think this is a good idea. We know it's got problems but data will help us in diagnosis
4. Any other comments? Thanks for leading the charge on this!

[lizziel]

Thanks @sdeastham! There will now be a third run to add to the list. I will summarize all three here to avoid confusion.

1. @lizziel: Dry pressure and 3-hourly winds in advection. "GEOS-FP" option when creating run directory. No manual changes needed.
2. @1Dandan: Dry pressure and moisture-corrected hourly C720 mass fluxes in advection. "GEOS-FP native data -> Use mass fluxes? yes" option when creating run directory. No manual changes needed.
3. @1Dandan: Total pressure and native hourly C720 mass fluxes (not moisure-corrected) in advection. "GEOS-FP native data -> Use mass fluxes? yes" option when creating run directory. Make manual changes in file GCHP.rc:

   USE_TOTAL_AIR_PRESSURE_IN_ADVECTION: 1
   CORRECT_MASS_FLUX_FOR_HUMIDITY: 0

I am still waiting for the restart file to show up on the Harvard ftp site. I will post here when it is available. I also want to double-check that the total air pressure and native mass flux options look okay before proceeding.

[lizziel]

The 14.2.0 transport tracer restart file to be used for the mass flux runs is now available. Download file GEOSChem.Restart.20190101_0000z.c24.nc4 at http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/14.2.0-rc.1/GCHP/TransportTracers/Restarts/.

[lizziel]

I am using the following libraries for the GEOS-FP winds run:

  1) gmp/6.2.1-fasrc01    5) openmpi/4.1.0-fasrc01   9) netcdf-c/4.8.0-fasrc01
  2) mpfr/4.1.0-fasrc01   6) zlib/1.2.11-fasrc01    10) netcdf-fortran/4.5.3-fasrc01
  3) mpc/1.2.1-fasrc01    7) szip/2.1.1-fasrc01     11) flex/2.6.4-fasrc01
  4) gcc/10.2.0-fasrc01   8) hdf5/1.10.7-fasrc01    12) cmake/3.25.2-fasrc01

I will use 96 cores across 2 nodes (48 cores per node) and enable monthly mid-run restart file.

[lizziel]

The dry pressure + winds run is now complete. @1Dandan, do you have an idea of when you will be able to do the mass flux runs? To share it with me you can post it at http://geoschemdata.wustl.edu/ExternalShare/.

[1Dandan]

Hi @lizziel, yes, it is running now and will probably take around 1 week to finish. It seems slower than wind runs. I will post the results at http://geoschemdata.wustl.edu/ExternalShare/ once they are ready.

[lizziel]

Great, thanks! It is also extra slow because you are using the native files instead of the usual processed files. The two sets of data are the same grid resolution but the processed are concatenated into daily files and many fewer collections. Opening and closing many files a day causes a performance hit.

[1Dandan]

@lizziel , thanks for your patience. Both simulations with total and moisture-corrected mass fluxes have been completed. I used same restart file as suggested. Configuration files, restarts and outputs are copied to: For total mass-flux run: http://geoschemdata.wustl.edu/ExternalShare/GCHP-v14.2.0-rc.1/rundir-TT-MF-c24-tot/ For moisture-corrected mass-flux run: http://geoschemdata.wustl.edu/ExternalShare/GCHP-v14.2.0-rc.1/rundir-TT-MF-c24-dry/ Let me know if you need any other files.

[lizziel]

Hi @1Dandan and @sdeastham. Unfortunately the recent 1-year fullchem benchmark for 14.2.0 showed a problem with the GC-Classic to GCHP restart file conversion that also impacts these runs that @1Dandan and I just did. GCPy was used for the first time to generate GCHP restart files and it went under the radar that the lev dimension retained the GC-Classic lev attribute positive "up". Upon GCHP read all 3D restart file variables are vertically flipped. Amazingly it does not crash the model, but does lead to incorrect values, particularly in the stratosphere for the transport tracer simulation.

Apologies that we will need to rerun these simulations. I will post where to get a new restart file once it is available. We are going back to using csregridtool to generate GCHP restarts for these benchmarks until GCPy GC-Classic to GCHP restart conversion is more thoroughly validated.

[1Dandan]

I see. I'll wait for the new restart file to rerun simulations.

[1Dandan]

Hi @lizziel, I am wondering if the new restart file is ready or not?

[lizziel]

Hi @1Dandan, yes, sorry for the radio silence! For the new runs we can use the restart file used for the 14.2.0 benchmarks found here: http://geoschemdata.wustl.edu/ExtData/GEOSCHEM_RESTARTS/GC_14.2.0/. Let's use the official 14.2.0 release for the runs since it is now released. Let me know if you have any questions.

[1Dandan]

Sure, I see that the GCHP v14.2.2 is released. @lizziel, do you want me to restart the transport tracer simulations with the official version, or is the version of v14.2.0-rc.1 also benign?

[lizziel]

We should use v14.2.2 since it will automatically links to the correct restart file and 14.2.1 includes a fix for using native GEOS-FP fields in the transport tracer simulation. Did you run into an error with the ocean mask in your last runs?

[1Dandan]

Sure, I'll use v14.2.2 then. I will let you know when the runs finished.

Did you run into an error with the ocean mask in your last runs?

Oh, yes. I forgot to let you know that I edited the line of ocean mask in ExtData.rc for v14.2.0-rc.1 as follows:

 #==============================================================================
 # Country/region masks
 #==============================================================================
 #OCEAN_MASK   1 N Y - none none FROCEAN    ./MetDir/2011/01/GEOSFP.20110101.CN.025x03125.nc
 OCEAN_MASK   1 N Y - none none FROCEAN    ./MetDir/GEOS.fp.asm.const_2d_asm_Nx.00000000_0000.V01.nc4
 #

[lizziel]

That's the bug. It should be fixed in 14.2.2. Let me know if you run into anything else that you need to fix. I can put any fixes into the next version.

[1Dandan]

Hi @lizziel, the two mass-flux transport tracer runs at C24 of dry pressure and total pressure have finished. They both run smoothly without any errors.

The mass-flux simulation results with dry pressure and moisture-corrected mass flux are at: http://geoschemdata.wustl.edu/ExternalShare/GCHP-v14.2.2/rundir-TT-MF-c24-dry/

The mass-flux simulation results with total pressure and non-corrected mass flux are at: http://geoschemdata.wustl.edu/ExternalShare/GCHP-v14.2.2/rundir-TT-MF-c24-tot/

Let me know if you need any other files.

[lizziel]

Excellent! I will download the data and make comparison plots.

[lizziel]

I generated comparison plots for (1) dry pressure versus total pressure, both using mass fluxes, and (2) winds versus mass fluxes, both using dry pressure. See https://ftp.as.harvard.edu/gcgrid/geos-chem/validation/gchp_mass_fluxes_vs_winds/.

[lizziel]

@1Dandan, I am looking into some discrepancies I see in GEOS-IT data and there is evidence suggesting we may be using the raw GMAO input files incorrectly, either in the config file or online. Would you be able to do the same run you did previously but using native lat-lon GMAO files and winds? My run represented lat-lon winds but I used processed files and we have limited space to download the raw files to the Harvard cluster.

[1Dandan]

@lizziel Sure. Just to confirm with you that: 1) I'll use v14.3.0 instead of v14.2.2, as it looks like GEOS-IT option is only available at v14.3.0. Would it be ok? Or should I copy ExtData.rc from v14.3.0 and stick to v14.2.2? 2) By native winds, you mean I choose GEOS-IT native lat-lon inputs at /ExtData/GEOS_0.5x0.625/GEOS_IT_Native, right? 3) Should I use the default dry pressure option?

[lizziel]

Hi @1Dandan, this is a GEOS-FP run so you can keep using the same code you were using before. I suspect that the issue is reading raw GMAO files in general and can confirm by looking at GEOS-FP raw vs processed winds runs.

When creating the run directory choose the following:

-----------------------------------------------------------
Choose simulation type:
-----------------------------------------------------------
   1. Full chemistry
   2. TransportTracers
   3. CO2 w/ CMS-Flux emissions
   4. Tagged O3
   5. Carbon
>>> 2

-----------------------------------------------------------
Choose meteorology source:
-----------------------------------------------------------
  1. MERRA-2 (Recommended)
  2. GEOS-FP 
  3. GEOS-FP native data
>>> 3
Do you want to use mass fluxes for advection? (yes/no, default=no): no
Do you want to use mass fluxes derived winds for advection? (yes/no, default=no): no

You don't need to rebuild the model if you plan on using the same libraries as before. Assuming your code and build directory are still where they were before, you can copy the executable from either your other run directory or build/bin/gchp.

Let me know if you have any questions.

[1Dandan]

I see. I'll set up the simulation and let you know when the output is ready.

[1Dandan]

Hi @lizziel, the simulation with native Geos-FP winds has been finished, whose outputs are available at: http://geoschemdata.wustl.edu/ExternalShare/GCHP-v14.2.2/rundir-TT-NWind-C24-dry/

The log file error is benign for the simulation period of 2022 to 2023 as I forgot to change the run duration when restarting it at the mid of 2022.

[lizziel]

Thanks @1Dandan! See new GCHP issue that came out of these runs: https://github.com/geoschem/GCHP/issues/386

I suggest not using raw GMAO data files in GCHP until this bug is fixed. Tagging @yidant about this as well to spread the word at WashU.

[lizziel]

@1Dandan and @yuanjianz, I am still have trouble testing the pressure fix because of a shortage of nodes on discover. Would you be able to redo the raw winds and mass fluxes GEOS-FP runs using GCHP branch dev/no-diff-to-benchmark? This contains the fix. I can then run the transport tracer benchmark comparisons on the results as I did last time. Let me know if you think you'll be able to do this and if you need any help setting it up.

[yuanjianz]

Sure. Is the setup the same as above? (i.e. 2022 Jan to 2023 Jan C24 GEOS-FP Transport Tracer with all default setting) @lizziel

[lizziel]

Yes, all the same except the GCHP version used. Thank you!

[lizziel]

Actually, I just realized the new advection diagnostics are about to merged in. Hold off until that happens, and then we can turn them on for these runs. I'll let you know when that happens.

[lizziel]

The diagnostics updates are now pushed to branch dev/no-diff-to-benchmark. @1Dandan, the version is ready now for you to run. The new diagnostics are on by default so you do not need to change anything for them.

The runs to do are:

• raw GMAO GEOS-FP using mass fluxes and dry pressure in advection
• raw GMAO GEOS-FP using winds and dry pressure in advection

We can skip total pressure runs unless you have time and @yuanjianz is interested in looking at results for that.

[yuanjianz]

Sure, already set up simulations and waiting in queue. I am running dry pressure first. @lizziel

[1Dandan]

Thanks @lizziel, @yuanjianz is interested in taking care of the two simulations and I passed the simulation setup I did previously to him.

[lizziel]

Great, thanks!

[lizziel]

@yuanjianz, can you confirm you are using the default 2019 c24 restart file that comes with the run directory? I am going to rerun my processed winds run with 14.3.1 for 2022 and want to make sure we use the same restart file.

[yuanjianz]

@lizziel, sure. I am using the 201901 restart file in the default restart folder.

[yuanjianz]

@lizziel, I am experiencing memory leaks and unusal low performances for my transport tracer simulation. I compared my log files with Dandan's last time. For cube sphere mass flux, her throughput is 3 times of mine and my memory statistics shows sharp increase. Please check my log file attached.(14.3 is mine and 14.2 is Dandan's) Currently I am trying to run another transport tracer under main branch to see if it is something introduced recently.

This happens to raw wind as well. My raw wind run dies of a signal 9 for OOM after 7 months of simulation. The log files show memory usage increases from 30% to 90% during the simulation.

14.3-gchp.20220101_0000z.log.txt 14.2-gchp.log.txt

P.S. I am running on 3 nodes, and each node is given 300GB of memory.

[lizziel]

Hi @yuanjianz, I will take a look and try a shorter test run on my end too.

[lizziel]

I don't see this in my processed winds 1-year run so that rules out the new diagnostics.

[lizziel]

I have a 1-month raw fields run in the queue. In the meantime, could you post your results for the first 6 months of the year? I can do a comparison separate from performance issues.

[yuanjianz]

@lizziel, here are my results for first 8 months with log files for first 6 months in log1 folder. http://geoschemdata.wustl.edu/ExternalShare/tt-geosfp-raw-wind/

[yuanjianz]

Hi @lizziel, the memory leak is probably related to my compiler and environment set-up. I tested processed wind between two compiler environment, and Dandan's environment would not have the memory leak issue on Compute1. I guess although there is an OOM issue with my previous gnu environment(which possibly originates from OFED), it would not cause much difference in the results. So I am just restarting with Dandan's intel environment. Please correct me if I am wrong.

[lizziel]

Hi @yuanjianz, I have not seen such a severe memory leak due to environment/system before, but it makes sense it would be this since I don't think the code changes would cause it. Could you give full details all libraries/system specs which result in the problem, along with libraries/system specs which do not? Regardless, as you say, the results should not be different. This can be tested by comparing the first month's data produced using the two different environments.