[QUESTION] Benchmarking GC 12.6.3 with 2x2.5 standard simulation

[msulprizio]

I am adding this issue on behalf of Yaoxian Huang (yaoxian.huang@wayne.edu):

After 3-month waiting, my group’s machine eventually arrived and now is up for running GEOS-Chem model. We are testing the running of GEOS-Chem version 12.6.3 at the horizontal resolution of 2 degrees latitude by 2.5 degrees longitude, with simulation type as ‘standard’. The reason why we are testing this resolution and model type is that we are going to generate model production at this model configuration in the future. The model runs successfully, however, we have concerns that it may be too slow. So I would like to compare our model simulation performance with that from Harvard.

For this benchmarking simulation, we assign one computing node, each node with 36 computing cores and a total of 200 GB memory for the runs (based on our university IT, it actually only occupies 8 computing cores). It takes about 0.94 wall hour to finish 1 model day, which is be equivalent to 28 wall hours for a simulation of one model month (14 days for a year simulation). Can you provide some information about the model performance specifically for model resolution of 2x2.5 for standard simulation type so that I can compare it with mine? We are figuring out solutions to speed up the simulation. I am not sure whether we could use multiple nodes to run GEOS-Chem? Thank you very much.

Best,

Yaoxian

---

Yaoxian Huang [Pronounced as 'Yao-Hsien Hung'] Assistant Professor Department of Civil and Environmental Engineering Wayne State University

[msulprizio]

Can you provide some information about the model performance specifically for model resolution of 2x2.5 for standard simulation type so that I can compare it with mine?

The GCST performs our GEOS-Chem benchmarks and timing tests at 4x5 resolution using the standard simulation, and we unfortunately do not have any timing results for 2x2.5 simulations. If you are interested the results from the 4x5 simulations, please see our Guide to GEOS-Chem performance for details on timing results. On that page, you will also find links to timing tests for past versions with results from many institutions. We have not performed timing tests since 12.0.0, but we will do so again when 13.0.0 is released (expected Spring 2020).

Please also see our wiki page on Speeding up GEOS-Chem. There you will find some tips that may help improve the model performance.

I am not sure whether we could use multiple nodes to run GEOS-Chem?

You can use GEOS-Chem High Performance (GCHP) to run GEOS-Chem on multiple nodes. GCHP takes advantage of MPI to communicate across nodes, while GEOS-Chem "Classic" still uses OpenMP which is only efficient on one node. For more information, see Getting Started with GCHP.

[yantosca]

Some of the GEOS-Chem 12 versions were known to be slower than prior versions. This is partially due to more complex chemistry being added. It also seems that the KPP chemistry solver is not properly load-balanced, and this is the subject of further research. We also suspect that using HEMCO to read hourly input data (met fields, offline emissions, etc) has caused some of the slowdown.

We have started to make modifications to GEOS-Chem 12.7.0 (which should ship later this month, if all goes well) to remove some of these computational bottlenecks. I will be doing more performance profiling with 12.7.0 so that we can see if it is really faster than older 12.. versions. Stay tuned!