[QUESTION] Timing tests on version 13.2.1 nested grid 0.25x0.3125?

[tessac2]

Ask a question about GEOS-Chem:

I was wondering if there have been any timing tests on the nested grid simulation on GEOS-Chem version 13 over North America? I have been attempting to run this, and it is taking a very long time even when using the pre-cropped met fields. For instance, running the script for 48 hours only yields about 1 week of simulation. I was wondering if this timeframe is typical for a 0.25x0.3125 simulation, or if there is any advice to reduce the time requirements?

GC.log.201907 (2).txt

[yantosca]

Thanks for writing @tessac2. The run with the pre-cropped met fields should go faster than if you are reading the global met fields.

How many cores are you using? Have you set OMP_NUM_THREADS in your run environment? You can type echo $OMP_NUM_THREADS to check. You can try using more cores to get better speedup. But with the nested-grid simulation I/O will always be the biggest bottleneck.

We haven't been able to run timing tests but maybe with the 14.0.0 release upon us that would be a good opportunity to run a fresh set of them.

[yantosca]

Also tagging @Jourdan-He and @SaptSinha

[tessac2]

Thanks @yantosca for your quick reply! I have 20 OMP_NUM_THREADS, 1 node and 20 tasks per node.

[yantosca]

Thanks @tessac2. We also use SLURM here. Our Research Computing staff told us that it was better to use #SBATCH -c (aka --cpus-per-task) than #SBATCH -n (aka -ntasks-per-node).

The run scripts that we use locally typically follow the geoschem.run script (found in the runScriptSamples folder in your run directory:

#!/bin/bash

#SBATCH -c 8
#SBATCH -N 1
#SBATCH -t 0-12:00
#SBATCH -p huce_intel
#SBATCH --mem=15000
#SBATCH --mail-type=END

###############################################################################
### Sample GEOS-Chem run script for SLURM
### You can increase the number of cores with -c and memory with --mem,
### particularly if you are running at very fine resolution (e.g. nested-grid)
###############################################################################

# Set the proper # of threads for OpenMP
# SLURM_CPUS_PER_TASK ensures this matches the number you set with -c above
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Run GEOS_Chem.  The "time" command will return CPU and wall times.
# Stdout and stderr will be directed to the "GC.log" log file
# (you can change the log file name below if you wish)
srun -c $OMP_NUM_THREADS time -p ./gcclassic >> GC.log

# Exit normally
exit 0
#EOC

[yantosca]

@tessac2: also SLURM should be able to pick up your run directory environment settings. If not you can add a . ~/.bashrc into the run script and that will force it to apply your settings.

Also, you could try turning on the GEOS-Chem timers in input.geos and then do a short run (maybe a day or so). Then you'd get an idea of where the run is spending its time.

[tessac2]

Thank you so much! I will try this and then provide an update.

[yantosca]

Hi @tessac2... any luck getting your runs to go faster?

[tessac2]

Yes a bit faster, thank you! I turned on the timers for a 1 day timer and it seems to have sped up to about 3.2 hours/day (or ~15 day simulation in 48 hour period).

[yantosca]

Thanks @tessac2. This seems more in line with what I would expect for timings. HEMCO takes a significant fraction of time due to the I/O happening on only one core. Also note the KPP chemistry solver (FlexChem) is now faster than the Aerosol Chemistry. This is on our list of things to investigate.

Feel free to close out this issue if you wish. Thanks again!

[tessac2]

Thank you so much for your help @yantosca, I will close this question!