search

geoschem / geos-chem

GEOS-Chem "Science Codebase" repository. Contains GEOS-Chem science routines, run directory generation scripts, and interface code. This repository is used as a submodule within the GCClassic and GCHP wrappers, as well as in other modeling contexts (external ESMs).
http://geos-chem.org
Other
169 stars 165 forks source link

TOMAS slow integration speed since v14.4.3 #2556

Open BettyCroft opened 2 weeks ago

BettyCroft commented 2 weeks ago

Your name

Betty Croft

Your affiliation

Washington University in St. Louis

What happened? What did you expect to happen?

TOMAS in v14.4.3 with GEOS-Chem in Classic (single-node) configuration appears to integrate at a speed that is > 10-fold slower than expected. TOMAS speed in v14.4.2 appears unaffected. Could there be a potential parallelization issue with TOMAS since v14.4.3, as was noted in #2413. I am not sure if this problem is arising from HEMCO or geos-chem. This slowness was also noted by other TOMAS users, @njune74, @samuelod-atmos, and @theloniuspunk. I would welcome advice on how to address this issue.

What are the steps to reproduce the bug?

Steps to reproduce the bug. Select -DTOMAS=y -DTOMAS_BINS=15 (or -DTOMAS=y -DTOMAS_BINS=40) when configuring the build, followed by compiling GEOS-Chem in classic configuration. Start integration of the model in a TOMAS run directory, set up for 4x5 with MERRA2. The integration appears to proceed at about 3 days/1 day real time - whereas expected speed is slightly faster than 30 days/1 day real time with this configuration.

Please attach any relevant configuration and log files.

No response

What GCClassic version were you using?

14.4.3

What environment were you running GCClassic on?

Local cluster

What compiler and version were you using?

gcc12.2.0

Will you be addressing this bug yourself?

No

Additional information

I am happy to assist with testing any advice on fixes. Related issue with GCHP (https://github.com/geoschem/GCHP/issues/457)

BettyCroft commented 2 weeks ago

Meant to add link to PR #2413 where possible parallelization error for TOMAS was noted in this update for v14.4.3

yantosca commented 1 week ago

Thanks @BettyCroft for raising this issue. I wonder if it could be from the unit conversion updates in commit 1fb6dd10e54505e44aae70267b149fed427e5f28 and dda6a6b798be46078a861cffa5a3d1a6bb415cc2.

Perhaps all of these unit checks are slowing down the code. We saw a similar thing in the photolysis code. Tagging @lizziel.

lizziel commented 1 week ago

Unit code that occurs in (I,J,L) loops will cause slow-down. Was anything like that added to the model in the first version that is slow?

theloniuspunk commented 1 week ago

Hi Lizzie, are you asking if there were any TOMAS-specific updates with new i,j,l loops in v14.4.3? It doesn't look like there were any TOMAS-specific updates for this version: https://wiki.seas.harvard.edu/geos-chem/index.php/GEOS-Chem_14.4.3

Jeff

lizziel commented 1 week ago

Yes, that question was geared to @yantosca about the changes he added in those commits.

@BettyCroft, do you have a log file you could share. The end of the log file contains timing. We should be able to see where the slow-down is from that information, particularly if compared to the same timing output from a previous version.

BettyCroft commented 1 week ago

Hi @lizziel - I have a log file for TOMAS in GCClassic v14.4.3 - but not for GCHP. Were you requesting a v14.4.3 GCHP-TOMAS log file with timing info at the end? So far GCHP with TOMAS is failing at the point of starting to time step in v14.4.3. However v14.4.2 appears to be time-stepping successfully - so it seems something happened between v14.4.2 and v14.4.3.