geoschem / GCHP

The "superproject" wrapper repository for GCHP, the high-performance instance of the GEOS-Chem chemical-transport model.
https://gchp.readthedocs.io
Other
23 stars 25 forks source link

Turning on UVFlux diagnostic collection results in a floating point error #436

Open yantosca opened 3 months ago

yantosca commented 3 months ago

Your name

Bob Yantosca

Your affiliation

Harvard + GCST

What happened? What did you expect to happen?

While adding an integration test with all diagnostics activated (see https://github.com/geoschem/geos-chem/pull/2399), I ran into this error when the UVFlux diagnostic collection was activated:

Backtrace for this error:
#0  0x14cdaebb85af in ???
#1  0x982696 in __cldj_interface_mod_MOD_run_cloudj
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/cldj_interface_mod.F90:953
#2  0x6c45b2 in __photolysis_mod_MOD_do_photolysis
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/photolysis_mod.F90:601
#3  0x577744 in __fullchem_mod_MOD_do_fullchem
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/fullchem_mod.F90:395
#4  0x4d708c in __chemistry_mod_MOD_do_chemistry
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/chemistry_mod.F90:434
#5  0x4b3cbf in __gchp_chunk_mod_MOD_gchp_chunk_run
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/Interfaces/GCHP/gchp_chunk_mod.F90:1357
#6  0x49752f in run_
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/Interfaces/GCHP/Chem_GridCompMod.F90:2880
#7  0x4a2204 in run2
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/Interfaces/GCHP/Chem_GridCompMod.F90:2052

This does not seem to be an error in GCClassic. The error happens is in this section of code (precisely, at line 953) in GeosCore/cldj_interface_mod.F90 (using Cloud-J 7.7.2):

https://github.com/geoschem/geos-chem/blob/9d766c47cd37393cb2bf94158ba9ed2dc445e316/GeosCore/cldj_interface_mod.F90#L940-L956

As @lizziel has mentioned in the comment, something is not quite right in this diagnostic.

I will leave this as a reminder for us to look into this as time allows. We haven't had a lot of demand for the UVFlux diagnostic recently.

What are the steps to reproduce the bug?

  1. Clone GCHP
  2. Create a fullchem_benchmark run directory and navigate to it.
  3. Change #'UVFlux', to 'UVFlux' in the COLLECTIONS list of HISTORY.rc
  4. Compile GCHP
  5. Run GCHP

Please attach any relevant configuration and log files.

The error trace was:

Backtrace for this error:
#0  0x14cdaebb85af in ???
#1  0x982696 in __cldj_interface_mod_MOD_run_cloudj
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/cldj_interface_mod.F90:953
#2  0x6c45b2 in __photolysis_mod_MOD_do_photolysis
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/photolysis_mod.F90:601
#3  0x577744 in __fullchem_mod_MOD_do_fullchem
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/fullchem_mod.F90:395
#4  0x4d708c in __chemistry_mod_MOD_do_chemistry
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/GeosCore/chemistry_mod.F90:434
#5  0x4b3cbf in __gchp_chunk_mod_MOD_gchp_chunk_run
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/Interfaces/GCHP/gchp_chunk_mod.F90:1357
#6  0x49752f in run_
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/Interfaces/GCHP/Chem_GridCompMod.F90:2880
#7  0x4a2204 in run2
        at /n/holyscratch01/jacob_lab/ryantosca/tests/nodiff/gc2399/GCHP_it/CodeDir/src/GCHP_GridComp/GEOSChem_GridComp/geos-chem/Interfaces/GCHP/Chem_GridCompMod.F90:2052

What GCHP version were you using?

14.4.2 + feature/all-diags-inttest branch

What environment were you running GCHP on?

Local cluster

What compiler and version were you using?

gcc 10.2.0 + Harvard 10.2.0 environment

What MPI library and version were you using?

OpenMPI 4.1.0

Will you be addressing this bug yourself?

Yes, but I will need some help

Additional information

No response

lizziel commented 1 month ago

I created a Cloud-J feature request to output the needed arrays from Cloud-J. https://github.com/geoschem/Cloud-J/issues/28