[BUG/ISSUE] some species is not found

[zhendonglu]

Describe the bug

There is something wrong with GFED-4 extension in HEMCO when I tried to run Geos-Chem 12.7.1. In HEMCO.log it says some species is not found in GFED, but it cannot show what exactly it is. I believe I have downloaded all required emission data in GFED-4. I cannot figure out what happens here. Could you help me with that?

To Reproduce

Compilation commands

1. make -j4 build (without modifying Makefile nor source code)

Run commands

1. module load parallel_studio/2017.1 module load hdf/4.2.12_parallel_studio-2017.1 module load hdf5/1.8.18_parallel_studio-2017.1 module load hdfeos/2.19_parallel_studio-2017.1 module load hdfeos5/1.15_parallel_studio-2017.1 module load netcdf-fortran/4.4.4_parallel_studio-2017.1

2. dkh debug unlimit core size and force core dump

   ulimit

3. ulimit -s unlimited

   export KMP_STACKSIZE=512M

4. Set number of threads

   export OMP_NUM_THREADS=32

5. time ./geos > logfile

   Expected behavior

   The model should complete the simulation successfully and output the results.

Error messages

HEMCO.log:

Required information

Please include the following:

• GEOS-Chem version you are using 12.7.1
• Compiler version that you are using ifort 17.0.1
• netCDF and netCDF-Fortran library version netcdf-fortran/4.4.4_parallel_studio-2017.1
• Computational environment a cluster
• Are you using "out of the box" code, or have you made modifications? I used "out of the box" code.

Input and log files to attach

For more info, see: http://wiki.geos-chem.org/Submitting_GEOS-Chem_support_requests

• The lastbuild file lastbuild.txt
• The input.geos file input.geos.txt
• The HEMCO_Config.rc file HEMCO_Config.rc.txt
• The GEOS-Chem "Classic" log file logfile.txt
• The HEMCO.log file HEMCO.log
• Error output from your scheduler, if applicable [e.g. slurm*.out] 4289095.out.txt
  • Any other error messages 4289095.err.txt

Additional context

I tried to run the version 12.7.0 with the same procedure, but I am confronted with the same problem.

[msulprizio]

Another user also encountered this error and was able to identify and resolve it by running with debug flags turned on. Please see https://github.com/geoschem/geos-chem/issues/116.

[zhendonglu]

Another user also encountered this error and was able to identify and resolve it by running with debug flags turned on. Please see #116.

Thanks for your reply, but it doesn't work. I tried to add all the recommended debug flags to recompile GEOS-Chem, i.e. make -j4 MET=GEOSFP DEBUG=yes BOUNDS=yes FPEX=yes OMP=no build, but the error remains.

[msulprizio]

In HEMCO/Extensions/hcox_gfed_mod.F90 can you add a debug print statement to see what strings it is comparing when looking for matching species names? For example:

       ! Search for matching GFED species by name
       Matched = .FALSE.
       DO M = 1, N_SPEC

          ! Add this line for debugging
          Print*, 'Debug GFED ', M, TRIM(SpcName), TRIM(GFED_SPEC_NAME(M))
          IF ( TRIM(SpcName) == TRIM(GFED_SPEC_NAME(M)) ) THEN
             Inst%GfedIDs(N) = M
             Matched    = .TRUE.

Then recompile, rerun, and check your GEOS-Chem log file (because Print* output will go there, not HEMCO.log). Verbose I wonder if there's a tab or spacing issue when reading the GFED species from HEMCO_Config.rc.

[msulprizio]

Also make sure you do make realclean before recompiling with the debug options turned on.

[zhendonglu]

In HEMCO/Extensions/hcox_gfed_mod.F90 can you add a debug print statement to see what strings it is comparing when looking for matching species names? For example:

       ! Search for matching GFED species by name
       Matched = .FALSE.
       DO M = 1, N_SPEC

          ! Add this line for debugging
          Print*, 'Debug GFED ', M, TRIM(SpcName), TRIM(GFED_SPEC_NAME(M))
          IF ( TRIM(SpcName) == TRIM(GFED_SPEC_NAME(M)) ) THEN
             Inst%GfedIDs(N) = M
             Matched    = .TRUE.

Then recompile, rerun, and check your GEOS-Chem log file (because Print* output will go there, not HEMCO.log). Verbose I wonder if there's a tab or spacing issue when reading the GFED species from HEMCO_Config.rc.

I do make realclean before recompiling. In addition to your recommended debug, I also put another two debug print statement in HEMCO/Core/hco_state_mod.F90 around line 730 and line 770: your recommendation: hco_state_mod_f90_line 730: hco_state_mod_f90_line 770: And the logfile is this: logfile.txt

[zhendonglu]

As you can see, the output is quite wierd. The subroutine HCO_GetExtHcoID in hco_state_mod.F90 works at first several times when it's called by other modules, but it failed to read the speciesnames when it's called by the subroutine HCOX_GFED_Init in hcox_gfed_mod.F90

[msulprizio]

I think I see the issue. GFED_SPEC_NAME is filled with the values defined in hcox_gfed_include_gfed4.H. In that file there are 34 species defined but N_SPEC is set to 35 at the top of hcox_gfed_mod.F90. Judging by the comments, I think that change came into 12.7.0 with the methanol (MOH) chemistry updates. I believe that is causing the strange printout for value M=35 in your log file. Try changing to:

INTEGER, PARAMETER :: N_SPEC = 34

and see if that fixes your problem. If so, we can add this as a fix in 12.7.2.

[zhendonglu]

I think I see the issue. GFED_SPEC_NAME is filled with the values defined in hcox_gfed_include_gfed4.H. In that file there are 34 species defined but N_SPEC is set to 35 at the top of hcox_gfed_mod.F90. Judging by the comments, I think that change came into 12.7.0 with the methanol (MOH) chemistry updates. I believe that is causing the strange printout for value M=35 in your log file. Try changing to:

INTEGER, PARAMETER :: N_SPEC = 34

and see if that fixes your problem. If so, we can add this as a fix in 12.7.2.

No, this doesn't work. As I mentioned before, the SpcName read by subroutine HCO_GetExtHcoID in hco_state_mod.F90 also has something wrong, not only GFED_SPEC_NAME:

[zhendonglu]

I tried ifort 17.0.4, then this error disappear now.

[msulprizio]

Glad you were able to resolve the issue. We will note that there may be issues with that compiler version.

[bsuther]

Hello, I am wondering if anyone ever figured out a solution to this other than switching the compiler? I am working with our system admin to get a different compiler on our system, but if there is a different solution which allows you to use ifort 17.0.1 and not get this error I would love to hear it. For reference I am getting this error when I run GEOS-Chem version 13.1.0 with boundary conditions and rrtmg turned on. Thanks!