search

geoschem / geos-chem

GEOS-Chem "Science Codebase" repository. Contains GEOS-Chem science routines, run directory generation scripts, and interface code. This repository is used as a submodule within the GCClassic and GCHP wrappers, as well as in other modeling contexts (external ESMs).
http://geos-chem.org
Other
164 stars 156 forks source link

[BUG/ISSUE] Cannot get field CMIP6_Sfc_CH4 #250

Closed bsuther closed 4 years ago

bsuther commented 4 years ago

Describe the bug

I am doing a North America nested grid simulation and GEOS-Chem seems to be trying to use the CMIP6 data from meinshausen et al. instead of the NOAA spatially resolved data. It is my understanding that since I am doing a simulation for 2014 it should be using the NOAA data, so I am not sure why it is trying to find the CMIP6 data at all. I used similar settings on the global run I used to get the boundary conditions for this simulation and did not encounter this error.

To Reproduce

Run a North American 0.25x0.3125 nested grid simulation for July 2014 with rrtmg turned on. The executable was copied from the global run used to get the boundary conditions.

Compilation commands 1.In global run directory: make -j4 build RRTMG=y

  1. copy to nested grid run

Run commands

#BSUB -W 10080
#BSUB -n 12
#BSUB -R span[hosts=1]
#BSUB -R "rusage[mem=99000]"
#BSUB -J GeosMP_na
#BSUB -o /gpfs_common/share02/meskhidze/bsuther/GEOSchem/rundirs/geosfp_025x03125_RRTMG_na/logfiles/out.%J

 source ~/Enviroment.sh

cd /share/meskhidze/bsuther/GEOSchem/rundirs/geosfp_025x03125_RRTMG_na
./geos >> GC.log
submit_GC_run_rrtmg_na.csh (END)

bsub < submit_GC_run_rrtmg_na.csh

Expected behavior

I expect it to use the NOAA dataset for CH4, not the CMIP6 data set since I am not doing a pre 1978 simulation.

Error messages

If you run with 

    --> GMD_SFC_CH4            :       false

in HEMCO_Config you get:

HEMCO ERROR: Container not found: CMIP6_Sfc_CH4
ERROR LOCATION: HCO_GetPtr_2D (hco_emislist_mod.F90)

If you run with 

    --> GMD_SFC_CH4            :       true

in HEMCO_Config you get:

 Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: CMIP6_Sfc_CH4
 Error in ReadList_Fill (3) called from HEMCO ReadList_Read
 Error in ReadList_Read called from hco_run
===============================================================================
GEOS-Chem ERROR: Error encountered in "HCO_Run"!
 -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90)

THIS ERROR ORIGINATED IN HEMCO!  Please check the HEMCO log file for
additional error messages!
===============================================================================

===============================================================================
GEOS-Chem ERROR: Error encountered in "HCOI_GC_Run"!
 -> at Emissions_Run (in module GeosCore/emissions_mod.F90)
===============================================================================

===============================================================================
GEOS-CHEM ERROR: Error encountered in "Emissions_Run", Phase 0

This error can indicate a missing file. Please check the HEMCO log file for
additional error messages!

STOP at  -> at GEOS-Chem (in GeosCore/main.F)

and in the HEMCO.log

HEMCO ERROR: Cannot find file for current simulation time: /gpfs_common/share02/meskhidze/bsuther/GEOSchem/InputData/gcgrid/data/ExtData/HEMCO/CMIP6/v2019-12/CMIP6_GHG_surface_VMR_1978.nc - Cannot get field CMIP6_Sfc_CH4. Please check file name and time (
ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90)
ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90)
ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90)
ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90)
ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
HEMCO ERROR: Error encountered in routine HCOIO_Read_Std!
ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90)
ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90)
ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90)
ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90)
ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

Required information

Please include the following: GEOS-Chem version you are using: 12.7.1 Compiler version that you are using: gfortran 9.2.0 netCDF and netCDF-Fortran library version: 4.6.3-gcc4.8.5 Computational environment: cluster Are you using "out of the box" code, or have you made modifications? Mainly out of the box, but I modified line 64 of GeosRad/Makefile from OPT := -O1 -override-limits to OPT += -fno-var-tracking-assignments because I was previously getting the error

"-DBPCH_TPBC -DRRTMG -DNC_HAS_COMPRESSION -DUSE_REAL8 -J../mod
-I/usr/local/apps/netcdf-centos7/4.6.3-gcc4.8.5/include -fdefault-real-8
-fdefault-double-8 -c -O1 -override-limits rrtmg_sw_rad.F90
rrtmg_lw_k_g.F90:4298:0:
4298 | subroutine lw_kgb03
|
note: variable tracking size limit exceeded with
‘-fvar-tracking-assignments’, retrying without
ar crs librad.a mcica_subcol_gen_lw.o ..."

Input and log files to attach

input_and_log_files.zip

bsuther commented 4 years ago

Of course after half a day of googling I find this https://github.com/geoschem/geos-chem/issues/233 immediately after starting a request. I will try to see if their solution works for me as well. Thanks.

yantosca commented 4 years ago

No worries! Yes, we had an issue with the timestamps in the CMIP6 files that the developer sent us. That is going to be fixed in 12.8.0 -- but you can change the file paths to HEMCO/CMIP6/v2020-03 in your HEMCO_Config.rc file and then do a dry-run to get those.

12.8.0 should go into benchmarking soon (as I write this). Not sure how long it will take to clear benchmarking.

yantosca commented 4 years ago

Pro tip - if you type a subject line in the issue, Github will present you with a list of similar issues that you can check out.

bsuther commented 4 years ago

I do not understand why changing the file paths is necessary if it should not be using those files anyway. Am I wrong that a 2014 run should not need the CMIP6 data?

Thanks, I will try that next time!

bsuther commented 4 years ago

I tried adjusting the filepaths, but I am still getting the error

---> DATE: 2014/07/01 UTC: 00:00 X-HRS: 0.000000 HEMCO already called for this timestep. Returning.

GEOS-Chem ERROR: Cannot get pointer to CMIP6_Sfc_CH4! Make sure it is turned on in HEMCO_Config.rc. -> at SET_CH4 (in module GeosCore/set_global_ch4_mod.F90)

Which makes sense because I have --> CMIP6_SFC_CH4 : false So I am still not sure why it is trying to access the CMIP6 data at all.

yantosca commented 4 years ago

Ah, I think I may see it. In set_global_ch4_mod.F90:

    IF ( FIRST ) THEN

       ! Get species ID
       id_CH4 = Ind_( 'CH4' )

       ! Get pointer to surface CH4 data
       IF ( HcoState%Clock%ThisEYear > 1978 ) THEN
           ! Use the NOAA spatially resolved data where available
          CALL HCO_GetPtr( HcoState, 'NOAA_GMD_CH4', SFC_CH4, RC )
          IF ( RC /= GC_SUCCESS ) THEN
             ErrMsg = 'Cannot get pointer to NOAA_GMD_CH4! Make sure it is turned on in HEMCO_Config.rc.'
             CALL GC_Error( ErrMsg, RC, ThisLoc )
             RETURN
          ENDIF
       ELSE
          ! Use the CMIP6 data from Meinshausen et al. 2017, GMD
          ! https://doi.org/10.5194/gmd-10-2057-2017a
          CALL HCO_GetPtr( HcoState, 'CMIP6_Sfc_CH4', SFC_CH4, RC )
          IF ( RC /= GC_SUCCESS ) THEN
             ErrMsg = 'Cannot get pointer to CMIP6_Sfc_CH4! Make sure it is turned on in HEMCO_Config.rc.'
             CALL GC_Error( ErrMsg, RC, ThisLoc )
             RETURN
          ENDIF
       ENDIF

       ! Reset first-time flag
       FIRST = .FALSE.

    ENDIF

So this is attempting to set the pointer solely on whether the year is > 1978 or not.

Also, I think there is an error there as well it is looking for CMIP6_Sfc_CH4 but the HEMCO bracket name is CMIP6_SFC_CH4.

@msulprizio: any thoughts?

yantosca commented 4 years ago

Also -- put a print statement for HcoState%Clock%ThisEYear and see what it is in your code. That might shed some more light on what is going on.

lizziel commented 4 years ago

There are a couple things related to these input files and their usage which can lead to this confusion.

  1. SET_CH4 is set up to always expect GMD surface methane to be available if running after 1978. Prior to then it will expect CMIP6. This is despite settings in HEMCO_Config.rc which dictates purely which files HEMCO should read and make available.

  2. 12.7.0 and 12.7.1 have a bug in SET_CH4 where the message Cannot get pointer to CMIP6_Sfc_CH4! is used as the error message if trying to use either CMIP6 or GMD surface methane. So if you are running 2014 and have GMD surface methane off, then it can't find the data (due to point 1 above) and errors with a message that it couldn't get a pointer to CMIP6. This is fixed in 12.7.2.

So in summary, for a 2014 run with 12.7.1:

Does the combination GMD_SFC_CH4=True and CMIP6_Sfc_CH4=False work for you? It is possible there is a problem reading the CMIP6 file on our end too since I'm not sure we've tested with 1978 or prior.

lizziel commented 4 years ago

I just checked our default HEMCO_Config.rc files for all simulations and we indeed have CMIP6 surface methane turned off for all simulations. This means a file read issue would not have shown up in our unit tests. Just to reiterate, CMIP6 should not be used for years 1978 and prior, but the setting in HEMCO_Config.rc will turn on file read anyway if you switch CMIP6_SFC_CH4 to true. We can check to see if we have a problem reading the file too on our end.

bsuther commented 4 years ago

I am getting the error with --> GMD_SFC_CH4 : true --> CMIP6_SFC_CH4 : false

I added the print statement "print*, 'HCoState Clock ThisEYear is', HcoState%Clock%ThisEYear" and got "HCoState Clock ThisEYear is -1" So that does explain why it is trying to get the CMIP6 data.

lizziel commented 4 years ago

@christophkeller I remember you were surprised when I said it returned the current year in GC Classic when I was looking into why it was returning -1 for GCHP. Do you remember more about this? I found it was returning -1 in GCHP because the first HEMCO run was not being called in the first timestep due to a bug, but this hasn't been the case for our GC Classic test runs.

lizziel commented 4 years ago

@bsuther yes, this definitely explains why it is trying to get the CMIP6 pointer.

lizziel commented 4 years ago

Here is a workaround from @christophkeller that we should consider adding into the model.

I think the cleaner solution in set_global_ch4_mod.F90 anyways should be to attempt to get the pointer to NOAA_GMD_CH4 no matter what, and only fall back to CMIP6_Sfc_CH4 if that failed:

LOGICAL :: FOUND
…
CALL HCO_GetPtr( am_I_Root, HcoState, ‘NOAA_GMD_CH4’, SFC_CH4, FOUND=FOUND, RC )
If ( .NOT. FOUND ) THEN
   CALL HCO_GetPtr( am_I_Root, HcoState, ‘CMIP6_Sfc_CH4’, SFC_CH4, RC )
ENDIF
lizziel commented 4 years ago

I will put this update into 12.8.

lizziel commented 4 years ago

This update is applied in commit https://github.com/geoschem/geos-chem/commit/51367546b2cde4ccbe9dabc27e2ffd3fcc9ae661.

bsuther commented 4 years ago

I tried the modification you suggested and am getting this error when I try to compile.

gfortran -cpp -w -std=legacy -fautomatic -fno-align-commons -fconvert=big-endian -fno-range-check -O3 -funroll-loops -fopenmp -mcmodel=medium -fbacktrace -g -DMODEL_CLASSIC -DLINUX_GFORTRAN -DBPCH_DIAG -DRRTMG -DNC_HAS_COMPRESSION -DUSE_REAL8 -J../mod -I/usr/local/apps/netcdf-centos7/4.6.3-gcc4.8.5/include -c -ffree-form -ffree-line-length-none set_global_ch4_mod.F90 set_global_ch4_mod.F90:152:82:

152 | CALL HCO_GetPtr( am_I_Root, HcoState, 'NOAA_GMD_CH4', SFC_CH4, FOUND=FOUND, RC ) | 1 Error: Missing keyword name in actual argument list at (1) make[4]: [set_global_ch4_mod.o] Error 1 make[4]: Waiting for unfinished jobs....

I am not so well versed in fortran that the correct solution to this is obvious to me.

lizziel commented 4 years ago

Try putting FOUND=FOUND after RC, rather than before it. Optional arguments need to be listed last.

bsuther commented 4 years ago

That compiled. Thank you very much.

lizziel commented 4 years ago

Great! Let us know if you have any more issues. In the meantime I will assume things are okay now and will close this issue.

christophkeller commented 4 years ago

FWIW, I figured out why this returns -1 on the first call in GEOS-Chem classic: Subroutine SET_CH4 is called before the emissions are being called (in main.F), and so the emissions year during that first call is still -1. The proposed solution may be cleaner as it makes HEMCO control the CH4 boundary conditions file. Otherwise, we could also use the year of the current time step instead of the HEMCO emissions year within subroutine SET_CH4.

bsuther commented 4 years ago

It worked for a while but then failed at hour 22 of the first day. Here is the error I am getting now:

4362 GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/07/01 21:50 4363 ---> DATE: 2014/07/01 UTC: 21:55 X-HRS: 21.916666 4364 GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/07/01 21:55 4365 ---> DATE: 2014/07/01 UTC: 22:00 X-HRS: 22.000000 4366 HEMCO: Opening /gpfs_common/share02/meskhidze/bsuther/GEOSchem/rundirs/geosfp_2x25_RRTMG/OutputDir/GEOSChem.Boundary 4366 Conditions.20140701_0000z.nc4 4367 Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: BC_NO 4368 Error in ReadList_Fill (5) called from HEMCO ReadList_Read 4369 Error in ReadList_Read called from hco_run 4370 =============================================================================== 4371 GEOS-Chem ERROR: Error encountered in "HCO_Run"! 4372 -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90) 4373 4374 THIS ERROR ORIGINATED IN HEMCO! Please check the HEMCO log file for 4375 additional error messages! 4376 =============================================================================== 4377 4378 =============================================================================== 4379 GEOS-Chem ERROR: Error encountered in "HCOI_GC_Run"! 4380 -> at Emissions_Run (in module GeosCore/emissions_mod.F90) 4381 =============================================================================== 4382 4383 =============================================================================== 4384 GEOS-CHEM ERROR: Error encountered in "Emissions_Run"! 4385 STOP at -> at GEOS-Chem (in GeosCore/main.F) 4386 =============================================================================== 4387 - CLEANUP: deallocating arrays now... (END)

and in HEMCO.log

--> LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90) !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! HEMCO ERROR: Cannot find field with valid time stamp in /gpfs_common/share02/meskhidze/bsuther/GEOSchem/rundirs/geosfp_2x25_RRTMG/OutputDir/GEOSChem.BoundaryConditions.20140701_0000z.nc4 - Cannot get field BC_NO. Please check file name and time (incl. tim ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90) ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90) ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90) ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90) ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! HEMCO ERROR: Error encountered in routine HCOIO_Read_Std! ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90) ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90) ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90) ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90) ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

It seems like this is an unrelated error, but I thought it would be worth checking since it does seem to be a time indexing issue. I am not sure why it would work for the first 21 hours and then fail. I did check the boundary conditions file and I see nothing suspicious in there. I think I am getting a negative tidx1 in hcoio_read_std. Does that mean that it should actually be looking at the midnight time for the next day instead of the current days .nc file? Thanks again for all of you help!

lizziel commented 4 years ago

I think this issue is the same as here: https://github.com/geoschem/geos-chem/issues/226. Take a look at the issue resolution there and see if it works for you.