[BUG/ISSUE] Potentially incorrect warning about species initial values

JiaweiZhuang commented 5 years ago

I notice this warning message even if the restart file initial_GEOSChem_rst.c24_standard.nc exists:

 Initialized species from INTERNAL state: NO
    WARNING: using background values from species database
 Initialized species from INTERNAL state: O3
    WARNING: using background values from species database
 Initialized species from INTERNAL state: PAN
    WARNING: using background values from species database
 Initialized species from INTERNAL state: CO
    WARNING: using background values from species database

Is this a wrong warning? From the output files it seems like correct initial conditions are used.

See, for example, the log file ubuntu_timing_1day.log in #6

yantosca commented 5 years ago

This is happening in Chem_GridCompMod.F90:

         ! Set spc conc to background value if rst skipped or var not there
         IF ( RC /= ESMF_SUCCESS .OR. RST == MAPL_RestartBootstrap .OR.   &
                      RST == MAPL_RestartSkipInitial ) THEN
                DO L = 1, LLPAR 
                DO J = 1, JJPAR
                DO I = 1, IIPAR
                   ! Special handling for MOH (mimicking GEOS-Chem Classic)
                   IF ( TRIM( ThisSpc%Name ) == 'MOH' ) THEN
                      ! Test for altitude (L < 9 is always in the trop)
                      IF ( L <= 9 ) THEN
                         ! Test for ocean/land boxes
                         IF ( State_Met%FRCLND(I,J) >= 0.5 ) THEN
                            ! Continental boundary layer: 2 ppbv MOH
                            State_Chm%Species(I,J,L,IND) = 2.000e-9_fp
                         ELSE
                            ! Marine boundary layer: 0.9 ppbv MOH
                            State_Chm%Species(I,J,L,IND) = 0.900e-9_fp
                         ENDIF
                      ELSE
                         ! Test for troposphere
                         IF ( State_Met%InTroposphere(I,J,L) ) THEN
                            ! Free troposphere: 0.6 ppbv MOH
                            State_Chm%Species(I,J,L,IND) = 0.600e-9_fp 
                         ELSE
                            ! Strat/mesosphere:
                            State_Chm%Species(I,J,L,IND) = 1.0E-30_FP 
                         ENDIF
                      ENDIF
                   ELSEIF ( L > LLCHEM .AND. &
                            ( .NOT. ThisSpc%Is_Advected ) ) THEN
                      ! For non-advected spc at L > LLCHEM, use small number
                      State_Chm%Species(I,J,L,IND) = 1.0E-30_FP           
                   ELSE
                      ! For all other cases, use the background value in spc db
                      State_Chm%Species(I,J,L,IND) = ThisSpc%BackgroundVV 
                   ENDIF
                ENDDO
                ENDDO
                ENDDO
                Ptr3D_R8(:,:,:) = State_Chm%Species(:,:,LM:1:-1,IND)
                IF ( MAPL_am_I_Root()) THEN
                   WRITE(*,*)  &
                   '   WARNING: using background values from species database'
                ENDIF
             ENDIF

Maybe the warning message is useless by now and we could remove it. Did this come in with the GEOS-5 updates?

lizziel commented 5 years ago

I have not been able to reproduce these warning messages for advected species using a full chemistry simulation with GCHP on the Odyssey cluster. I only get these messages for internal state variables NOT in the restart file, such as recently added species pFe and HEMCO restart variables. This indicates the warnings are being issued correctly for my particular run. This code was put into GCHP in March 2018 and was not part of a GEOS-5 update.

I notice that your log file includes messages like this for every single internal state variable: Bootstrapping Variable: SPC_NMAO3 in initial_GEOSChem_rst.c24_standard.nc Bootstrapping Variable: SPC_NO in initial_GEOSChem_rst.c24_standard.nc Bootstrapping Variable: SPC_NO2 in initial_GEOSChem_rst.c24_standard.nc Bootstrapping Variable: SPC_NO3 in initial_GEOSChem_rst.c24_standard.nc Bootstrapping Variable: SPC_NPMN in initial_GEOSChem_rst.c24_standard.nc

If the restart file is read correctly then you should see a much shorter list, specifically this in 12.1.0, just with the restart filename swapped out:

Using parallel NetCDF for file: initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: ARCHV_DRY_TOTN in initial_GEOSChem_rst.c48_benchmark. nc Bootstrapping Variable: ARCHV_WET_TOTN in initial_GEOSChem_rst.c48_benchmark. nc Bootstrapping Variable: DEP_RESERVOIR in initial_GEOSChem_rst.c48_benchmark.n c Bootstrapping Variable: DRYPERIOD in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: GCCTROPP in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: GWET_PREV in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: LAI_PREVDAY in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: PARDF_DAVG in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: PARDR_DAVG in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: PFACTOR in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: SPC_LCH4 in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: SPC_LOx in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: SPC_PH2O2 in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: SPC_POx in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: SPC_pFe in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: STATE_PSC in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: T_DAVG in initial_GEOSChem_rst.c48_benchmark.nc Bootstrapping Variable: T_PREVDAY in initial_GEOSChem_rst.c48_benchmark.nc

Based on code in Chem_GridCompMod right before the conditional in which the background value warning message is issued, it appears that the call to ESMF_AttributeGet for the 'RESTART' attribute is returning MAPL_RestartBootstrap which is one of the conditions that will lead to using background values from the species database. We know the attribute is that value due to the bootstrapping warnings that are in the log file. Those warnings, along with the restart attribute per internal state variable, were created by MAPL earlier in the execution.

         ! Determine if species in restart file
         CALL ESMF_StateGet( INTERNAL, TRIM(SPFX) // TRIM(ThisSpc%Name),  &
              trcFIELD, RC=RC )
         CALL ESMF_AttributeGet( trcFIELD, NAME="RESTART",                &
              VALUE=RST, RC=STATUS )

         ! Set spc conc to background value if rst skipped or var not there
         IF ( RC /= ESMF_SUCCESS .OR. RST == MAPL_RestartBootstrap .OR.   &
                  RST == MAPL_RestartSkipInitial ) THEN

Check that your restart file exists in the run directory and contains the content you think it does.

JiaweiZhuang commented 5 years ago

Found the issue.

I was using the latest GEOSCHEM_RESTARTS/v2018-11/initial_GEOSChem_rst.c24_standard.nc, which uses variable name SpeciesRst_O3, but GCHP 12.1.0 still looks for SPC_O3. Using the old restart file GEOSCHEM_RESTARTS/v2016-07/initial_GEOSChem_rst.c24_standard.nc fixes the problem.

JiaweiZhuang commented 5 years ago

Oops the situation actually gets worse.

With the "proper" restart file v2016-07/initial_GEOSChem_rst.c24_standard.nc, the simulation always crashes at hour 1 (when the diagnostics first get written), even if I set the simulation period to 2 days. This is the default setting in ami-0a5973f14aad7413a

The simulation is able to proceed with the "improper" restart file v2018-11/initial_GEOSChem_rst.c24_standard.nc. This is the default setting in ami-0f44e999c80ef6e66.

lizziel commented 5 years ago

We should delete any GCHP restart files that are in that folder since the older files should be used instead. I will replace them with symbolic links to the older files in case people go to that folder looking for restarts. We do not plan on changing the internal state name in GCHP so SPC_ is the correct name to use. Thanks for reporting that!

Could you provide your log file for your latest run problem?

msulprizio commented 5 years ago

Hi all! I agree with Lizzie and I meant to do that earlier. Sorry I dropped the ball. I'll fix it now.

lizziel commented 5 years ago

Okay, see this new issue and the logs are posted in https://github.com/geoschem/gchp/issues/6. Let's keep the discussion of that issue to here rather than in two places. The printing of timing values issue is unrelated.

msulprizio commented 5 years ago

The GCHP restart files are now removed from ExtData/GEOSCHEM_RESTARTS/v2018-11. It may take a while to sync to the Harvard ftp and to AWS.

lizziel commented 5 years ago

Thanks Melissa! Jiawei, try turning off all collections. If that works, then turn on one collection, e.g. SpeciesConc, and check if results are okay. If that all works please open a new issue related to writing multiple files on the cloud.

yantosca commented 5 years ago

I logged into the cloud with AMI: GCHP12.1.0_tutorial_20181210 (ami-0f44e999c80ef6e66)

When I run with all collections turned, off, I get an error in offline tracer advection at 00:10 into the run.

When I run with only SpeciesConc_avg and SpeciesConc_inst turned on, then we still get the bad termination error right after the GIGCenv time is printed out.

Will dig a bit further.

JiaweiZhuang commented 5 years ago

@yantosca Thanks for looking into this! I have the same finding, which is quite weird:

0 output: crashes at 00:10
1, 2, 3 outputs (Choose from SpeciesConc_avg, SpeciesConc_inst, StateMet_avg): proceeds and generates output files without problems. The final timing info is incomplete, but it is a separate issue.
4 outputs (SpeciesConc_avg, SpeciesConc_inst, StateMet_avg, StateMet_inst): crashes at 01:00 when writing the first output. This can be "solved" by using the incorrect restart file v2018-11, for unknown reasons.

lizziel commented 5 years ago

I suspect these are three separate issues all unrelated to the restart file. I think we should close this issue (incorrect warning about species initial values) and create one or two new issues for the others.

-- Lizzie Lundgren Scientific Programmer GEOS-Chem Support Team geos-chem-support@as.harvard.edumailto:geos-chem-support@as.harvard.edu http://wiki.geos-chem.org/GEOS-Chem_Support_Team

Please direct all GEOS-Chem support issues to the entire GEOS-Chem Support Team at geos-chem-support@as.harvard.edumailto:geos-chem-support@as.harvard.edu. This will allow us to serve you better.

From: Jiawei Zhuang notifications@github.com Reply-To: geoschem/gchp reply@reply.github.com Date: Tuesday, December 11, 2018 at 6:16 PM To: geoschem/gchp gchp@noreply.github.com Cc: "Lundgren, Elizabeth W" elundgren@seas.harvard.edu, Comment comment@noreply.github.com Subject: Re: [geoschem/gchp] Potentially incorrect warning about species initial values (#8)

@yantoscahttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_yantosca&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=VyyBl8eF72Uw5bhJwoVODdKCdZCT8Qcap6kbrQAfIto&s=frU-eTznxzsbHlL6rMaAmcLzdI0ov9OpXZCVlJux39k&e= Thanks for looking into this! I have the same finding, which is quite weird:

0 output: crashes at 00:10
1, 2, 3 outputs (Choose from SpeciesConc_avg, SpeciesConc_inst, StateMet_avg): proceeds and generate output files without problems. The final timing info is incomplete, but it is a separate issue.
4 outputs (SpeciesConc_avg, SpeciesConc_inst, StateMet_avg, StateMet_inst): crashes at 01:00 when writing the first output.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geoschem_gchp_issues_8-23issuecomment-2D446399608&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=VyyBl8eF72Uw5bhJwoVODdKCdZCT8Qcap6kbrQAfIto&s=p7VTU33gjq2g7rpJY4Z7d7DabK-JaJW6DLvWENSmO84&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAnyqyEWdxgMryAMvvMBIHw3fIo3UlIHks5u4DzQgaJpZM4ZJdEl&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=VyyBl8eF72Uw5bhJwoVODdKCdZCT8Qcap6kbrQAfIto&s=fwYMuwE9itIOIZy4lQ5Z8jIupe3NLwFtiav5OqQAdY4&e=.

JiaweiZhuang commented 5 years ago

Agreed @lizziel

geoschem / gchp_legacy

[BUG/ISSUE] Potentially incorrect warning about species initial values #8