search

NGEET / fates

repository for the Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
Other
102 stars 92 forks source link

Testing CLM in ed repository #81

Closed bandre-ucar closed 8 years ago

bandre-ucar commented 8 years ago

Summary of Issue:

In order to merge ED/FATES work back into CLM, we need to ensure that we are not breaking any CLM functionality.

Expected behavior and actual behavior:

All tests in the clm test suite should have the same pass / fail as in the parent clm trunk tag.

Steps to reproduce the problem (should include create_newcase or create_test command along with any user_nl or xml changes):

ed-clm based on > clm-r18x:

clm short test suite

execca ./create_test -xml_category clm_short_45 -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm_short-20160811-1133 -testid 08111133-45i

--------------------------------------------------------------------------------
execca ./create_test -xml_category clm_short_45 -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm_short-20160811-1133 -testid 08111133-45p

--------------------------------------------------------------------------------
execca ./create_test -xml_category clm_short_45 -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm_short-20160811-1133 -testid 08111133-45g

--------------------------------------------------------------------------------
execca ./create_test -xml_category clm_short_50 -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm_short-20160811-1133 -testid 08111133-50i

--------------------------------------------------------------------------------
execca ./create_test -xml_category clm_short_50 -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm_short-20160811-1133 -testid 08111133-50p

--------------------------------------------------------------------------------
execca ./create_test -xml_category clm_short_50 -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm_short-20160811-1133 -testid 08111133-50g

main clm test suite

execca ./create_test -xml_category aux_clm40 -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm-20160811-1347 -testid 08111347-40i

--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm40 -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm-20160811-1347 -testid 08111347-40p

--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm40 -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm-20160811-1347 -testid 08111347-40g

started as pid : 14307
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm45 -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm-20160811-1347 -testid 08111347-45i

--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm45 -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm-20160811-1347 -testid 08111347-45p

--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm45 -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_11_r188 -testroot /glade/scratch/andre/tests-clm-20160811-1347 -testid 08111347-45g

clm-r120 based version of ed-clm:

clm short test suite

execca ./create_test -xml_category aux_clm_short -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm_short-20160707-1431 -testid 07071431-rti

started as pid : 8846
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm_short -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm_short-20160707-1431 -testid 07071431-rtp

started as pid : 8850
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm_short -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm_short-20160707-1431 -testid 07071431-rtg

main clm test suite

execca ./create_test -xml_category aux_clm40 -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm-20160708-1241 -testid 07081241-40i

started as pid : 27428
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm40 -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm-20160708-1241 -testid 07081241-40p

started as pid : 27430
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm40 -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm-20160708-1241 -testid 07081241-40g

started as pid : 27435
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm45 -mach yellowstone -compiler intel -xml_mach yellowstone -xml_compiler intel -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm-20160708-1241 -testid 07081241-45i

started as pid : 27452
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm45 -mach yellowstone -compiler pgi -xml_mach yellowstone -xml_compiler pgi -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm-20160708-1241 -testid 07081241-45p

started as pid : 27457
--------------------------------------------------------------------------------
execca ./create_test -xml_category aux_clm45 -mach yellowstone -compiler gnu -xml_mach yellowstone -xml_compiler gnu -compare clm4_5_1_r120 -testroot /glade/scratch/andre/tests-clm-20160708-1241 -testid 07081241-45g

What is the changeset ID of the code, and the machine you are using:

have you modified the code? If so, it must be committed and available for testing:

Screen output or output files showing the error message and context:

bandre-ucar commented 8 years ago

Results of the short clm test suite with changeset c0654dba:

ExpectedTestFails file list:
    /glade/p/work/andre/ed/clm-testing/ed-clm/cime/scripts/Testing/Testlistxml/ExpectedTestFails.xml
    /glade/p/work/andre/ed/clm-testing/ed-clm/components/clm/cimetest/ExpectedTestFails.xml

RUN ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_gnu.clm-default.C.07071431-rtg 

RUN ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_pgi.clm-default.C.07071431-rtp 

RUN ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_intel.clm-default.C.07071431-rti

Error from cesm log and stack trace from the failed intel run:

   7: ERROR: get_clump_bounds ERROR: Calling from inside a non-threaded region)

   7:Image              PC                Routine            Line        Source             
   7:cesm.exe           0000000003E17E2D  Unknown               Unknown  Unknown
   7:cesm.exe           00000000033F985B  shr_sys_mod_mp_sh         401  shr_sys_mod.F90
   7:cesm.exe           00000000033F9373  shr_sys_mod_mp_sh         230  shr_sys_mod.F90
   7:cesm.exe           00000000008BF99C  decompmod_mp_get_         148  decompMod.F90
   7:cesm.exe           000000000086F23F  clmfatesinterface         230  clmfates_interfaceMod.F90
   7:cesm.exe           0000000000859EA3  clm_instmod_mp_cl         431  clm_instMod.F90
   7:cesm.exe           000000000084E646  clm_initializemod         444  clm_initializeMod.F90
   7:cesm.exe           00000000007E9C90  lnd_comp_mct_mp_l         231  lnd_comp_mct.F90
   7:cesm.exe           000000000045FAA2  component_mod_mp_         229  component_mod.F90
   7:cesm.exe           0000000000429638  cesm_comp_mod_mp_        1086  cesm_comp_mod.F90
   7:cesm.exe           0000000000450360  MAIN__                     92  cesm_driver.F90

So:

  1. the fates interface is active in a clm run that should not have fates running
  2. there is a threading related error in fates.
bandre-ucar commented 8 years ago

Explicitly calling out @rosiealice @ckoven @rgknox per last software engineering telecon.

bandre-ucar commented 8 years ago

Full clm test suite with r120 based changeset c0654db: 

ExpectedTestFails file list:
    /glade/p/work/andre/ed/clm-testing/ed-clm/cime/scripts/Testing/Testlistxml/ExpectedTestFails.xml
    /glade/p/work/andre/ed/clm-testing/ed-clm/components/clm/cimetest/ExpectedTestFails.xml

RUN ERP_P15x2_D.f10_f10.ICLM45BGC.yellowstone_gnu.clm-snowlayers_12.C.07081241-45g 

RUN ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_gnu.clm-default.C.07081241-45g 

    BFAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_1_r120 file) (baseline history file does not exist)
    BFAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest.cpl.hi.nc : baseline compare cpl.hi (baseline: compare .base file with clm4_5_1_r120 file) (baseline history file does not exist)
    BFAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest : baseline compare summary (baseline: compare .base file with clm4_5_1_r120 file) (baseline history file does not exist)
    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest.memcomp.clm4_5_1_r120 
    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest.tputcomp.clm4_5_1_r120 
    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest.C.07081241-45g.nlcomp

RUN ERP_P180x2_D_Ld5.f19_g16.ICLM45BGCDVCROP.yellowstone_pgi.clm-crop.C.07081241-45p 

RUN ERP_P15x2_D_Ld5.f10_f10.I1850CLM45BGC.yellowstone_pgi.clm-ciso.C.07081241-45p 

    FAIL SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.yellowstone_pgi.clm-edTest.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.yellowstone_pgi.clm-edTest.cpl.hi.nc : baseline compare cpl.hi (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.yellowstone_pgi.clm-edTest : baseline compare summary (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.yellowstone_pgi.clm-edTest.C.07081241-45p.nlcomp

RUN ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_pgi.clm-default.C.07081241-45p 

RUN ERP_P15x2_Lm13.f10_f10.IHISTCLM45BGC.yellowstone_pgi.clm-monthly.C.07081241-45p 

CFAIL ERP.f19_g16.IG1850CLM45.yellowstone_pgi.C.07081241-45p

    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest.cpl.hi.nc : baseline compare cpl.hi (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest : baseline compare summary (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest.C.07081241-45p.nlcomp

RUN ERP_D_P15x2_Ld3.f10_f10.ICRUCLM50BGC.yellowstone_intel.clm-luna.C.07081241-45i 

    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest.cpl.hi.nc : baseline compare cpl.hi (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest : baseline compare summary (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest.C.07081241-45i.nlcomp

RUN ERP_P180x2_D_Ld5.f19_g16.I1850CLM45.yellowstone_intel.clm-default.C.07081241-45i 

CFAIL ERP_D_Ld5.f19_g16.ICLM45GLCMEC.yellowstone_intel.clm-glcMEC_changeFlags.C.07081241-45i

CFAIL ERS_D_Ld10.T31_g37.IGHISTCLM45.yellowstone_intel.clm-glcMEC_decrease.C.07081241-45i

    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest.cpl.hi.nc : baseline compare cpl.hi (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest : baseline compare summary (baseline: compare .base file with clm4_5_1_r120 file) 
    FAIL SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest.C.07081241-45i.nlcomp

RUN ERP_D_P15x2_Ld3.f10_f10.ICRUCLM50BGC.yellowstone_intel.clm-flexibleCN.C.07081241-45i 

CFAIL SMS_Lm37.T31_g37.IG1850CLM45.yellowstone_intel.clm-glcMEC_long.C.07081241-45i

RUN ERP_P15x2_Lm36.f10_f10.ICLM45BGCCROP.yellowstone_intel.clm-irrig_o3_reduceOutput.C.07081241-45i 

RUN ERP_P15x2_Lm25.f10_f10.ICLM45BGCDVCROP.yellowstone_intel.clm-monthly.C.07081241-45i 

RUN ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_intel.clm-default.C.07081241-45i 

CFAIL ERI_D.f19_g16.IG1850CN.yellowstone_pgi.C.07081241-40p

Looks like primarily ERP tests, probably for the same reasons as the short test list, but I didn't look.

The CFAIL for IG tests is because the r120 based version of the ed repo has an inconsistent version of cism. I believe this is fixed in one of the trunk merge changes.

ckoven commented 8 years ago

@bandre-ucar ugg. ok, since the short test is giving us a fail, and there may be others, maybe we ought to start by trying to narrow down when this came into the code. could you run the short test suite on tags: d8a9ee5 c23cf02 0471ef9 94118a5 57c533c 18613d1

rgknox commented 8 years ago

Thanks @bandre-ucar for getting these going. Just to clarify, are their failures just on the ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_intel.clm-default.C.07071431-rti, or others as well?

ERP is a "2 month exact restart from startup", which should be similar to the ERS during the initialization call sequence. The P15x2 indicates that a different threading scheme should be used on the restart right?

I think we have had one restart test on the CLMBGC previously (the hybrid/branch restart type): ERI_D_Ld9.f09_g16.ICLM45BGC

bandre-ucar commented 8 years ago

@rgknox I put the entire contens of the unexpected results for the short and full test suites in the above comments. Unexpected means filtering out junk like PASS lines, namelist changes, throughput and memory comp.

The erp test is an exact restart with PE layout change. I believe the 15x2 indicates the layout, and it is halved or something on restart. The Ld3 indicates it is over-riding the default length and running for three days.

ckoven commented 8 years ago

@bandre-ucar a couple things:

(1) could you copy in the PASS results too? Its not clear to me whether these are mainly differences in model configuration or testing configuration and so it may be helpful to see the whole list to get a better perspective.

(2) what are the actual commands to run the full and fast CLM test suites? I am sort of curious about what this looks like for some code snapshots (e.g. my message from friday) and so would like to know what command triggers these.

more generally, I'm trying to figure out what these results mean and what the general sense of severity of these results is. If these are just some weird edge case that happens from the ERP test configurations then it seems much less severe than if there are large swaths of the model configuration space that we haven't been covering until now. do you have a sense of which of these is more likely?

thanks-

bandre-ucar commented 8 years ago

@ckoven See attached for full results. clm-results-c0654db.txt

The commands to run both the short and full test suite are above in the issue description.

I have a script that will run tests for a set of changes. I just won't have time to set it up until later today or tomorrow.

In general, I think there is a very high probability that threading is broken, and we need to be more careful about ensuring all ed call are contained is some sort of 'use_ed' logic. The cism issue is a scripting bug in pulling clm trunk tags and should be addressed in a future merge. Other than that, I think we've had a minor effect on most clm functionality.

ckoven commented 8 years ago

@bandre-ucar thanks!

bandre-ucar commented 8 years ago

@ckoven Results from running the failing test on the changeset list you requested.

Test: ERP_D_Ld3_P15x2.f10_f10.ICLM45BGC.yellowstone_intel.clm-default Baseline comparison: none

Base changeset: c0654db - runtime failure

Changeset perturbation: d8a9ee5 - pass c23cf02 - pass 0471ef9 - runtime failure 94118a5 - runtime failure 57c533c - runtime failure 18613d1 - runtime failure

ckoven commented 8 years ago

@bandre-ucar thanks -- that is very helpful to know where the failures came in.

rgknox commented 8 years ago

The ERP is failing because when we first allocate FATES sites, we are calling the function:

call get_clump_bounds(nc, bounds_clump)

Inside get_clump_bounds, there is a check that evaluates the threading environment:

#ifdef _OPENMP
     ! FIX(SPM, 090314) - for debugging ED and openMP
     !write(iulog,*) 'SPM omp debug decompMod 1 ', &
          !OMP_GET_NUM_THREADS(),OMP_GET_MAX_THREADS(),OMP_GET_THREAD_NUM()

     if ( OMP_GET_NUM_THREADS() == 1 .and. OMP_GET_MAX_THREADS() > 1 )then
        call shr_sys_abort( trim(subname)//' ERROR: Calling from inside a non-threaded region)')
     end if
#endif

It is going into that shr_sysabort sequence. I suppose that we had not been using shared memory paralellism (OMP stuff) until this test.

rgknox commented 8 years ago

I believe the solution is to add the OMP pragmas that you see in clm_driv to the clump loop in init_allocate. I had originally left this out because it seemed overkill to use multi-threading during a one time process, but if the functions within demand it, then we must obey.

bandre-ucar commented 8 years ago

Full clm test suite on yellowstone for changeset 08014274, comparing to clm4_5_9_r182 baselines.

ExpectedTestFails file list:
    /glade/p/work/andre/ed/issue91/ed-clm/components/clm/cime_config/testdefs/ExpectedTestFails.xml

CFAIL ERI_D.f19_g16_gl5.IG1850CN.yellowstone_pgi.C.08041054-40p
CFAIL ERP.f19_g16_gl5.IG1850CLM45.yellowstone_pgi.C.08041054-45p
CFAIL PEM_D.f19_g16_gl5.IG1850CLM45.yellowstone_pgi.clm-glcMEC_increase.C.08041054-45p
CFAIL LII_D_Ld3.f19_g16_gl5.IG1850CLM45.yellowstone_intel.clm-glcMEC_spunup_1way.C.08041054-45i
CFAIL SMS_Lm1.f09_g16_gl5.IG1850CRUCLM50BGC.yellowstone_intel.clm-clm50KitchenSink.C.08041054-45i
CFAIL ERS_D_Ld12.T31_g37_gl5.IG1850CRUCLM50BGC.yellowstone_intel.clm-glcMEC_spunup_inc_dec_bgc.C.08041054-45i
CFAIL ERS_D_Ld10.T31_g37_gl5.IGHISTCLM45.yellowstone_intel.clm-glcMEC_decrease.C.08041054-45i
CFAIL ERS_D_Ld3.f09_g16_gl5.IG1850CRUCLM50BGCCROP.yellowstone_intel.clm-clm50KitchenSink.C.08041054-45i
CFAIL ERS_D_Ld3.f19_g16_gl5.IG1850CRUCLM50BGCCROP.yellowstone_intel.clm-clm50dynroots.C.08041054-45i
CFAIL SMS_Lm1.f19_g16_gl5.IG1850CRUCLM50BGC.yellowstone_intel.clm-clm50dynroots.C.08041054-45i
CFAIL ERP_D_Ld5.f19_g16_gl5.ICLM45GLCMEC.yellowstone_intel.clm-glcMEC_changeFlags.C.08041054-45i 
CFAIL SMS_Lm37.T31_g37_gl5.IG1850CLM45.yellowstone_intel.clm-glcMEC_long.C.08041054-45i
CFAIL ERS_Lm3.f19_g16_gl5.IGRCP60CN.yellowstone_intel.C.08041054-40i

BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.yellowstone_pgi.clm-edTest does not exist)
BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest does not exist)
BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/ERS_D_Mmpi-serial_Ld5.1x1_brazil.ICLM45ED.yellowstone_gnu.clm-edTest does not exist)
BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest does not exist)
BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/ERS_D_Mmpi-serial_Ld5.1x1_brazil.ICLM45ED.yellowstone_intel.clm-edTest does not exist)
BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest does not exist)
BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_8_r182/SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest does not exist)

RUN ERP_Lm40.1x1_numaIA.ICRUCLM50BGCCROP.yellowstone_gnu.clm-monthly.C.08041054-45g 

FAIL SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_8_r182 file) 
FAIL SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.clm2.h1.nc : baseline compare clm2.h1 (baseline: compare .base file with clm4_5_8_r182 file) 
FAIL SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default : baseline compare summary (baseline: compare .base file with clm4_5_8_r182 file) 

FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_pgi.clm-default.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_8_r182 file) 
FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_pgi.clm-default.clm2.h1.nc : baseline compare clm2.h1 (baseline: compare .base file with clm4_5_8_r182 file) 
FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_pgi.clm-default : baseline compare summary (baseline: compare .base file with clm4_5_8_r182 file) 

/glade/scratch/andre/SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.C.08041054-45i.ref1/run/SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.C.08041054-45i.ref1.clm2.h0.0001-01-06-00000.nc.base.cprnc.out had the following fields that are NOT b4b  

   RMS ACTUAL_IMMOB                     2.0690E-09            NORMALIZED  9.7269E-03
    RMS AGNPP                            1.5931E-08            NORMALIZED  8.4822E-03
    RMS ALT                              2.1083E-16            NORMALIZED  8.6289E-18
    RMS AR                               1.0836E-08            NORMALIZED  1.4364E-03
    RMS BAF_PEATF                        2.9978E-22            NORMALIZED  6.5151E-11
    RMS BGNPP                            8.4961E-09            NORMALIZED  5.0476E-03
    RMS BTRAN                            5.2878E-07            NORMALIZED  1.9897E-06
    RMS CH4PROD                          1.0697E-10            NORMALIZED  3.1603E-03
    RMS CH4_SURF_AERE_SAT                7.8659E-11            NORMALIZED  1.0302E-02
    RMS CH4_SURF_AERE_UNSAT              2.9193E-12            NORMALIZED  2.0909E-02
    RMS CH4_SURF_DIFF_SAT                4.1773E-13            NORMALIZED  3.7451E-03
    RMS CH4_SURF_DIFF_UNSAT              4.7495E-13            NORMALIZED  1.2702E-03
    RMS CH4_SURF_EBUL_SAT                6.4085E-13            NORMALIZED  8.1238E-04
    RMS CH4_SURF_EBUL_UNSAT              2.4179E-14            NORMALIZED  1.2897E-02
    RMS COL_CTRUNC                       1.1905E-09            NORMALIZED  2.7342E-08
    RMS COL_FIRE_CLOSS                   9.3583E-11            NORMALIZED  8.3812E-04
    RMS COL_FIRE_NLOSS                   1.0907E-12            NORMALIZED  4.5311E-04
    RMS COL_NTRUNC                       1.2899E-11            NORMALIZED  6.3085E-09
    RMS CONC_O2_SAT                      4.2579E-02            NORMALIZED  8.0631E-01
    RMS CONC_O2_UNSAT                    4.8737E-05            NORMALIZED  4.5327E-05
    RMS CWDC                             5.8016E+00            NORMALIZED  6.0625E-03
    RMS CWDC_LOSS                        1.2465E-08            NORMALIZED  8.5991E-03
    RMS CWDC_TO_LITR2C                   9.4679E-09            NORMALIZED  8.6571E-03
    RMS CWDC_TO_LITR3C                   2.9899E-09            NORMALIZED  8.6571E-03
    RMS CWDC_vr                          2.6266E+01            NORMALIZED  1.5493E-02
    RMS CWDN                             1.1894E-02            NORMALIZED  6.0696E-03
    RMS CWDN_TO_LITR2N                   1.9404E-11            NORMALIZED  8.6714E-03
    RMS CWDN_TO_LITR3N                   6.1277E-12            NORMALIZED  8.6714E-03
    RMS DEADCROOTC                       1.3051E-03            NORMALIZED  1.9300E-06
    RMS DEADCROOTC_STORAGE               4.8568E-04            NORMALIZED  7.6092E-05
    RMS DEADCROOTC_XFER                  2.0340E-08            NORMALIZED  6.2850E-06
    RMS DEADCROOTN                       2.6102E-06            NORMALIZED  1.9300E-06
    RMS DEADCROOTN_STORAGE               9.7135E-07            NORMALIZED  7.6092E-05
    RMS DEADCROOTN_XFER                  4.0679E-11            NORMALIZED  6.2850E-06
    RMS DEADSTEMC                        4.3504E-03            NORMALIZED  2.0082E-06
    RMS DEADSTEMC_STORAGE                1.6189E-03            NORMALIZED  7.5921E-05
    RMS DEADSTEMC_XFER                   6.5680E-08            NORMALIZED  6.0661E-06
    RMS DEADSTEMN                        8.7008E-06            NORMALIZED  1.9724E-06
    RMS DEADSTEMN_STORAGE                3.2378E-06            NORMALIZED  7.5711E-05
    RMS DEADSTEMN_XFER                   1.3150E-10            NORMALIZED  6.0524E-06
    RMS DENIT                            2.5115E-10            NORMALIZED  2.3657E-02
    RMS DISPVEGC                         1.1082E-02            NORMALIZED  3.6889E-06
    RMS DISPVEGN                         1.6075E-04            NORMALIZED  1.5961E-05
    RMS DSTFLXT                          1.9536E-17            NORMALIZED  1.0522E-08
    RMS EFLX_LH_TOT                      3.0122E-03            NORMALIZED  1.5485E-04
    RMS EFLX_LH_TOT_R                    4.0934E-03            NORMALIZED  1.6334E-04
    RMS ELAI                             4.3308E-05            NORMALIZED  3.6543E-05
    RMS ER                               2.0531E-08            NORMALIZED  1.6063E-03
    RMS ERRH2O                           2.4435E-14            NORMALIZED  5.3880E-01
    RMS ERRH2OSNO                        5.0676E-18            NORMALIZED  7.9691E-03
    RMS ERRSEB                           7.2191E-15            NORMALIZED  9.3800E-01
    RMS ERRSOI                           1.2235E-11            NORMALIZED  7.0534E-02
    RMS ERRSOL                           1.6110E-15            NORMALIZED  5.6892E-01
    RMS ESAI                             5.6431E-07            NORMALIZED  1.4210E-06
    RMS FAREA_BURNED                     2.6425E-13            NORMALIZED  6.9122E-04
    RMS FCEV                             2.0898E-04            NORMALIZED  4.1151E-05
    RMS FCH4                             7.0937E-14            NORMALIZED  7.8133E-03
    RMS FCH4TOCO2                        8.8699E-11            NORMALIZED  3.7881E-03
    RMS FCH4_DFSAT                       3.0981E-16            NORMALIZED  2.9419E-03
    RMS FCO2                             1.4639E-10            NORMALIZED  1.1655E-02
    RMS FCOV                             3.0058E-10            NORMALIZED  4.1170E-09
    RMS FCTR                             2.5717E-03            NORMALIZED  3.3686E-04
    RMS FGEV                             7.7799E-04            NORMALIZED  1.1401E-04
    RMS FGR                              7.7414E-04            NORMALIZED  8.2733E-05
    RMS FGR12                            7.6007E-04            NORMALIZED  9.4112E-05
    RMS FGR_R                            9.3987E-04            NORMALIZED  1.1664E-04
    RMS FH2OSFC                          1.3437E-07            NORMALIZED  4.5804E-05
    RMS FINUNDATED                       9.5599E-09            NORMALIZED  1.0294E-07
    RMS FINUNDATED_LAG                   5.7475E-10            NORMALIZED  2.1569E-08
    RMS FIRA                             5.4307E-04            NORMALIZED  9.9769E-06
    RMS FIRA_R                           7.1985E-04            NORMALIZED  1.3475E-05
    RMS FIRE                             5.4307E-04            NORMALIZED  1.8127E-06
    RMS FIRE_R                           7.1985E-04            NORMALIZED  2.2430E-06
    RMS FPG                              5.8535E-04            NORMALIZED  6.0051E-04
    RMS FPI                              7.0422E-04            NORMALIZED  7.5980E-04
    RMS FPSN                             3.2722E-05            NORMALIZED  2.9566E-05
    RMS FPSN_WC                          1.0606E-05            NORMALIZED  2.6176E-05
    RMS FPSN_WJ                          2.7913E-05            NORMALIZED  4.0988E-05
    RMS FPSN_WP                          4.7483E-07            NORMALIZED  2.3067E-05
    RMS FROOTC                           2.4102E-03            NORMALIZED  2.9056E-05
    RMS FROOTC_ALLOC                     5.3136E-09            NORMALIZED  3.7898E-03
    RMS FROOTC_LOSS                      5.1297E-11            NORMALIZED  1.9736E-05
    RMS FROOTC_STORAGE                   3.6770E-03            NORMALIZED  3.8665E-05
    RMS FROOTC_XFER                      2.9051E-07            NORMALIZED  3.8384E-07
    RMS FROOTN                           5.7395E-05            NORMALIZED  2.9006E-05
    RMS FROOTN_STORAGE                   8.7520E-05            NORMALIZED  3.8738E-05
    RMS FROOTN_XFER                      9.0845E-09            NORMALIZED  5.0389E-07
    RMS FSA                              3.3700E-05            NORMALIZED  3.9436E-07
    RMS FSAT                             3.0058E-10            NORMALIZED  4.1170E-09
    RMS FSA_R                            3.9743E-05            NORMALIZED  4.4142E-07
    RMS FSH                              3.1373E-03            NORMALIZED  1.0865E-04
    RMS FSH_G                            2.7318E-04            NORMALIZED  1.7007E-05
    RMS FSH_R                            4.3848E-03            NORMALIZED  1.1774E-04
    RMS FSH_TO_COUPLER                   3.1373E-03            NORMALIZED  1.0865E-04
    RMS FSH_V                            2.9665E-03            NORMALIZED  2.2602E-04
    RMS FSM                              8.8082E-06            NORMALIZED  1.4284E-05
    RMS FSM_R                            1.0247E-05            NORMALIZED  1.2481E-05
    RMS FSNO                             6.6529E-07            NORMALIZED  1.1247E-06
    RMS FSNO_EFF                         6.6572E-07            NORMALIZED  1.0846E-06
    RMS FSR                              3.3700E-05            NORMALIZED  3.2919E-07
    RMS FSRND                            2.7451E-05            NORMALIZED  7.5302E-07
    RMS FSRNDLN                          7.2030E-05            NORMALIZED  1.0672E-06
    RMS FSRNI                            5.9280E-06            NORMALIZED  6.4457E-07
    RMS FSRVD                            3.5573E-06            NORMALIZED  9.7213E-08
    RMS FSRVDLN                          1.5290E-05            NORMALIZED  2.3323E-07
    RMS FSRVI                            3.4861E-06            NORMALIZED  1.7321E-07
    RMS FUELC                            5.7515E+00            NORMALIZED  2.4069E-03
    RMS F_DENIT                          2.5115E-10            NORMALIZED  2.3657E-02
    RMS F_DENIT_vr                       7.4353E-10            NORMALIZED  6.1386E-02
    RMS F_N2O_DENIT                      8.5553E-12            NORMALIZED  1.6604E-02
    RMS F_N2O_NIT                        1.6169E-13            NORMALIZED  1.2424E-02
    RMS F_NIT                            2.6948E-10            NORMALIZED  1.2424E-02
    RMS F_NIT_vr                         7.5775E-10            NORMALIZED  2.9093E-02
    RMS GC_HEAT1                         1.2108E-04            NORMALIZED  6.9380E-09
    RMS GC_ICE1                          3.5782E-05            NORMALIZED  2.9132E-09
    RMS GC_LIQ1                          1.0473E-04            NORMALIZED  2.2857E-08
    RMS GPP                              4.6070E-08            NORMALIZED  4.1454E-03
    RMS GR                               1.0601E-08            NORMALIZED  7.8554E-03
    RMS GROSS_NMIN                       7.4170E-11            NORMALIZED  2.5063E-04
    RMS H2OCAN                           3.7364E-05            NORMALIZED  6.1676E-04
    RMS H2OSFC                           1.1043E-05            NORMALIZED  1.4818E-04
    RMS H2OSNO                           1.4530E-05            NORMALIZED  4.1312E-08
    RMS H2OSNO_TOP                       9.7899E-07            NORMALIZED  6.1050E-07
    RMS H2OSOI                           1.4360E-07            NORMALIZED  7.8318E-07
    RMS HC                               1.2751E-04            NORMALIZED  5.5238E-09
    RMS HCSOI                            1.5875E-04            NORMALIZED  6.6504E-09
    RMS HR                               1.7782E-08            NORMALIZED  3.3947E-03
    RMS HR_vr                            8.5130E-08            NORMALIZED  9.7046E-03
    RMS HTOP                             2.7375E-06            NORMALIZED  5.6392E-07
    RMS INT_SNOW                         5.3299E-05            NORMALIZED  2.9768E-08
    RMS LAISHA                           4.1771E-05            NORMALIZED  3.7468E-05
    RMS LAISUN                           1.5435E-06            NORMALIZED  2.1959E-05
    RMS LEAFC                            2.4069E-03            NORMALIZED  3.4234E-05
    RMS LEAFC_ALLOC                      5.3052E-09            NORMALIZED  5.6377E-03
    RMS LEAFC_LOSS                       5.1302E-11            NORMALIZED  3.0425E-05
    RMS LEAFC_STORAGE                    2.2038E-03            NORMALIZED  3.2998E-05
    RMS LEAFC_XFER                       2.0459E-07            NORMALIZED  5.0799E-07
    RMS LEAFN                            7.9679E-05            NORMALIZED  3.6882E-05
    RMS LEAFN_STORAGE                    8.8128E-05            NORMALIZED  3.3051E-05
    RMS LEAFN_XFER                       5.0853E-08            NORMALIZED  3.1549E-06
    RMS LEAF_MR                          1.2144E-10            NORMALIZED  6.5476E-05
    RMS LITFALL                          1.1093E-10            NORMALIZED  1.7730E-05
    RMS LITHR                            1.7390E-08            NORMALIZED  6.4514E-03
    RMS LITR1C                           2.0858E-03            NORMALIZED  7.4948E-05
    RMS LITR1C_TO_SOIL1C                 1.5376E-09            NORMALIZED  3.1213E-03
    RMS LITR1C_vr                        1.6015E-02            NORMALIZED  3.3241E-04
    RMS LITR1N                           4.6969E-05            NORMALIZED  7.3248E-05
    RMS LITR1N_TO_SOIL1N                 7.7053E-11            NORMALIZED  3.0063E-03
    RMS LITR1_HR                         1.8793E-09            NORMALIZED  3.1213E-03
    RMS LITR2C                           6.5550E-01            NORMALIZED  3.0338E-03
    RMS LITR2C_TO_SOIL1C                 1.3928E-08            NORMALIZED  9.5243E-03
    RMS LITR2C_vr                        3.2374E+00            NORMALIZED  8.7251E-03
    RMS LITR2N                           1.3902E-03            NORMALIZED  4.0285E-04
    RMS LITR2N_TO_SOIL1N                 1.0179E-10            NORMALIZED  2.3925E-03
    RMS LITR2_HR                         1.3928E-08            NORMALIZED  9.5243E-03
    RMS LITR3C                           2.0748E-01            NORMALIZED  2.1530E-03
    RMS LITR3C_TO_SOIL2C                 4.1630E-09            NORMALIZED  6.5970E-03
    RMS LITR3C_vr                        1.0263E+00            NORMALIZED  6.3492E-03
    RMS LITR3N                           4.4897E-04            NORMALIZED  2.6385E-04
    RMS LITR3N_TO_SOIL2N                 5.3475E-11            NORMALIZED  2.5632E-03
    RMS LITR3_HR                         4.1630E-09            NORMALIZED  6.5970E-03
    RMS LITTERC                          8.6443E-01            NORMALIZED  2.5405E-03
    RMS LITTERC_HR                       1.7390E-08            NORMALIZED  6.4514E-03
    RMS LITTERC_LOSS                     3.4900E-08            NORMALIZED  6.5759E-03
    RMS LIVECROOTC                       1.4401E-04            NORMALIZED  7.4059E-05
    RMS LIVECROOTC_STORAGE               5.4174E-05            NORMALIZED  7.3316E-05
    RMS LIVECROOTC_XFER                  7.8839E-09            NORMALIZED  6.8449E-06
    RMS LIVECROOTN                       2.8802E-06            NORMALIZED  7.4059E-05
    RMS LIVECROOTN_STORAGE               1.0835E-06            NORMALIZED  7.3316E-05
    RMS LIVECROOTN_XFER                  1.5768E-10            NORMALIZED  6.8449E-06
    RMS LIVESTEMC                        4.8003E-04            NORMALIZED  7.4172E-05
    RMS LIVESTEMC_STORAGE                1.8058E-04            NORMALIZED  7.3137E-05
    RMS LIVESTEMC_XFER                   2.0192E-08            NORMALIZED  5.2193E-06
    RMS LIVESTEMN                        9.6007E-06            NORMALIZED  7.4172E-05
    RMS LIVESTEMN_STORAGE                3.6116E-06            NORMALIZED  7.3137E-05
    RMS LIVESTEMN_XFER                   4.0385E-10            NORMALIZED  5.2193E-06
    RMS MR                               3.1629E-10            NORMALIZED  5.1060E-05
    RMS M_LITR1C_TO_LEACHING             1.1938E-19            NORMALIZED  7.6668E-01
    RMS M_LITR2C_TO_LEACHING             1.4096E-18            NORMALIZED  1.1320E+00
    RMS M_LITR3C_TO_LEACHING             6.1354E-19            NORMALIZED  1.1592E+00
    RMS M_SOIL1C_TO_LEACHING             6.0844E-19            NORMALIZED  1.3072E+00
    RMS M_SOIL2C_TO_LEACHING             1.9644E-17            NORMALIZED  1.6015E+00
    RMS M_SOIL3C_TO_LEACHING             4.8302E-17            NORMALIZED  4.0928E-01
    RMS NBP                              3.9924E-08            NORMALIZED  1.1655E-02
    RMS NDEPLOY                          6.3490E-10            NORMALIZED  6.5714E-03
    RMS NEE                              3.9924E-08            NORMALIZED  1.1655E-02
    RMS NEM                              6.6546E-11            NORMALIZED  5.4972E-03
    RMS NEP                              3.9924E-08            NORMALIZED  1.1897E-02
    RMS NET_CARBON_EXCHANGE              3.9924E-08            NORMALIZED  1.1655E-02
    RMS NET_NMIN                         2.0040E-09            NORMALIZED  2.3927E-02
    RMS NFIRE                            1.8397E-13            NORMALIZED  8.8893E-04
    RMS NFIX_TO_SMINN                    2.6753E-12            NORMALIZED  3.0081E-04
    RMS NPP                              3.5237E-08            NORMALIZED  6.6183E-03
    RMS O_SCALAR                         1.0930E-04            NORMALIZED  1.1556E-04
    RMS PARVEGLN                         5.0269E-04            NORMALIZED  9.4704E-06
    RMS PFT_CTRUNC                       1.4012E-13            NORMALIZED  9.7612E-09
    RMS PFT_FIRE_CLOSS                   2.8030E-11            NORMALIZED  3.7164E-04
    RMS PFT_FIRE_NLOSS                   7.4863E-13            NORMALIZED  4.0881E-04
    RMS PFT_NTRUNC                       1.1581E-06            NORMALIZED  4.8014E-04
    RMS PLANT_NDEMAND                    2.5095E-12            NORMALIZED  1.5735E-05
    RMS POTENTIAL_IMMOB                  3.5804E-09            NORMALIZED  1.2950E-02
    RMS POT_F_DENIT                      2.4935E-10            NORMALIZED  2.2653E-02
    RMS POT_F_NIT                        2.6938E-10            NORMALIZED  1.2409E-02
    RMS PSNSHA                           1.3729E-05            NORMALIZED  7.5868E-05
    RMS PSNSHADE_TO_CPOOL                2.0785E-08            NORMALIZED  3.8563E-03
    RMS PSNSUN                           2.1328E-05            NORMALIZED  2.8791E-05
    RMS PSNSUN_TO_CPOOL                  2.5777E-08            NORMALIZED  4.5033E-03
    RMS Q2M                              1.7894E-07            NORMALIZED  3.6751E-05
    RMS QCHARGE                          6.5011E-12            NORMALIZED  1.0235E-06
    RMS QDRAI                            9.2177E-13            NORMALIZED  3.3880E-07
    RMS QDRAI_PERCH                      2.7829E-19            NORMALIZED  9.6565E-06
    RMS QDRIP                            8.5170E-11            NORMALIZED  5.2768E-06
    RMS QFLX_SNOW_DRAIN                  2.6435E-11            NORMALIZED  2.6921E-05
    RMS QH2OSFC                          1.5355E-11            NORMALIZED  6.7726E-05
    RMS QINFL                            3.0670E-10            NORMALIZED  3.2696E-05
    RMS QINTR                            9.6232E-11            NORMALIZED  3.8459E-05
    RMS QOVER                            1.3813E-11            NORMALIZED  1.0211E-05
    RMS QOVER_LAG                        1.4557E-12            NORMALIZED  7.8151E-07
    RMS QRUNOFF                          2.5483E-11            NORMALIZED  5.5873E-06
    RMS QRUNOFF_ICE_TO_COUPLER           4.3517E-22            NORMALIZED  2.1665E-16
    RMS QRUNOFF_R                        2.9837E-11            NORMALIZED  5.0813E-06
    RMS QRUNOFF_TO_COUPLER               2.5483E-11            NORMALIZED  5.5873E-06
    RMS QSNOFRZ                          3.2879E-12            NORMALIZED  3.0342E-06
    RMS QSNOMELT                         2.6395E-11            NORMALIZED  1.4284E-05
    RMS QSOIL                            3.1107E-10            NORMALIZED  1.1626E-04
    RMS QVEGE                            8.3558E-11            NORMALIZED  4.1151E-05
    RMS QVEGT                            1.0283E-09            NORMALIZED  3.3686E-04
    RMS RETRANSN                         2.5281E-07            NORMALIZED  1.1732E-07
    RMS RETRANSN_TO_NPOOL                5.5229E-13            NORMALIZED  2.3635E-05
    RMS RH2M                             8.1107E-04            NORMALIZED  9.1868E-06
    RMS RH2M_R                           1.0653E-03            NORMALIZED  1.2278E-05
    RMS RR                               4.5329E-09            NORMALIZED  9.8181E-04
    RMS SABG                             3.7456E-04            NORMALIZED  7.0938E-06
    RMS SABG_PEN                         1.2449E-06            NORMALIZED  5.2823E-07
    RMS SABV                             3.4340E-04            NORMALIZED  1.0516E-05
    RMS SMINN                            1.1543E-03            NORMALIZED  3.0109E-04
    RMS SMINN_TO_NPOOL                   6.3459E-10            NORMALIZED  8.6634E-03
    RMS SMINN_TO_PLANT                   6.3459E-10            NORMALIZED  8.6634E-03
    RMS SMINN_TO_SOIL1N_L1               1.1522E-10            NORMALIZED  3.2052E-03
    RMS SMINN_TO_SOIL1N_L2               1.7377E-09            NORMALIZED  1.2390E-02
    RMS SMINN_TO_SOIL1N_S2               8.3578E-12            NORMALIZED  1.2514E-04
    RMS SMINN_TO_SOIL1N_S3               6.5801E-13            NORMALIZED  2.5243E-04
    RMS SMINN_TO_SOIL2N_L3               3.8431E-10            NORMALIZED  1.0528E-02
    RMS SMINN_TO_SOIL2N_S1               7.2121E-11            NORMALIZED  3.3106E-04
    RMS SMINN_TO_SOIL3N_S1               6.1136E-13            NORMALIZED  3.3881E-04
    RMS SMINN_TO_SOIL3N_S2               8.6122E-13            NORMALIZED  1.2514E-04
    RMS SMINN_vr                         3.9465E-03            NORMALIZED  7.5927E-04
    RMS SMIN_NH4                         1.1433E-03            NORMALIZED  1.1320E-03
    RMS SMIN_NH4_vr                      3.9297E-03            NORMALIZED  1.4642E-03
    RMS SMIN_NO3                         4.5619E-05            NORMALIZED  1.6154E-05
    RMS SMIN_NO3_LEACHED                 2.2360E-13            NORMALIZED  1.6258E-03
    RMS SMIN_NO3_RUNOFF                  3.1028E-12            NORMALIZED  2.8914E-03
    RMS SMIN_NO3_vr                      9.7396E-05            NORMALIZED  3.8744E-05
    RMS SMP                              5.5108E-01            NORMALIZED  1.6324E-08
    RMS SNOBCMCL                         9.0031E-15            NORMALIZED  8.4493E-09
    RMS SNOBCMSL                         2.3766E-14            NORMALIZED  8.8963E-07
    RMS SNODSTMCL                        3.2382E-13            NORMALIZED  1.0720E-09
    RMS SNODSTMSL                        1.5988E-11            NORMALIZED  2.3194E-07
    RMS SNOINTABS                        1.1355E-09            NORMALIZED  1.3548E-08
    RMS SNOOCMCL                         7.7677E-14            NORMALIZED  9.7344E-09
    RMS SNOOCMSL                         1.8766E-13            NORMALIZED  1.3181E-06
    RMS SNOWDP                           1.9534E-08            NORMALIZED  3.0677E-08
    RMS SNOWICE                          3.2007E-06            NORMALIZED  9.1043E-09
    RMS SNOWLIQ                          6.3994E-08            NORMALIZED  7.1156E-07
    RMS SNOW_DEPTH                       2.0029E-08            NORMALIZED  3.1159E-08
    RMS SNOW_SINKS                       1.1187E-12            NORMALIZED  3.8919E-07
    RMS SNOW_SOURCES                     4.4893E-12            NORMALIZED  8.1691E-07
    RMS SOIL1C                           7.9706E-03            NORMALIZED  6.2465E-05
    RMS SOIL1C_TO_SOIL2C                 4.7355E-10            NORMALIZED  3.3237E-04
    RMS SOIL1C_TO_SOIL3C                 3.6950E-12            NORMALIZED  3.2972E-04
    RMS SOIL1C_vr                        3.6156E-02            NORMALIZED  2.5094E-04
    RMS SOIL1N                           9.9633E-04            NORMALIZED  6.2465E-05
    RMS SOIL1N_TO_SOIL2N                 1.1454E-10            NORMALIZED  3.2974E-04
    RMS SOIL1N_TO_SOIL3N                 9.3900E-13            NORMALIZED  3.3260E-04
    RMS SOIL1_HR_S2                      4.5166E-10            NORMALIZED  3.3352E-04
    RMS SOIL1_HR_S3                      3.9311E-12            NORMALIZED  3.4546E-04
    RMS SOIL2C                           8.2996E-03            NORMALIZED  1.7441E-06
    RMS SOIL2C_TO_SOIL1C                 1.0851E-10            NORMALIZED  1.2514E-04
    RMS SOIL2C_TO_SOIL3C                 7.7510E-12            NORMALIZED  1.2514E-04
    RMS SOIL2C_vr                        3.7303E-02            NORMALIZED  1.0353E-05
    RMS SOIL2N                           7.5451E-04            NORMALIZED  1.7441E-06
    RMS SOIL2N_TO_SOIL1N                 2.1922E-11            NORMALIZED  1.2514E-04
    RMS SOIL2N_TO_SOIL3N                 1.5659E-12            NORMALIZED  1.2514E-04
    RMS SOIL2_HR_S1                      1.3263E-10            NORMALIZED  1.2514E-04
    RMS SOIL2_HR_S3                      9.4735E-12            NORMALIZED  1.2514E-04
    RMS SOIL3C                           3.5425E-04            NORMALIZED  2.1069E-08
    RMS SOIL3C_TO_SOIL1C                 8.5434E-12            NORMALIZED  2.5243E-04
    RMS SOIL3C_vr                        7.2107E-04            NORMALIZED  1.5759E-07
    RMS SOIL3N                           3.2205E-05            NORMALIZED  2.1069E-08
    RMS SOIL3N_TO_SOIL1N                 1.7259E-12            NORMALIZED  2.5243E-04
    RMS SOIL3_HR                         1.0442E-11            NORMALIZED  2.5243E-04
    RMS SOILC                            1.5881E-02            NORMALIZED  7.3181E-07
    RMS SOILC_HR                         4.9114E-10            NORMALIZED  1.9317E-04
    RMS SOILC_LOSS                       4.9114E-10            NORMALIZED  1.9317E-04
    RMS SOILICE                          1.0736E-05            NORMALIZED  6.9532E-07
    RMS SOILLIQ                          1.5015E-05            NORMALIZED  3.1515E-07
    RMS SOILPSI                          8.0265E-06            NORMALIZED  8.8001E-07
    RMS SOILWATER_10CM                   2.4860E-05            NORMALIZED  8.6001E-07
    RMS SOMC_FIRE                        1.2643E-12            NORMALIZED  9.6630E-05
    RMS SOMHR                            4.9114E-10            NORMALIZED  1.9317E-04
    RMS SOM_C_LEACHED                    5.1678E-17            NORMALIZED  4.6428E-01
    RMS SR                               1.8179E-08            NORMALIZED  1.8447E-03
    RMS STORVEGC                         7.8132E-03            NORMALIZED  4.0279E-05
    RMS STORVEGN                         1.8236E-04            NORMALIZED  2.5207E-05
    RMS SoilAlpha                        1.4840E-07            NORMALIZED  1.5803E-07
    RMS TAUX                             2.3256E-06            NORMALIZED  3.9794E-05
    RMS TAUY                             2.3256E-06            NORMALIZED  3.9794E-05
    RMS TG                               9.0493E-05            NORMALIZED  3.3908E-07
    RMS TG_R                             1.1672E-04            NORMALIZED  4.3043E-07
    RMS TH2OSFC                          9.0486E-05            NORMALIZED  3.3841E-07
    RMS TLAI                             4.3308E-05            NORMALIZED  3.6410E-05
    RMS TOTCOLC                          5.3814E+00            NORMALIZED  2.0566E-04
    RMS TOTCOLCH4                        2.9584E-06            NORMALIZED  7.3591E-06
    RMS TOTCOLN                          1.0628E-02            NORMALIZED  5.2981E-06
    RMS TOTECOSYSC                       5.3814E+00            NORMALIZED  2.0543E-04
    RMS TOTECOSYSN                       1.0628E-02            NORMALIZED  5.2981E-06
    RMS TOTLITC                          8.6443E-01            NORMALIZED  2.5405E-03
    RMS TOTLITC_1m                       8.5684E-01            NORMALIZED  2.6595E-03
    RMS TOTLITN                          1.8727E-03            NORMALIZED  3.2323E-04
    RMS TOTLITN_1m                       1.8570E-03            NORMALIZED  3.4091E-04
    RMS TOTPFTC                          1.6986E-02            NORMALIZED  5.3613E-06
    RMS TOTPFTN                          3.0487E-04            NORMALIZED  1.7614E-05
    RMS TOTSOMC                          1.5881E-02            NORMALIZED  7.3181E-07
    RMS TOTSOMC_1m                       1.5844E-02            NORMALIZED  1.5786E-06
    RMS TOTSOMN                          1.7070E-03            NORMALIZED  8.6337E-07
    RMS TOTSOMN_1m                       1.7033E-03            NORMALIZED  1.8592E-06
    RMS TOTVEGC                          1.7026E-02            NORMALIZED  5.3236E-06
    RMS TOTVEGN                          3.0497E-04            NORMALIZED  1.7623E-05
    RMS TREFMNAV                         5.4874E-06            NORMALIZED  2.0813E-08
    RMS TREFMNAV_R                       6.5242E-06            NORMALIZED  2.4361E-08
    RMS TREFMXAV                         9.3518E-06            NORMALIZED  3.4570E-08
    RMS TREFMXAV_R                       1.3058E-05            NORMALIZED  4.7463E-08
    RMS TSA                              5.0957E-05            NORMALIZED  1.9081E-07
    RMS TSAI                             5.5688E-07            NORMALIZED  1.2080E-06
    RMS TSA_R                            6.9559E-05            NORMALIZED  2.5637E-07
    RMS TSOI                             5.4518E-05            NORMALIZED  1.9531E-07
    RMS TSOI_10CM                        8.8394E-05            NORMALIZED  3.3060E-07
    RMS TV                               9.0317E-05            NORMALIZED  3.3670E-07
    RMS TWS                              1.0149E-04            NORMALIZED  6.1046E-09
    RMS T_SCALAR                         2.4315E-06            NORMALIZED  5.1334E-06
    RMS U10                              4.1739E-05            NORMALIZED  1.7393E-05
    RMS VOLR                             1.1760E+02            NORMALIZED  3.0420E-06
    RMS VOLRMCH                          1.1760E+02            NORMALIZED  3.0420E-06
    RMS WA                               6.6299E-07            NORMALIZED  1.7049E-10
    RMS WF                               7.6288E-07            NORMALIZED  1.8719E-06
    RMS WOODC                            6.2795E-03            NORMALIZED  2.2026E-06
    RMS WOODC_ALLOC                      1.3839E-08            NORMALIZED  1.1360E-02
    RMS WOODC_LOSS                       9.2365E-12            NORMALIZED  5.0329E-06
    RMS WTGQ                             1.7718E-07            NORMALIZED  5.6827E-05
    RMS W_SCALAR                         7.0442E-07            NORMALIZED  2.4907E-06
    RMS XSMRPOOL                         4.4778E-05            NORMALIZED  1.4913E-06
    RMS XSMRPOOL_RECOVER                 2.8906E-10            NORMALIZED  8.4736E-05
    RMS ZWT                              7.1843E-09            NORMALIZED  7.4243E-10
    RMS ZWT_PERCH                        1.0836E-09            NORMALIZED  8.2366E-10
 /glade/scratch/andre/SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.C.08041054-45i.ref1/run/SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.C.08041054-45i.ref1.clm2.h1.0001-01-06-00000.nc.base.cprnc.out had the following fields that are NOT b4b  

   RMS ERRSOI                           1.0802E-11            NORMALIZED  9.8458E-02
    RMS FCEV                             2.1398E-04            NORMALIZED  5.0110E-05
    RMS FCTR                             5.3466E-04            NORMALIZED  6.2287E-05
    RMS FGEV                             1.6504E-03            NORMALIZED  1.5315E-04
    RMS FGR                              2.1238E-03            NORMALIZED  5.7092E-05
    RMS FIRA                             3.7410E-04            NORMALIZED  8.9377E-06
    RMS FIRE                             3.7410E-04            NORMALIZED  1.3912E-06
    RMS FSA                              5.0119E-05            NORMALIZED  8.6610E-07
    RMS FSH                              3.6157E-04            NORMALIZED  1.1053E-05
    RMS FSR                              5.0119E-05            NORMALIZED  1.9872E-06
    RMS FSRND                            2.7697E-05            NORMALIZED  2.8083E-06
    RMS FSRNI                            6.9729E-06            NORMALIZED  2.4903E-06
    RMS FSRVD                            1.3340E-05            NORMALIZED  1.7394E-06
    RMS FSRVI                            1.1658E-05            NORMALIZED  2.3844E-06
    RMS H2OSNO                           7.0301E-06            NORMALIZED  5.9972E-08
    RMS H2OSOI                           9.0667E-08            NORMALIZED  4.0058E-07
    RMS LWup                             3.7410E-04            NORMALIZED  1.3912E-06
    RMS QRUNOFF_R                        2.8811E-11            NORMALIZED  4.8193E-06
    RMS RH2M                             6.4134E-04            NORMALIZED  7.4638E-06
    RMS RH2M_R                           8.4780E-04            NORMALIZED  9.5860E-06
    RMS SABG                             3.4148E-04            NORMALIZED  1.0196E-05
    RMS SABV                             3.1059E-04            NORMALIZED  1.2742E-05
    RMS SOILICE                          5.9584E-06            NORMALIZED  1.9159E-08
    RMS SOILLIQ                          9.9000E-06            NORMALIZED  2.5711E-07
    RMS SWup                             5.0119E-05            NORMALIZED  1.9872E-06
    RMS TG                               4.0938E-05            NORMALIZED  1.4680E-07
    RMS TG_R                             9.2633E-05            NORMALIZED  3.4234E-07
    RMS TREFMNAV_R                       2.2204E-05            NORMALIZED  8.2528E-08
    RMS TREFMXAV_R                       2.5098E-05            NORMALIZED  9.0827E-08
    RMS TSA                              1.4401E-05            NORMALIZED  5.2241E-08
    RMS TSA_R                            1.9037E-05            NORMALIZED  6.9910E-08
    RMS TSOI                             2.0249E-05            NORMALIZED  7.2317E-08

---finished summarizing more details of test failures: ----
bandre-ucar commented 8 years ago

As of 8773cb4 the IG failures have been fixed by correctly importing the the cism build utilities directory. Remaining failures on yellowntose are BFAIL for ICLM45ED (ok, not maintained in trunk) and SSP (needs to be fixed). Note: no testing has been done with nag.

ExpectedTestFails file list:
    /glade/p/work/andre/ed/ed-clm-r16x/components/clm/cime_config/testdefs/ExpectedTestFails.xml

    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/ERS_D_Mmpi-serial_Ld5.1x1_brazil.ICLM45ED.yellowstone_intel.clm-edTest does not exist)
    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest does not exist)
    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest does not exist)
    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.yellowstone_pgi.clm-edTest does not exist)
    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest does not exist)
    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/ERS_D_Mmpi-serial_Ld5.1x1_brazil.ICLM45ED.yellowstone_gnu.clm-edTest does not exist)
    BFAIL (baseline directory /glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_11_r188/SMS_Ld5.f19_g16.ICLM45ED.yellowstone_gnu.clm-edTest does not exist)

    FAIL SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_11_r188 file) 
    FAIL SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.clm2.h1.nc : baseline compare clm2.h1 (baseline: compare .base file with clm4_5_11_r188 file) 
    FAIL SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default : baseline compare summary (baseline: compare .base file with clm4_5_11_r188 file) 

    FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_pgi.clm-default.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_11_r188 file) 
    FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_pgi.clm-default.clm2.h1.nc : baseline compare clm2.h1 (baseline: compare .base file with clm4_5_11_r188 file) 
    FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_pgi.clm-default : baseline compare summary (baseline: compare .base file with clm4_5_11_r188 file)
ckoven commented 8 years ago

@bandre-ucar I suspect the FAIL for SSP is related to the bug I identified in https://github.com/bandre-ucar/ed-clm/pull/5 and which I have since closed and put into 1fdb59c instead. I am almost ready to submit a PR on that branch and hope that it will solve the default CLM restart problem.

bandre-ucar commented 8 years ago

Quick test of SSP with 1fdb59c978a based on r183.

Steps to recreate spinup test

./create_test -testname SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default -testid junk-ssp -compare clm4_5_9_r183
cd SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.C.junk-ssp/
./case.test_build 
./case.submit 
cat TestStatus

Results:

cat TestStatus
DONE SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default : (test finished, successful coupler log) 
 --- Test Functionality: ---
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default : successful coupler log 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default : test functionality summary 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.memleak
--- Baseline Comparison ---: 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.clm2.h0.nc : baseline compare clm2.h0 (baseline: compare .base file with clm4_5_9_r183 file) 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.clm2.h1.nc : baseline compare clm2.h1 (baseline: compare .base file with clm4_5_9_r183 file) 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default : baseline compare summary (baseline: compare .base file with clm4_5_9_r183 file) 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.memcomp.clm4_5_9_r183 
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.tputcomp.clm4_5_9_r183 
--- Test time is 332 seconds ---
PASS SSP_D_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default.C.junk-ssp.nlcomp

@ckoven #97 appears to fix the problem.

ckoven commented 8 years ago

@bandre-ucar Great, thanks!

bandre-ucar commented 8 years ago

As of e6d830e the full clm test suite on yellowstone gnu, intel pgi, passes bit for bit r188, with the exception of the ED tests, which are not maintained in clm-trunk.

Testing on hobart with nag compiler:

ExpectedTestFails file list:
    /home/andre/work/ed/ed-clm/components/clm/cime_config/testdefs/ExpectedTestFails.xml

CFAIL ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edNoFire.G.08151440-edn
CFAIL SMS_D_Mmpi-serial_Ld5.5x5_amazon.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL SMS_Ld5.f19_g16.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL ERS_D_Ld5.f45_g37.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL SMS_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL ERI_D_Ld9.f09_g16.ICLM45BGC.hobart_nag.clm-default.G.08151440-edn
CFAIL ERS_D_Mmpi-serial_Ld5.1x1_brazil.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL ERS_D_Ld5.5x5_amazon.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL SMS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL ERS_D_Ld5.f19_g16.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn
CFAIL SMS_D_Ld3.f10_f10.ICLM45BGC.hobart_nag.clm-default.G.08151440-edn
CFAIL ERS_D_Ld5.f09_g16.ICLM45ED.hobart_nag.clm-edTest.G.08151440-edn

RUN ERI_D_Ld9.f09_g16.ICLM45BGC.hobart_pgi.clm-default.G.08151440-edp
RUN SMS_D_Ld3.f10_f10.ICLM45BGC.hobart_pgi.clm-default.G.08151440-edp 

Error: /home/andre/work/ed/ed-clm/components/clm/src/ED/biogeochem/EDCohortDynamicsMod.F90, line 222: D is not a component of UDATA
       detected at %@D

 Flushing FATES control parameters prior to transfer from host
 Transfering num_sw_bbands =             2  to FATES
 Transfering num_lev_ground =            15  to FATES
 Transfering num_levdecomp_full =            15  to FATES
 Checked. All control parameters sent to FATES.
 ERROR: Unknown error submitted to shr_sys_abort.
 masterlist_addfld ERROR:CROPPROD1ROOTN_TO already on list
 ENDRUN:
 ERROR in /home/andre/work/ed/ed-clm/components/clm/src/main/histFileMod.F90 at line 340       
 ERROR: Unknown error submitted to shr_sys_abort.
  masterlist_addfld ERROR:^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ already on list
 ENDRUN:
 ERROR in /home/andre/work/ed/ed-clm/components/clm/src/main/histFileMod.F90 at line 340
bandre-ucar commented 8 years ago

8e8ebca passed full clm test suite for yellowstone gnu, intel, pgi

bandre-ucar commented 8 years ago

Note to self: The first nag compiler error above refers to the userdata%deltat field. Appears to be related to line length issues. see EDCohortDynamicsMod.F90, line 222

bandre-ucar commented 8 years ago

Fixed nag compilation errors. Getting a runtime error:

Runtime Error: /home/andre/work/ed/ed-clm/components/clm/src/soilbiogeochem/Soil
BiogeochemPotentialMod.F90, line 105: Reference to undefined POINTER
Program terminated by fatal error
/home/andre/work/ed/ed-clm/components/clm/src/soilbiogeochem/SoilBiogeochemPoten
tialMod.F90, line 105: Error occurred in SOILBIOGEOCHEMPOTENTIALMOD:SOILBIOGEOCH
EMPOTENTIAL
/home/andre/work/ed/ed-clm/components/clm/src/biogeochem/EDBGCDynMod.F90, line 194: Called by EDBGCDYNMOD:EDBGCDYN
/home/andre/work/ed/ed-clm/components/clm/src/main/clm_driver.F90, line 850: Called by CLM_DRIVER:CLM_DRV
/home/andre/work/ed/ed-clm/components/clm/src/cpl/lnd_comp_mct.F90, line 436: Called by LND_COMP_MCT:LND_RUN_MCT
/home/andre/work/ed/ed-clm/cime/driver_cpl/driver/component_mod.F90, line 1079: Called by COMPONENT_MOD:COMPONENT_RUN
/home/andre/work/ed/ed-clm/cime/driver_cpl/driver/cesm_comp_mod.F90, line 2509: Called by CESM_COMP_MOD:CESM_RUN
/home/andre/work/ed/ed-clm/cime/driver_cpl/driver/cesm_driver.F90, line 93: Call

The undefined pointer is: decomp_cascade_con%cascade_receiver_pool is not being allocated? I remember seeing something with this before, but I can't remember why.....

@rgknox , @ckoven or @rosiealice do any of you recognize this?

ckoven commented 8 years ago

@bandre-ucar this is something that should probably be my problem to deal with. is it calling init_decomp_cascade_constants?

if so, looking at the code in init_decomp_cascade_constants, there is also the issue that some of these things are being indexed 0:x whereas others are being indexed 1:x, when all should be indexed either 1:ndecomp_pools or 1:ndecomp_cascade_transitions. but I don't think that ought to be the problem...

bandre-ucar commented 8 years ago

@ckoven : The problem is only present with nag, not intel or pgi, and doesn't occur in bgc mode, just ed.

It looks like nag isn't reporting the error correctly. More debugging points to the soilbiogeochem_nitrogenstate_inst not being initialized. Looking at the code, that struct is only allocated for use_cn. It is not initialized with use_ed. Is that intentional, or a bug? Adding use_ed to the condition allows the simulation to run further and die with a similar looking error for a different data structure.

bandre-ucar commented 8 years ago

@ckoven Next error: it looks like crop_inst is used inside EDBGCDynMod the crop_inst needs to be initialized or removed from ED.

ckoven commented 8 years ago

OK, I guess that is consistent then. I had thought it was ok to pass uninitialized instances as arguments when they were not used (they need to be passed since the EDBGCDynMod calls classic CLM BGC routines that take those instances as arguments, but here we've turned off those parts of the code for the time being). if not, then I guess we need to initialize both the soilbiogeochem_nitrogenstate_inst and the crop_inst.

bandre-ucar commented 8 years ago

next error:

Runtime Error: /home/andre/work/ed/ed-clm/components/clm/src/ED/biogeochem/EDCohortDynamicsMod.F90, line 198: Reference to dangling pointer CURRENTCOHORT%SITEPTR
Target was RETURNed from procedure EDINITMOD:INIT_PATCHES
Program terminated by fatal error
/home/andre/work/ed/ed-clm/components/clm/src/ED/biogeochem/EDCohortDynamicsMod.F90, line 198: Error occurred in EDCOHORTDYNAMICSMOD:ALLOCATE_LIVE_BIOMASS
/home/andre/work/ed/ed-clm/components/clm/src/ED/biogeochem/EDPhysiologyMod.F90, line 775: Called by EDPHYSIOLOGYMOD:GROWTH_DERIVATIVES
/home/andre/work/ed/ed-clm/components/clm/src/ED/biogeochem/EDPhysiologyMod.F90, line 67: Called by EDPHYSIOLOGYMOD:CANOPY_DERIVS
/home/andre/work/ed/ed-clm/components/clm/src/ED/main/EDMainMod.F90, line 171: Called by EDMAINMOD:ED_INTEGRATE_STATE_VARIABLES
/home/andre/work/ed/ed-clm/components/clm/src/ED/main/EDMainMod.F90, line 79: Called by EDMAINMOD:ED_ECOSYSTEM_DYNAMICS
/home/andre/work/ed/ed-clm/components/clm/src/utils/clmfates_interfaceMod.F90, line 471: Called by CLMFATESINTERFACEMOD:DYNAMICS_DRIV
/home/andre/work/ed/ed-clm/components/clm/src/main/clm_driver.F90, line 837: Called by CLM_DRIVER:CLM_DRV
/home/andre/work/ed/ed-clm/components/clm/src/cpl/lnd_comp_mct.F90, line 436: Called by LND_COMP_MCT:LND_RUN_MCT
/home/andre/work/ed/ed-clm/cime/driver_cpl/driver/component_mod.F90, line 1079: Called by COMPONENT_MOD:COMPONENT_RUN

I'm off for a few hours. Maybe someone can look at the above error. My most recent commit is: d9b887c02734148

rgknox commented 8 years ago

Here is one hypothesis:

Cohorts are created during create_cohort(). When this happens, the cohort's site pointer, points to the cohort's patch's site pointer. So its a pointing to a pointer. A dangling pointer is created because the cohort continues to exist on a new patch, but it's original patch ceases to exist. I need to look at patch fusion again, but we would cirumvent this problem by passing the site as a target directly to the create_cohort subroutine (and pointing to that target).

rgknox commented 8 years ago

or, during EDPatchDynamicsMod.F90: fuse_2_patches()

at line 1257, we can also tell the cohort to update it's siteptr. ie add

currentCohort%siteptr => rp%siteptr

I like the first way better though, set it once, a safer way, during creation, and that is it

bandre-ucar commented 8 years ago

I hacked around the dangling pointer problem by making sites a pointer instead of an allocatable. The compilers and runtime are happy, but it's still a dangerous design that essentially creates globals out of sites. It unnecessarily opens up the possibility of bugs. A much better approach is just pass the site level information needed by the patches and cohorts down through function calls, updating as necessary.

for patch in site%patches():
    patch%update_site_info(...)

patch::update_site_info(...):
    for cohort in patch%cohorts():
        cohort%update_site_info(...)
bandre-ucar commented 8 years ago

in clm4_5_12_r192