NGEET / fates

repository for the Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
Other
99 stars 92 forks source link

canopy structure crash #442

Closed ckoven closed 5 years ago

ckoven commented 5 years ago

In some of the ensembles I've been running, competing two PFTs at the BCI testbed, and initialized from inventory, I'm seeing a consistent error from a couple of the ensemble members. This happens both with and without the changes in #441.

I'm not sure what's going here. it seems to happen with two of the ensemble members. both ensemble memebrs aer generating unrealistically large trees, which may be what's happening. possibly something like the error in conservation of crown area when fusing trees that are large means that more iterations are required to shuffle the canopy around. so possibly its as simple as increasing the allowable number of iterations.

here's the relevant part of the cesm log file of one of the ensemble members that crashes:

586:  ERROR: Unknown error submitted to shr_abort_abort.
586: Image              PC                Routine            Line        Source             
586: cesm.exe           00000000013FF426  Unknown               Unknown  Unknown
586: cesm.exe           0000000000DF5E7F  shr_abort_mod_mp_         114  shr_abort_mod.F90
586: cesm.exe           00000000007C73B7  fatesglobals_mp_f          62  FatesGlobals.F90
586: cesm.exe           000000000076DD08  edcanopystructure         266  EDCanopyStructureMod.F90
586: cesm.exe           000000000077BCB6  edmainmod_mp_ed_u         391  EDMainMod.F90
586: cesm.exe           00000000005381F5  clmfatesinterface         670  clmfates_interfaceMod.F90
586: cesm.exe           0000000000512FD8  clm_driver_mp_clm         879  clm_driver.F90
586: cesm.exe           00000000004FF415  lnd_comp_mct_mp_l         451  lnd_comp_mct.F90
586: cesm.exe           0000000000433C9F  component_mod_mp_         743  component_mod.F90
586: cesm.exe           00000000004180EF  cime_comp_mod_mp_        2677  cime_comp_mod.F90
586: cesm.exe           000000000043391D  MAIN__                    125  cime_driver.F90
586: cesm.exe           0000000000415A5E  Unknown               Unknown  Unknown
586: libc-2.22.so       00002AAAB2947725  __libc_start_main     Unknown  Unknown
586: cesm.exe           0000000000415969  Unknown               Unknown  Unknown
586: Rank 586 [Sat Nov 10 19:40:45 2018] [c1-3c0s13n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 586

The lnd log from the first ensemble member that crashes ends with this:

 clm: completed timestep       433439
 clm: completed timestep       433440
 clm: calling FATES model       433441
FATES Dynamics:   26-02-26
 PATCH AREA CHECK NOT CLOSING
 patch area:   5298.27184705595     
 layer:            1  area:    5297.50091097957     
 rel error:  -1.455070820520817E-004
 layer:            2  area:    5298.27193409247     
 rel error:   1.642734100712646E-008
 layer:            3  area:    5289.79277669230     
 rel error:  -1.600346416419225E-003
 lat:   9.15300000000000     
 lon:   280.153900000000     
 spread:  0.000000000000000E+000
 coh ilayer:           1
 coh dbh:   1476.16889026868     
 coh pft:           1
 coh n:  2.233505020155443E-003
 coh carea:   78.5612143410932     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1358.23389205525     
 coh pft:           1
 coh n:  1.340103012093285E-002
 coh carea:   414.801832372711     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1272.06800034316     
 coh pft:           1
 coh n:  3.126907028217620E-002
 coh carea:   875.218071626878     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1179.37472851914     
 coh pft:           1
 coh n:  6.030463554419742E-002
 coh carea:   1502.81075478663     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1078.52925770195     
 coh pft:           1
 coh n:  0.111675251007774     
 coh carea:   2426.10903785226     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           2
 coh dbh:   968.673234123878     
 coh pft:           1
 coh n:  0.170708629075018     
 coh carea:   3144.70521874260     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           2
 coh dbh:   871.324382504735     
 coh pft:           1
 coh n:  0.137547840793408     
 coh carea:   2153.56671534987     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           3
 coh dbh:   856.240799321923     
 coh pft:           1
 coh n:  7.904155844870081E-002
 coh carea:   1204.80400078292     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           3
 coh dbh:   753.528330616068     
 coh pft:           1
 coh n:  0.255160871786085     
 coh carea:   3196.47562654562     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           3
 coh dbh:   667.421179669445     
 coh pft:           1
 coh n:  8.545083297459678E-002
 coh carea:   888.513149363755     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 ENDRUN:
 ERROR in EDCanopyStructureMod.F90 at line 266

here's the lnd log from the second ensemble member that crashes:

 clm: completed timestep       391293
 clm: completed timestep       391294
 clm: completed timestep       391295
 clm: completed timestep       391296

 hist_htapes_wrapup : Writing current time sample to local history file 
 ./fates_ctsm_cori_fullmodel_bci_parameter_ensemble_2pfts_v010_multidriver_inven
 tory_959f7b55_81862b9.clm2_0022.h0.0021-07-01-00000.nc at nstep =       391296 
  for history time interval beginning at    8122.00000000000       and ending at
     8152.00000000000     

 clm: calling FATES model       391297
FATES Dynamics:   23-10-01
 PATCH AREA CHECK NOT CLOSING
 patch area:   5987.15963920766     
 layer:            1  area:    5987.14814579372     
 rel error:  -1.919677215756220E-006
 layer:            2  area:    5987.15965397446     
 rel error:   2.466410644689500E-009
 layer:            3  area:    5986.98829091597     
 rel error:  -2.861929562877841E-005
 lat:   9.15300000000000     
 lon:   280.153900000000     
 spread:  0.000000000000000E+000
 coh ilayer:           1
 coh dbh:   332.303764976510     
 coh pft:           2
 coh n:  1.699701547935756E-002
 coh carea:   55.5486432750394     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   310.693186485133     
 coh pft:           1
 coh n:  3.750888941575140E-003
 coh carea:   10.9983728699655     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   265.670197946282     
 coh pft:           2
 coh n:  1.699701547935756E-002
 coh carea:   39.0430237554909     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   265.064574801938     
 coh pft:           1
 coh n:  1.125266682472535E-002
 coh carea:   25.9831397632638     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   248.172805721233     
 coh pft:           1
 coh n:  7.501777883150280E-003
 coh carea:   15.6886192271532     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   245.729187177944     
 coh pft:           2
 coh n:  2.549552321903634E-002
 coh carea:   51.7900097975698     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   234.650736101460     
 coh pft:           1
 coh n:  1.125266682472535E-002
 coh carea:   21.6309695151070     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   218.868798323253     
 coh pft:           2
 coh n:  4.249253869839360E-002
 coh carea:   71.9263581262207     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   218.522889752619     
 coh pft:           1
 coh n:  1.875444470787575E-002
 coh carea:   32.3897904578002     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   201.688018410478     
 coh pft:           2
 coh n:  3.399403095871513E-002
 coh carea:   50.5872032031601     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   197.642435485223     
 coh pft:           1
 coh n:  1.500355576630056E-002
 coh carea:   22.2787546254652     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   191.102059971572     
 coh pft:           2
 coh n:  5.948955417775113E-002
 coh carea:   81.3182461288779     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   184.024204051326     
 coh pft:           1
 coh n:  1.875444470787575E-002
 coh carea:   25.0128741027392     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   181.072788847637     
 coh pft:           2
 coh n:  5.099104643807267E-002
 coh carea:   64.0257272710618     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   172.674284773391     
 coh pft:           1
 coh n:  3.750888941575140E-003
 coh carea:   4.54576768540083     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   169.868853006303     
 coh pft:           2
 coh n:  0.110480600615823     
 coh carea:   125.441773228107     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   159.850039681620     
 coh pft:           2
 coh n:  5.099104643807267E-002
 coh carea:   52.6083145118860     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   159.808147271210     
 coh pft:           1
 coh n:  1.125266682472535E-002
 coh carea:   12.1379393076483     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   151.742569416396     
 coh pft:           2
 coh n:  0.161471647053896     
 coh carea:   153.476287703654     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   150.647303167007     
 coh pft:           1
 coh n:  3.750888941575140E-003
 coh carea:   3.70220388925563     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   138.430607798808     
 coh pft:           1
 coh n:  4.501066729890139E-002
 coh carea:   39.1195971411529     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   137.169183353170     
 coh pft:           2
 coh n:  0.526907479860080     
 coh carea:   427.160219421672     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   125.626111531919     
 coh pft:           1
 coh n:  4.876155624047691E-002
 coh carea:   36.6228530320512     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   125.480219412131     
 coh pft:           2
 coh n:  0.518408972120404     
 coh carea:   365.245817152120     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   116.362234481051     
 coh pft:           2
 coh n:  0.671382111434627     
 coh carea:   420.013190539569     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   114.531019064711     
 coh pft:           1
 coh n:  6.001422306520227E-002
 coh carea:   39.2216584800388     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   108.203610537452     
 coh pft:           2
 coh n:  0.764865696571081     
 coh carea:   426.715336951047     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   107.867761982446     
 coh pft:           1
 coh n:  0.127530224013554     
 coh carea:   76.1603650093424     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   102.424025406374     
 coh pft:           2
 coh n:  0.798859727529803     
 coh carea:   408.754920065573     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   100.129777259495     
 coh pft:           1
 coh n:  0.112526668247253     
 coh carea:   60.0819127317284     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   96.1345011368718     
 coh pft:           2
 coh n:   1.53822990088187     
 coh carea:   712.280356828925     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   93.9743509771417     
 coh pft:           1
 coh n:  0.157537335546156     
 coh carea:   76.4585170637369     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   89.4751012240644     
 coh pft:           2
 coh n:   1.57222393184057     
 coh carea:   650.164959109759     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   89.1182861106143     
 coh pft:           1
 coh n:  0.157537335546156     
 coh carea:   70.5936926561068     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   84.2401804564165     
 coh pft:           2
 coh n:   1.77618811759287     
 coh carea:   667.953208294733     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           1
 coh dbh:   83.8325292841831     
 coh pft:           1
 coh n:  0.202548002845056     
 coh carea:   82.7881381425653     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   77.7357040742685     
 coh pft:           1
 coh n:  0.483568214033594     
 coh carea:   176.430203163214     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           1
 coh dbh:   69.1361377283111     
 coh pft:           2
 coh n:   1.20254385449582     
 coh carea:   331.249181565516     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   65.7629070225745     
 coh pft:           2
 coh n:  0.413220696646158     
 coh carea:   105.198335943361     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   59.1815824833392     
 coh pft:           2
 coh n:   1.38901022496955     
 coh carea:   299.489462358436     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   54.4703272287356     
 coh pft:           2
 coh n:   2.46765552787628     
 coh carea:   466.873801649516     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   50.3050982918434     
 coh pft:           2
 coh n:   2.21834600031765     
 coh carea:   370.265048386408     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   47.2881081390483     
 coh pft:           1
 coh n:  0.249246889708312     
 coh carea:   43.0567007606111     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   47.0201413545430     
 coh pft:           2
 coh n:   2.09114726176735     
 coh carea:   313.805581165780     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   44.4351547069239     
 coh pft:           2
 coh n:   2.31501704161592     
 coh carea:   317.789446072503     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   43.2143645990899     
 coh pft:           1
 coh n:  0.350292926076546     
 coh carea:   52.8437108496522     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   41.9530729148293     
 coh pft:           2
 coh n:   2.61520606459468     
 coh carea:   327.914976420185     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   39.3194656883723     
 coh pft:           2
 coh n:   3.42927799131675     
 coh carea:   388.237082354355     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   39.1892554839903     
 coh pft:           1
 coh n:  0.285174369305906     
 coh carea:   37.1367136274903     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   36.7162133002305     
 coh pft:           1
 coh n:  0.226792214959814     
 coh carea:   26.7756013056585     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   36.5397504977301     
 coh pft:           2
 coh n:   5.62218424392429     
 coh carea:   567.062829228378     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   34.2062325715710     
 coh pft:           1
 coh n:  0.348047458601694     
 coh carea:   36.9396009097389     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   32.7263675014302     
 coh pft:           2
 coh n:   3.82105010605172     
 coh carea:   323.960089938764     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   32.0042294106385     
 coh pft:           1
 coh n:  0.327838251328051     
 coh carea:   31.4808100307202     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   29.8969629215489     
 coh pft:           2
 coh n:   4.02965603727431     
 coh carea:   296.281385367409     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   29.6564276909733     
 coh pft:           1
 coh n:  0.518702986690265     
 coh carea:   44.4154497447060     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   27.1964807972773     
 coh pft:           1
 coh n:  0.446848027495079     
 coh carea:   33.5899191957754     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   27.1673824559947     
 coh pft:           2
 coh n:   5.07777364292893     
 coh carea:   321.070307430075     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   25.5718196449444     
 coh pft:           1
 coh n:  0.345801991126844     
 coh carea:   23.6939952166350     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   24.1138159644522     
 coh pft:           2
 coh n:   7.45893402859095     
 coh carea:   390.858423859310     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   24.0507636028524     
 coh pft:           1
 coh n:  0.514212051740571     
 coh carea:   32.1286676819697     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   21.8785856665633     
 coh pft:           1
 coh n:  0.424260308159269     
 coh carea:   22.9904138644994     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   20.4576703311396     
 coh pft:           2
 coh n:   8.18142286355690     
 coh carea:   330.874275052714     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   19.8404640456819     
 coh pft:           1
 coh n:  0.153927581635082     
 coh carea:   7.20032991212008     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   18.1470646299840     
 coh pft:           1
 coh n:  4.513846161980330E-002
 coh carea:   1.84630214567602     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   17.9421921614183     
 coh pft:           2
 coh n:   6.81785238629736     
 coh carea:   224.236726115540     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   16.5910733149125     
 coh pft:           1
 coh n:  2.169079406050338E-002
 coh carea:  0.775302293309614     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   15.9787263007824     
 coh pft:           2
 coh n:   9.20918867104352     
 coh carea:   252.335642722835     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   14.9810286796028     
 coh pft:           1
 coh n:  1.472746670015973E-002
 coh carea:  0.451480813160760     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   13.2455596092200     
 coh pft:           1
 coh n:  1.547298654081944E-002
 coh carea:  0.394144216416294     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   13.0693404441190     
 coh pft:           2
 coh n:   8.71249838793517     
 coh carea:   173.929769186406     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   11.9699934518326     
 coh pft:           2
 coh n:   2.69468448525792     
 coh carea:   46.8400249315480     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           2
 coh dbh:   11.7832695610241     
 coh pft:           1
 coh n:  1.911542414079549E-002
 coh carea:  0.408362715647950     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   10.4571207104012     
 coh pft:           1
 coh n:  1.491753652278152E-002
 coh carea:  0.266294513018819     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           2
 coh dbh:   10.0879766886809     
 coh pft:           2
 coh n:   5.55463758738435     
 coh carea:   73.7426459941266     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   9.67368582077219     
 coh pft:           2
 coh n:   7.19576396490013     
 coh carea:   89.4223583221061     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   9.56860895870525     
 coh pft:           1
 coh n:  1.200191892181778E-002
 coh carea:  0.187460519257824     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   8.86106561390774     
 coh pft:           2
 coh n:   14.0529166350399     
 coh carea:   152.089181560069     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   8.76863908102501     
 coh pft:           1
 coh n:  1.423157770013812E-002
 coh carea:  0.194930119365445     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   8.16475026222954     
 coh pft:           2
 coh n:   16.0473928555088     
 coh carea:   152.664142285120     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   8.00064789414967     
 coh pft:           1
 coh n:  1.635283541580812E-002
 coh carea:  0.195138161424562     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   7.53594220747003     
 coh pft:           2
 coh n:   19.1052505302587     
 coh carea:   160.194747432223     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   7.27468343806651     
 coh pft:           1
 coh n:  1.636665335160608E-002
 coh carea:  0.169266088530799     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   6.91455500628721     
 coh pft:           2
 coh n:   23.4605353382221     
 coh carea:   171.771257072912     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   6.60493231258378     
 coh pft:           1
 coh n:  1.460652542727197E-002
 coh carea:  0.130636119078803     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   6.51466259255950     
 coh pft:           2
 coh n:   12.8928641394608     
 coh carea:   85.9407756565251     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   6.18147149942724     
 coh pft:           2
 coh n:   19.5784298376656     
 coh carea:   120.144929584806     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   6.14235109227911     
 coh pft:           1
 coh n:  1.265595115412834E-002
 coh carea:  0.101479631109893     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   5.84226173276176     
 coh pft:           1
 coh n:  3.084693519117804E-003
 coh carea:  2.293901917408659E-002
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   5.79823397308805     
 coh pft:           2
 coh n:   17.2654851913306     
 coh carea:   95.7883311389344     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 coh ilayer:           3
 coh dbh:   5.53806785679859     
 coh pft:           1
 coh n:  1.388425311034904E-002
 coh carea:  9.526958675770282E-002
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   5.10558608131793     
 coh pft:           1
 coh n:  1.723599152948222E-002
 coh carea:  0.104652977576885     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.50417518615723     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.522745966911316     
 d2ca_max:   0.522745966911316     
 coh ilayer:           3
 coh dbh:   1.16812724576527     
 coh pft:           2
 coh n:   11153.6830754652     
 coh carea:   4957.77079564100     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.57553446292877     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.347964614629745     
 d2ca_max:   0.347964614629745     
 ENDRUN:
 ERROR in EDCanopyStructureMod.F90 at line 267                                  
rgknox commented 5 years ago

The relative error in the patches is definitely too large, its not a case of "the error checker is too strict". I am noticing that the crown areas are really massive, EDIT, I thought is was printing the per-plant cohort area... redacting this statement.

like some are about the size of the patch itself ~5k m2, for instance the last cohort printed. It is possible that with such massive crowns, that when we pare off the number density for promotion/demotion the math precision gets compromised...

rgknox commented 5 years ago

@ckoven , I'm noticing also that there are 3 layers, each of which are roughly the size of the patch. I'm curious, did you bump up nclmax to 3?

ckoven commented 5 years ago

Just discussed this with @rgknox. While the conditions here that are causing the crash are clearly unrealistic (trees that are gigantic), there may be something more relevant to general conditions here in what seems to be happening. What I think is happening is that in the midst of the canopy structure loop there is a cohort fusion step. Because crown area is not conserved during cohort fusion, this then leads to total crown areas not being what was anticipated from the promotion/demotion. There is an iteration loop to allow this to be sorted out, but in the case of these big trees (possibly only relevant when trees in canopy layers 2 or 3 are large), it has to iterate several times and so is running into the max iteration logic.

So the key question is whether the lack of crown area conservation is important here more generally. Now, when we fuse cohorts, we conserve biomass (important!) and take a weighted average of DBH. Because crown area is DBH to some power, we can't conserve crown area. Because biomass is DBH to some (larger) power, we also don't conserve the relationship between DBH and biomass, but the flexible allometry scheme allows us to temporarily go off allometry and then grown back towards allometry on subsequent timesteps. A possible approach to this problem would be to, instead of making the fused cohort's DBH a population-weighted average of the fusing cohorts' DBH, to instead conserve the total crown area and then calculate the DBH of the fused cohort as that which conserves crown area during fusion. This should avoid the iteration loop and give us exactly the crown areas that we seek.

The short term fix that I'm exploring for this particular case (which, again, is unrealistic) is to leave the logic as-is and instead just increase the number of iterations permitted. But we may want to revisit the question of whether or not we should try to conserve crown area during cohort fusion in general.

rgknox commented 5 years ago

Another thing to consider is that we force the upper layers to be "perfectly" plastic. Ideally, we would have a canopy that is imperfect, which allows for some gap space between crowns, and a radiation transfer that allows for some bypass of radiation through each layer.

If we code this in, then we may be able to get away with less precision? ... because the promotion/demotion error will be subsumed by the open gap area... thinking extemporaneously here

ckoven commented 5 years ago

just to say that increasing the number of iterations from 10 to 20 doesn't solve this, i still get a crash in the same place. log file pasted below, the differences start at the boundary between the first and second canopy layers:

 clm: completed timestep       433439
 clm: completed timestep       433440
 clm: calling FATES model       433441
FATES Dynamics:   26-02-26
 PATCH AREA CHECK NOT CLOSING
 patch area:   5298.27184704583     
 layer:            1  area:    5297.50091423565     
 rel error:  -1.455064655875225E-004
 layer:            2  area:    5298.27193345459     
 rel error:   1.630885784912938E-008
 layer:            3  area:    5282.08435213392     
 rel error:  -3.055240534881608E-003
 lat:   9.15300000000000     
 lon:   280.153900000000     
 spread:  0.000000000000000E+000
 coh ilayer:           1
 coh dbh:   1476.16889080131     
 coh pft:           1
 coh n:  2.233505020155443E-003
 coh carea:   78.5612143846136     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1358.23389256322     
 coh pft:           1
 coh n:  1.340103012093285E-002
 coh carea:   414.801832610886     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1272.06800083262     
 coh pft:           1
 coh n:  3.126907028217620E-002
 coh carea:   875.218072143910     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1179.37472898808     
 coh pft:           1
 coh n:  6.030463554419742E-002
 coh carea:   1502.81075570403     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           1
 coh dbh:   1078.52925814784     
 coh pft:           1
 coh n:  0.111675251007774     
 coh carea:   2426.10903939221     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           2
 coh dbh:   968.673234548291     
 coh pft:           1
 coh n:  0.170290134074354     
 coh carea:   3136.99592449363     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           2
 coh dbh:   871.269606347760     
 coh pft:           1
 coh n:  0.138053556046577     
 coh carea:   2161.27600896097     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           3
 coh dbh:   856.240810343689     
 coh pft:           1
 coh n:  7.853584316935310E-002
 coh carea:   1197.09557582510     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           3
 coh dbh:   753.528330790387     
 coh pft:           1
 coh n:  0.255160873131248     
 coh carea:   3196.47564453217     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 coh ilayer:           3
 coh dbh:   667.421179856545     
 coh pft:           1
 coh n:  8.545083124641673E-002
 coh carea:   888.513131776645     
 maxh:   99999.0000000000     
 lmode:    3.00000000000000     
 d2bl2:    1.53530120849609     
 d2bl_ediff:   0.000000000000000E+000
 d2ca_min:   0.479339092969894     
 d2ca_max:   0.479339092969894     
 ENDRUN:
 ERROR in EDCanopyStructureMod.F90 at line 267       
ckoven commented 5 years ago

just to say that i tried reworking the cohort fusion to conserve crown area instead of total dbh (212a056) and so far it least it fixes this problem. so we may want to discuss further whether that should be the general solution.

rosiealice commented 5 years ago

I've no objection in principle to using crown area, particularly given it's a more physiologically relevant property, and that neither option is 'realistic' anyway.

Le sam. 17 nov. 2018 à 21:17, Charlie Koven notifications@github.com a écrit :

just to say that i tried reworking the cohort fusion to conserve crown area instead of total dbh (212a056 https://github.com/NGEET/fates/commit/212a0564c826d7ab0519e38f4a045e13a44842fc) and so far it least it fixes this problem. so we may want to discuss further whether that should be the general solution.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NGEET/fates/issues/442#issuecomment-439644753, or mute the thread https://github.com/notifications/unsubscribe-auth/AMWsQ85_nLVX3zZABgWwX0KJnW-SHJ6Mks5uwG7egaJpZM4YYrwf .

--

Dr Rosie A. Fisher

Staff Scientist Terrestrial Sciences Section Climate and Global Dynamics National Center for Atmospheric Research 1850 Table Mesa Drive Boulder, Colorado, 80305, USA

and

Visitor @ C.E.R.F.A.C.S Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique 42 Avenue Gaspard Coriolis 31057, Toulouse, France http://www.cgd.ucar.edu/staff/rfisher/