CH-Earth / summa

Structure for Unifying Multiple Modeling Alternatives:
http://www.ral.ucar.edu/projects/summa
GNU General Public License v3.0
79 stars 103 forks source link

Restart error in satHydCond() #550

Open NicWayand opened 8 months ago

NicWayand commented 8 months ago

Bug Reports

- Version of SUMMA that you are using (e.g. SUMMA 1.0 - even better if you can provide the specific tag or commit) develop - 06f557181f9c2ae02fbb857ab828415b9dabc280 - Name and version of the fortran compiler you are using gfortran - Operating system Linux - A description of relevant model settings Running one GRU with 296 HRUs (greater Seattle area). I do a cold start and run for 24 hours, saving the restart file. Then I do another run but get the below error message before the first time step.

- A summary of the bug or error message you are getting

Below is the back trace. I only get this error using the "Debugging" gfortran Make file flags, "production" does not raise this error. I have checked the IC file that it is reading and all variable look reasonable to me. The only unknown I see is some of the layer variables (MLayerTemp) have values of 9.96920997e+36 (at snow layer heights when nSnow=0). From the error message it appears mlayersathydcondmp does not have the correct length as it expects. The line it fails on is here. Not sure what to check next, so any suggestions are appreciated!

`At line 370 of file /summa/build/source/engine/var_derive.f90 Fortran runtime error: Index '0' of dimension 1 of array 'mlayersathydcondmp' below lower bound of 1

Error termination. Backtrace:

0 0x7ff8f87f18b0 in ???

1 0x7ff8f87f2395 in ???

2 0x7ff8f87f273a in ???

3 0x561b88e5b8c0 in __var_derive_module_MOD_sathydcond

at /summa/build/source/engine/var_derive.f90:370

4 0x561b88e074fb in __summa_restart_MOD_summa_readrestart

at /summa/build/source/driver/summa_restart.f90:181

5 0x561b88df82f2 in summa_driver

at /summa/build/source/driver/summa_driver.f90:104

6 0x561b88df85d0 in main

at /summa/build/source/driver/summa_driver.f90:27

file_master is '/tmp/data/6cdad786-0f69-49d0-b3ed-0e1ab69aa512/forecast/settings/summa_fileManager.txt'. GRU-Parallelization run activated. 1 GRUs are selected for simulation. 1 controlVersion: SUMMA_FILE_MANAGER_V3.0.0 2 simStartTime: 2022-12-20 00:00 3 simEndTime: 2022-12-25 23:00 4 tmZoneInfo: utcTime 5 settingsPath: /tmp/data/6cdad786-0f69-49d0-b3ed-0e1ab69aa512/forecast/settings/ 6 forcingPath: /tmp/data/6cdad786-0f69-49d0-b3ed-0e1ab69aa512/forecast/input/ 7 outputPath: /tmp/data/6cdad786-0f69-49d0-b3ed-0e1ab69aa512/forecast/output/ 8 decisionsFile: summa_zDecisions.txt 9 outputControlFile: Model_Output.txt 10 globalHruParamFile: summa_zLocalParamInfo.txt 11 globalGruParamFile: summa_zBasinParamInfo.txt 12 attributeFile: summa_zLocalAttributes.nc 13 trialParamFile: summa_zParamTrial.nc 14 forcingListFile: summa_zForcingFileList.txt 15 initConditionFile: summa_zInitialCond.nc 16 outFilePrefix: forecast 17 vegTableFile: VEGPARM.TBL 18 soilTableFile: SOILPARM.TBL 19 generalTableFile: GENPARM.TBL 20 noahmpTableFile: MPTABLE.TBL decisions file = /tmp/data/6cdad786-0f69-49d0-b3ed-0e1ab69aa512/forecast/settings/summa_zDecisions.txt 1 soilCatTbl: ROSETTA 2 vegeParTbl: USGS 3 soilStress: NoahType 4 stomResist: BallBerry 5 num_method: itertive 6 fDerivMeth: analytic 7 LAI_method: specified 8 f_Richards: mixdform 9 groundwatr: noXplict 10 hc_profile: constant 11 bcUpprTdyn: nrg_flux 12 bcLowrTdyn: zeroFlux 13 bcUpprSoiH: liq_flux 14 bcLowrSoiH: zeroFlux 15 veg_traits: CM_QJRMS1988 16 canopyEmis: simplExp 17 snowIncept: stickySnow 18 windPrfile: logBelowCanopy 19 astability: louisinv 20 canopySrad: CLM_2stream 21 alb_method: varDecay 22 compaction: anderson 23 snowLayers: CLM_2010 24 thCondSnow: smnv2000 25 thCondSoil: funcSoilWet 26 spatial_gw: localColumn 27 subRouting: qInstant startTime: iyyy, im, id, ih, imin = 2022 12 20 0 0 finshTime: iyyy, im, id, ih, imin = 2022 12 25 23 0 number of time steps = 144 Skipping over SLTYPE = STAS Skipping over SLTYPE = STAS-RUC WARNING: routingRunoffFuture is not in the initial conditions file ... using zeros `

NicWayand commented 8 months ago

Update: I can also reproduce this error anytime the cold start has nSnow>0.

NicWayand commented 8 months ago

I think the solution is to change this line from:

if(iLayer > 0)then

to

if(iLayer > 0 .and. iLayer/=nSnow)then To not do the check if the iLayer is at the first snow layer.

wknoben commented 8 months ago

Hi Nic, Thanks for reporting and suggesting a fix. Can you expand on the part where this only occurs with debugging compile flags? It's unclear to me why this would trigger such a specific error and more info might help. If you're able to share a test case that would be great too.

NicWayand commented 8 months ago

I mean when choosing the Production or Debug run settings here. From my understanding of the -O3 flag, gfortran will suppress some errors, which I don't fully understand how it decides which are "ok" or not.

wknoben commented 8 months ago

Seems we ran into this a while ago and have a fix ready, in case you want to compare yours to this one: https://github.com/ashleymedin/summa/pull/2

I'll leave this open until we can merge this PR into our main branches. Thanks for bringing this up.