"PCR-GLOBWB / Rhine 5 Min. / 2002 - 2002 / ERA5"

[sverhoeven]

Failed at simulated date 2002-07-20 with "Exception calling application: Input/output error"

Log of run

``` PCR-GLOBWB (PCRaster Global Water Balance) Global Hydrological Model Copyright (C) 2016, Ludovicus P. H. (Rens) van Beek, Edwin H. Sutanudjaja, Yoshihide Wada, Joyce H. C. Bosmans, Niels Drost, Inge E. M. de Graaf, Kor de Jong, Patricia Lopez Lopez, Stefanie Pessenteiner, Oliver Schmitz, Menno W. Straatsma, Niko Wanders, Dominik Wisser, and Marc F. P. Bierkens, Faculty of Geosciences, Utrecht University, Utrecht, The Netherlands This program comes with ABSOLUTELY NO WARRANTY This is free software, and you are welcome to redistribute it under certain conditions See the LICENSE file for more details 2021-09-03 06:46:14,888 configuration INFO Model run started at 2021-09-03 06:46:14.877912 2021-09-03 06:46:14,888 configuration INFO Logging output to /home/sverhoeven/6b45df02-3c59-4123-b6bc-80d951729ff5/ewatercycle_output/pcrglobwb_20210903_064604/log/pcrglobwb_ewatercycle.ini_2021-09-03T06.46.14.877912.log 2021-09-03 06:46:14,888 configuration INFO Debugging output to /home/sverhoeven/6b45df02-3c59-4123-b6bc-80d951729ff5/ewatercycle_output/pcrglobwb_20210903_064604/log/pcrglobwb_ewatercycle.ini_2021-09-03T06.46.14.877912.dbg 2021-09-03 06:46:14,888 configuration WARNING The option "useMODFLOW" is not defined in the "groundwaterOptions" of the configuration file. We assume "False" for this option. 2021-09-03 06:46:14,888 configuration WARNING The option "maximumDailyGroundwaterAbstraction" is not defined in the "groundwaterOptions" of the configuration file. This run assumes "0.050 m/day" for this option. 2021-09-03 06:46:14,888 configuration WARNING The option "maximumDailyFossilGroundwaterAbstraction" is not defined in the "groundwaterOptions" of the configuration file. This run assumes "0.020 m/day" for this option. 2021-09-03 06:46:15,151 currTimeStep INFO number of time steps: 365 2021-09-03 06:46:15,152 bmiPcrglobwb INFO Shape of maps is (108, 126) 2021-09-03 06:46:15,152 bmiPcrglobwb INFO PCRGlobWB: initialize_model, source dir is not used 2021-09-03 06:46:16,221 meteo INFO Precipitation forcing will be downscaled to the cloneMap resolution. 2021-09-03 06:46:16,221 meteo INFO Temperature forcing will be downscaled to the cloneMap resolution. 2021-09-03 06:46:18,752 landSurface INFO Irrigation is included/considered in this run. 2021-09-03 06:46:18,752 landSurface INFO Domestic water demand is included in the calculation. 2021-09-03 06:46:18,752 landSurface INFO Industry water demand is included in the calculation. 2021-09-03 06:46:18,752 landSurface INFO Livestock water demand is included in the calculation. 2021-09-03 06:46:18,752 landSurface INFO Using the dynamicIrrigationArea option. Extent of irrigation areas is based on the file provided in the 'historicalIrrigationArea'. 2021-09-03 06:46:19,171 landSurface INFO Monthly desalination water is included. 2021-09-03 06:46:21,684 landSurface INFO Using/incorporating the predefined surface water source of Siebert et al. (2010) for satisfying irrigation and livestock demand. 2021-09-03 06:46:23,450 landSurface INFO Using/incorporating the predefined surface water source of McDonald et al. (2014) for satisfying domestic and industrial demand. 2021-09-03 06:46:34,533 landSurface INFO Using the default set of soil and topo parameters as defined in the landSurfaceOptions of the ini/configuration file. 2021-09-03 06:46:34,534 landSurface INFO Using the default set of soil and topo parameters as defined in the landSurfaceOptions of the ini/configuration file. 2021-09-03 06:46:34,534 landSurface INFO Using the default set of soil and topo parameters as defined in the landSurfaceOptions of the ini/configuration file. 2021-09-03 06:46:34,534 landSurface INFO Using the default set of soil and topo parameters as defined in the landSurfaceOptions of the ini/configuration file. 2021-09-03 06:46:45,784 landCover WARNING As the "Original" interceptionModuleType is used, the "minInterceptCap" value is ignored. The interception scope is only "canopy". 2021-09-03 06:46:49,386 landCover WARNING This run assumes zero canopy interception capacity for this run, UNLESS minInterceptCap (minimum interception capacity) is bigger than zero. 2021-09-03 06:46:49,386 landCover WARNING As the "Original" interceptionModuleType is used, the "minInterceptCap" value is ignored. The interception scope is only "canopy". 2021-09-03 06:46:53,477 landCover WARNING This run assumes zero canopy interception capacity for this run, UNLESS minInterceptCap (minimum interception capacity) is bigger than zero. 2021-09-03 06:46:53,549 landSurface INFO Determining fraction of total irrigated areas over each cell 2021-09-03 06:47:22,840 groundwater INFO Limit for annual regional groundwater abstraction is used. 2021-09-03 06:47:25,002 groundwater INFO Fossil groundwater abstractions are allowed with LIMIT. 2021-09-03 06:47:25,002 groundwater INFO Estimating fossil groundwater capacities based on aquifer thicknesses and specific yield. 2021-09-03 06:47:33,107 groundwater INFO Using a pre-defined initial condition for fossil groundwater storage. 2021-09-03 06:47:33,880 groundwater INFO The pre-defined initial condition for fossil groundwater is limited by fossilWaterCap (full capacity). 2021-09-03 06:47:51,032 reporting INFO Creating the netcdf file for daily reporting for variable gwRecharge. 2021-09-03 06:47:51,052 reporting INFO Creating the netcdf file for daily reporting for variable storGroundwater. 2021-09-03 06:47:51,054 reporting INFO Creating the netcdf file for daily reporting for variable totalGroundwaterAbstraction. 2021-09-03 06:47:51,055 reporting INFO Creating the netcdf file for daily reporting for variable discharge. 2021-09-03 06:47:51,057 reporting INFO Creating the netcdf file for daily reporting for variable storLowTotal. 2021-09-03 06:47:51,058 reporting INFO Creating the netcdf file for daily reporting for variable precipitation_at_irrigation. 2021-09-03 06:47:51,060 reporting INFO Creating the netcdf file for daily reporting for variable surfaceWaterStorage. 2021-09-03 06:47:51,061 reporting INFO Creating the netcdf file for daily reporting for variable temperature. 2021-09-03 06:47:51,063 reporting INFO Creating the netcdf file for daily reporting for variable domesticWaterWithdrawal. 2021-09-03 06:47:51,064 reporting INFO Creating the netcdf file for daily reporting for variable interceptStor. 2021-09-03 06:47:51,066 reporting INFO Creating the netcdf file for daily reporting for variable nonIrrGrossDemand. 2021-09-03 06:47:51,067 reporting INFO Creating the netcdf file for daily reporting for variable irrPaddyWaterWithdrawal. 2021-09-03 06:47:51,069 reporting INFO Creating the netcdf file for daily reporting for variable totalEvaporation. 2021-09-03 06:47:51,070 reporting INFO Creating the netcdf file for daily reporting for variable channelStorage. 2021-09-03 06:47:51,072 reporting INFO Creating the netcdf file for daily reporting for variable storUppTotal. 2021-09-03 06:47:51,073 reporting INFO Creating the netcdf file for daily reporting for variable nonIrrWaterConsumption. 2021-09-03 06:47:51,075 reporting INFO Creating the netcdf file for daily reporting for variable surfaceWaterAbstraction. 2021-09-03 06:47:51,077 reporting INFO Creating the netcdf file for daily reporting for variable surfaceWaterLevel. 2021-09-03 06:47:51,078 reporting INFO Creating the netcdf file for daily reporting for variable evaporation_from_irrigation. 2021-09-03 06:47:51,080 reporting INFO Creating the netcdf file for daily reporting for variable snowFreeWater. 2021-09-03 06:47:51,081 reporting INFO Creating the netcdf file for daily reporting for variable totalWaterStorageThickness. 2021-09-03 06:47:51,083 reporting INFO Creating the netcdf file for daily reporting for variable precipitation. 2021-09-03 06:47:51,084 reporting INFO Creating the netcdf file for daily reporting for variable snowCoverSWE. 2021-09-03 06:47:51,086 reporting INFO Creating the netcdf file for daily reporting for variable totalRunoff. 2021-09-03 06:47:51,087 reporting INFO Creating the netcdf file for daily reporting for variable dynamicFracWat. 2021-09-03 06:47:51,089 reporting INFO Creating the netcdf file for daily reporting for variable industryWaterWithdrawal. 2021-09-03 06:47:51,090 reporting INFO Creating the netcdf file for daily reporting for variable topWaterLayer. 2021-09-03 06:47:51,096 bmiPcrglobwb INFO Shape of maps is (108, 126) 2021-09-03 06:47:51,096 bmiPcrglobwb INFO PCRGlobWB Initialized 2021-09-03 06:47:51,719 pcrglobwb INFO Reading forcings for time 2002-01-01 2021-09-03 06:47:52,851 pcrglobwb INFO Updating model for time 2002-01-01 2021-09-03 06:47:53,937 landSurface INFO Transfering states for the variable interceptStor 2021-09-03 06:47:53,942 landSurface INFO Successful in transfering states (after change in land cover fractions) ... Min -0.000000 Max 0.000000 Mean 0.000000 2021-09-03 06:47:53,943 landSurface INFO Transfering states for the variable snowCoverSWE 2021-09-03 06:47:53,947 landSurface INFO Successful in transfering states (after change in land cover fractions) ... Min -0.000000 Max 0.000000 Mean 0.000000 2021-09-03 06:47:53,947 landSurface INFO Transfering states for the variable snowFreeWater 2021-09-03 06:47:53,951 landSurface INFO Successful in transfering states (after change in land cover fractions) ... Min -0.000000 Max 0.000000 Mean 0.000000 2021-09-03 06:47:53,951 landSurface INFO Transfering states for the variable topWaterLayer 2021-09-03 06:47:53,956 landSurface INFO Successful in transfering states (after change in land cover fractions) ... Min 0.000000 Max 0.000000 Mean 0.000000 2021-09-03 06:47:53,956 landSurface INFO Transfering states for the variable storUpp 2021-09-03 06:47:53,960 landSurface INFO Successful in transfering states (after change in land cover fractions) ... Min -0.000000 Max 0.000000 Mean 0.000000 2021-09-03 06:47:53,960 landSurface INFO Transfering states for the variable storLow 2021-09-03 06:47:53,965 landSurface INFO Successful in transfering states (after change in land cover fractions) ... Min -0.000000 Max 0.000000 Mean 0.000000 2021-09-03 06:50:29,944 landSurface INFO Updating land cover: forest 2021-09-03 06:50:30,099 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalTall/cropCoefficientForest.nc ; variable: kc !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:31,333 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalTall/interceptCapInputForest.nc ; variable: interceptCapInput !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:31,875 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalTall/coverFractionInputForest.nc ; variable: coverFractionInput !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:32,352 landSurface INFO Updating land cover: grassland 2021-09-03 06:50:32,509 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalShort/cropCoefficientGrassland.nc ; variable: kc !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:32,981 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalShort/interceptCapInputGrassland.nc ; variable: interceptCapInput !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:33,751 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalShort/coverFractionInputGrassland.nc ; variable: coverFractionInput !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:34,281 landSurface INFO Updating land cover: irrPaddy 2021-09-03 06:50:34,414 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_30min/landSurface/landCover/irrPaddy/Global_CropCoefficientKc-IrrPaddy_30min.nc ; variable: kc !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:34,615 landSurface INFO Updating land cover: irrNonPaddy 2021-09-03 06:50:34,786 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_30min/landSurface/landCover/irrNonPaddy/Global_CropCoefficientKc-IrrNonPaddy_30min.nc ; variable: kc !!!!!! The date 2002-01-01 is NOT available. The date 2000-1-1 is used. 2021-09-03 06:50:35,255 groundwater INFO Updating groundwater 2021-09-03 06:50:35,258 routing INFO routing in progress 2021-09-03 06:50:36,686 reporting INFO reporting for time 2002-01-01 2021-09-03 06:50:36,750 bmiPcrglobwb INFO getting value for var discharge 2021-09-03 06:50:36,755 pcrglobwb INFO Reading forcings for time 2002-01-02 2021-09-03 06:50:36,989 pcrglobwb INFO Updating model for time 2002-01-02 2021-09-03 06:50:37,000 landSurface INFO Updating land cover: forest 2021-09-03 06:50:37,001 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalTall/cropCoefficientForest.nc ; variable: kc !!!!!! The date 2002-01-02 is NOT available. The date 2000-1-2 is used .... 2021-09-03 07:10:02,927 landSurface INFO Updating land cover: irrPaddy 2021-09-03 07:10:02,929 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_30min/landSurface/landCover/irrPaddy/Global_CropCoefficientKc-IrrPaddy_30min.nc ; variable: kc !!!!!! The date 2002-07-20 is NOT available. The date 2000-7-20 is used. 2021-09-03 07:10:03,062 landSurface INFO Updating land cover: irrNonPaddy 2021-09-03 07:10:03,063 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_30min/landSurface/landCover/irrNonPaddy/Global_CropCoefficientKc-IrrNonPaddy_30min.nc ; variable: kc !!!!!! The date 2002-07-20 is NOT available. The date 2000-7-20 is used. 2021-09-03 07:10:03,215 groundwater INFO Updating groundwater 2021-09-03 07:10:03,219 routing INFO routing in progress 2021-09-03 07:10:03,268 reporting INFO reporting for time 2002-07-20 2021-09-03 07:10:03,333 bmiPcrglobwb INFO getting value for var discharge 2021-09-03 07:10:03,336 pcrglobwb INFO Reading forcings for time 2002-07-21 2021-09-03 07:10:03,599 pcrglobwb INFO Updating model for time 2002-07-21 2021-09-03 07:10:03,609 landSurface INFO Updating land cover: forest 2021-09-03 07:10:03,610 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalTall/cropCoefficientForest.nc ; variable: kc !!!!!! The date 2002-07-21 is NOT available. The date 2000-7-21 is used. 2021-09-03 07:10:03,822 virtualOS WARNING WARNING related to the netcdf file: /mnt/data/parameter-sets/pcrglobwb_global/global_05min/landSurface/landCover/naturalTall/interceptCapInputForest.nc ; variable: interceptCapInput !!!!!! The date 2002-07-21 is NOT available. The date 2000-7-21 is used. 2021-09-03 07:10:12,595 grpc._server ERROR Exception calling application: Input/output error Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line 435, in _call_behavior response_or_iterator = behavior(argument, context) File "/usr/local/lib/python2.7/dist-packages/grpc4bmi/bmi_grpc_server.py", line 31, in update self.bmi_model_.update() File "/opt/PCR-GLOBWB_model/model/bmiPcrglobwb.py", line 214, in update self.model.update(report_water_balance=True) File "/opt/PCR-GLOBWB_model/model/pcrglobwb.py", line 449, in update self.landSurface.update(self.meteo, self.groundwater, self.routing, self._modelTime) File "/opt/PCR-GLOBWB_model/model/landSurface.py", line 1343, in update self.groundwater_pumping_region_ids,self.regionalAnnualGroundwaterAbstractionLimit) File "/opt/PCR-GLOBWB_model/model/landCover.py", line 868, in updateLC self.interceptionUpdate(meteo, currTimeStep) File "/opt/PCR-GLOBWB_model/model/landCover.py", line 1043, in interceptionUpdate cloneMapFileName = self.cloneMap), 0.0) File "/opt/PCR-GLOBWB_model/model/virtualOS.py", line 365, in netcdf2PCRobjClone cropData = f.variables[varName][int(idx),:,:] # still original data File "netCDF4/_netCDF4.pyx", line 4408, in netCDF4._netCDF4.Variable.__getitem__ File "netCDF4/_netCDF4.pyx", line 5352, in netCDF4._netCDF4.Variable._get File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success RuntimeError: Input/output error ```

[sverhoeven]

Full disk is likely culpred

df -h /data/volume_2/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   50G   20K 100% /data/volume_2

The global 5min pcrglob parameter set does not fit the 50Gb cache disk

du -hs  /data/volume_2/vfs/dcache/parameter-sets/pcrglobwb_global/*
49G     /data/volume_2/vfs/dcache/parameter-sets/pcrglobwb_global/global_05min
544M    /data/volume_2/vfs/dcache/parameter-sets/pcrglobwb_global/global_30min
24K     /data/volume_2/vfs/dcache/parameter-sets/pcrglobwb_global/rhine_05min_era5.ini

and real set is

du -hs /mnt/data/parameter-sets/pcrglobwb_global/global_05min/
231G    /mnt/data/parameter-sets/pcrglobwb_global/global_05min/

[sverhoeven]

I set the rclone vfs-cache-max-size from 500Gb to 48Gb and stopped rclone mount, cleared cache, started rclone mount.

@Peter9192 should I create a new VM with a 500Gb cache disk?

[sverhoeven]

Restarted notebook and got past 2002-07-29. Cache dir became full and then some file got deleted and it filled again

root@localhost:~# df -h /data/volume_2/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   50G  908K 100% /data/volume_2
root@localhost:~# df -h /data/volume_2/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   49G  1.9G  97% /data/volume_2
root@localhost:~# df -h /data/volume_2/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   49G  1.8G  97% /data/volume_2

[sverhoeven]

A fix for running nb is in https://github.com/eWaterCycle/infra/pull/96

[sverhoeven]

Notebook runs but pixel is incorrect as hydrograph shows flat line around 0, added custom model location in https://github.com/eWaterCycle/TerriaMap/blob/ewatercycle-v8/wwwroot/init/ewatercycle.json

[sverhoeven]

Notebook now runs ok with nse of -0.61