search

eWaterCycle / infra

Instructions for system administrators to deploy the eWaterCycle platform
Apache License 2.0
0 stars 0 forks source link

Wflow merrimack catalog item not running #94

Closed sverhoeven closed 3 years ago

sverhoeven commented 3 years ago

Known as Wflow / Merrimack / 1991-01-01 - 2001-12-31 / ERA5 in explorer. Which has the following props:

{
                  "model": {
                    "name": "Wflow",
                    "version": "2020.1.1"
                  },
                  "parameter_set": "wflow_merrimack_calibrated",
                  "variable": "SurfaceRunoff",
                  "observation": {
                    "station_id": "1160235"
                  }
}

First it failed because it could not find the nc file.

So generated forcing with 

import ewatercycle.forcing
import ewatercycle.parameter_sets

parameter_set = ewatercycle.parameter_sets.get_parameter_set("wflow_merrimack_calibrated")

forcing = ewatercycle.forcing.generate(
    target_model="wflow",
    dataset="ERA5",
    start_time="1990-01-01T00:00:00Z",
    end_time="1990-12-31T00:00:00Z",
    shape="/mnt/data/climate-data/aux/Lorentz_Basin_Shapefiles/Merrimack/Merrimack.shp",
    model_specific_options={
        "dem_file": str(parameter_set.directory / "staticmaps/wflow_dem.map"),
    },
)

forcing.directory
# /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/esmvaltool_output/recipe_wflow_20210902_122049/work/wflow_daily/script

Now it fails with Exception calling application: 'pcraster._pcraster.Field' object is not iterable" grpc error during initialize().

`model.bmi.logs()` ``` 2021-09-02 12:57:25,585 - wflow_ewatercycle.ini - pcrut - DEBUG - File logging to /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/run_default/wflow.log 2021-09-02 12:57:25,585 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - wflow_ewatercycle.ini 2020.1.2.dev8+gfdb404c5 Case: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718 Runid: run_default 2021-09-02 12:57:25,585 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initialise framework version: 2020.1.2.dev8+gfdb404c5 2021-09-02 12:57:25,586 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - Ini file made for wflow_sbm but found wflow.wflow_sbm in code. 2021-09-02 12:57:25,586 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Found following input variables to get from netcdf file: ['pr', 'pet', 'tas', 'IF', 'pr', 'tas', 'pet', 'IF'] 2021-09-02 12:57:25,659 - wflow_ewatercycle.ini - wf_netcdfio - INFO - Reading input from netCDF file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/wflow_ERA5_Merrimack_1990_1990.nc 2021-09-02 12:57:25,668 - wflow_ewatercycle.ini - wf_netcdfio - WARNING - Variable IF not found in netcdf file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/wflow_ERA5_Merrimack_1990_1990.nc 2021-09-02 12:57:25,668 - wflow_ewatercycle.ini - wf_netcdfio - WARNING - Variable IF not found in netcdf file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/wflow_ERA5_Merrimack_1990_1990.nc 2021-09-02 12:57:25,670 - wflow_ewatercycle.ini - wf_netcdfio - INFO - Setting up netcdf output: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/run_default/outmaps.nc 2021-09-02 12:57:25,671 - wflow_ewatercycle.ini - wf_netcdfio - INFO - Setting up dimensions and attributes. Steps: 365 lat: 242 lon: 154 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: Sl (['staticmaps/Sl.map', 'staticmap', '0.1', '1']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: Kext (['staticmaps/Kext.map', 'staticmap', '0.6', '1']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: Swood (['staticmaps/Swood.map', 'staticmap', '0.5', '1']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: LAI (['staticmaps/clim/LAI', 'monthlyclim', '1.0', '1']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ReserVoirSimpleLocs (['staticmaps/wflow_reservoirlocs.map', 'staticmap', '0.0', '0']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ResTargetFullFrac (['intbl/ResTargetFullFrac.tbl', 'tbl', '0.8', '0', 'staticmaps/wflow_reservoirlocs.map']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ResTargetMinFrac (['intbl/ResTargetMinFrac.tbl', 'tbl', '0.4', '0', 'staticmaps/wflow_reservoirlocs.map']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ResMaxVolume (['intbl/ResMaxVolume.tbl', 'tbl', '0.0', '0', 'staticmaps/wflow_reservoirlocs.map']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ResMaxRelease (['intbl/ResMaxRelease.tbl', 'tbl', '1.0', '0', 'staticmaps/wflow_reservoirlocs.map']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ResDemand (['intbl/ResDemand.tbl', 'tbl', '1.0', '0', 'staticmaps/wflow_reservoirlocs.map']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ReservoirSimpleAreas (['staticmaps/wflow_reservoirareas.map', 'staticmap', '0.0', '0']) 2021-09-02 12:57:25,743 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Creating parameter specification for par: ResSimpleArea (['intbl/ResSimpleArea.tbl', 'tbl', '0', '0', 'staticmaps/wflow_reservoirlocs.map']) 2021-09-02 12:57:25,745 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding csv output at /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/grdc_station.map function: average 2021-09-02 12:57:25,745 - wflow_ewatercycle.ini - wflow_sbm - INFO - running for 365 timesteps 2021-09-02 12:57:25,745 - wflow_ewatercycle.ini - wflow_sbm - INFO - Using sub timestep for kinematic wave (iterate) 2021-09-02 12:57:25,745 - wflow_ewatercycle.ini - wflow_sbm - INFO - Using a fixed timestep (seconds) for kinematic wave river flow: 900 2021-09-02 12:57:25,746 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_riverlength_fact.map) not present, returning 1.0 2021-09-02 12:57:25,747 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_inflow.map) not present, returning 0.0 2021-09-02 12:57:25,748 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718\staticmaps/wflow_tempcor.map) not present, returning 0.0 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wflow_sbm - INFO - Linking parameters to landuse, catchment and soil... 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding Precipitation to model. 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/pr.map) not present, returning 0.0 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding PotenEvap to model. 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/pet.map) not present, returning 0.0 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding Temperature to model. 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/tas.map) not present, returning 10.0 2021-09-02 12:57:25,749 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding Inflow to model. 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding IrrigationAreas to model. 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_irrigationareas.map) not present, returning 0.0 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding IrrigationSurfaceIntakes to model. 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_irrisurfaceintake.map) not present, returning 0.0 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding IrrigationPaddyAreas to model. 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_irrigationpaddyareas.map) not present, returning 0.0 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding IrrigationSurfaceReturn to model. 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_irrisurfacereturns.map) not present, returning 0.0 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding h_max to model. 2021-09-02 12:57:25,750 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_hmax.map) not present, returning 0.0 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding h_min to model. 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_hmin.map) not present, returning 0.0 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding h_p to model. 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Static input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/staticmaps/wflow_hp.map) not present, returning 0.0 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding Sl to model. 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding Kext to model. 2021-09-02 12:57:25,751 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding Swood to model. 2021-09-02 12:57:25,752 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding LAI to model. 2021-09-02 12:57:25,752 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding ReserVoirSimpleLocs to model. 2021-09-02 12:57:25,752 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initial: Adding ResTargetFullFrac to model. 2021-09-02 12:57:25,753 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initial: Adding ResTargetMinFrac to model. 2021-09-02 12:57:25,753 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initial: Adding ResMaxVolume to model. 2021-09-02 12:57:25,754 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initial: Adding ResMaxRelease to model. 2021-09-02 12:57:25,755 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initial: Adding ResDemand to model. 2021-09-02 12:57:25,756 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Adding ReservoirSimpleAreas to model. 2021-09-02 12:57:25,756 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initial: Adding ResSimpleArea to model. 2021-09-02 12:57:25,757 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/RootingDepth.map 2021-09-02 12:57:25,757 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/AirEntryPressure.tbl) returning default value: 10.0 2021-09-02 12:57:25,760 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/rootdistpar.tbl 2021-09-02 12:57:25,762 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/InfiltCapSoil.tbl 2021-09-02 12:57:25,763 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/CapScale.tbl) returning default value: 100.0 2021-09-02 12:57:25,766 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/InfiltCapPath.tbl 2021-09-02 12:57:25,768 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/MaxLeakage.tbl 2021-09-02 12:57:25,769 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/MaxPercolation.tbl) returning default value: 0.0 2021-09-02 12:57:25,770 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/PathFrac.map 2021-09-02 12:57:25,771 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/SoilThickness.map 2021-09-02 12:57:25,771 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/thetaR.map 2021-09-02 12:57:25,771 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/thetaS.map 2021-09-02 12:57:25,771 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/SoilMinThickness.map 2021-09-02 12:57:25,772 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/KsatVer.map 2021-09-02 12:57:25,772 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/MporeFrac.tbl) returning default value: 0.0 2021-09-02 12:57:25,774 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/KsatHorFrac.tbl 2021-09-02 12:57:25,775 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/M.map 2021-09-02 12:57:25,776 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/N.map 2021-09-02 12:57:25,777 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/WaterFrac.map 2021-09-02 12:57:25,778 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/et_reftopot.tbl) returning default value: 1.0 2021-09-02 12:57:25,780 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/TTI.tbl 2021-09-02 12:57:25,782 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/TT.tbl 2021-09-02 12:57:25,785 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/TTM.tbl 2021-09-02 12:57:25,787 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/Cfmax.tbl 2021-09-02 12:57:25,790 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Creating map from table: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/WHC.tbl 2021-09-02 12:57:25,791 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/w_soil.tbl) returning default value: 0.1125 2021-09-02 12:57:25,802 - wflow_ewatercycle.ini - wflow_sbm - INFO - A total of 2 simple reservoirs and 0 lakes found. 2021-09-02 12:57:25,812 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/KsatVerFrac.tbl) returning default value: 1.0 2021-09-02 12:57:25,813 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/c_0.map 2021-09-02 12:57:25,813 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/KsatVerFrac.tbl) returning default value: 1.0 2021-09-02 12:57:25,814 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/c_1.map 2021-09-02 12:57:25,815 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/KsatVerFrac.tbl) returning default value: 1.0 2021-09-02 12:57:25,816 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/c_2.map 2021-09-02 12:57:25,816 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - tbl file not found (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/KsatVerFrac.tbl) returning default value: 1.0 2021-09-02 12:57:25,817 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - reading map parameter file: /home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/intbl/../staticmaps/c_3.map 2021-09-02 12:57:25,818 - wflow_ewatercycle.ini - wflow_sbm - INFO - Initializing of model variables.. 2021-09-02 12:57:28,258 - wflow_ewatercycle.ini - wflow_sbm - INFO - Saving summary maps... 2021-09-02 12:57:28,258 - wflow_ewatercycle.ini - wflow_sbm - INFO - End of initial section 2021-09-02 12:57:28,258 - wflow_ewatercycle.ini - wflow_sbm - INFO - Setting initial conditions from state files 2021-09-02 12:57:28,258 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/RiverRunoff.map) not present, returning 0.0 2021-09-02 12:57:28,258 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/WaterLevelR.map) not present, returning 0.0 2021-09-02 12:57:28,258 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/LandRunoff.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/WaterLevelL.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/SatWaterDepth.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/Snow.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/TSoil.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/UStoreLayerDepth.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/SnowWater.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/CanopyStorage.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/SubsurfaceFlow.map) not present, returning 0.0 2021-09-02 12:57:28,259 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/sverhoeven/aeb9b473-19a7-486f-8573-3a71024b3e36/ewatercycle_output/wflow_20210902_125718/instate/ReservoirVolume.map) not present, returning 0.0 ```
Peter9192 commented 3 years ago

Shouldn't it be "RiverRunoff" in the model version we're using?

sverhoeven commented 3 years ago

Trying to get better error without grpc4bmi

cd <cfg_dir>
singularity exec --contain --pwd $PWD --bind $PWD:$PWD /mnt/data/singularity-images/ewatercycle-wflow-grpc4bmi_2020.1.1.sif python
from wflow.wflow_bmi import wflowbmi_csdms
model = wflowbmi_csdms()
model.initialize('/home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211/wflow_ewatercycle.ini')
2021-09-08 10:04:31,875 - wflow_ewatercycle.ini - pcrut - DEBUG - File logging to /home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211/run_default/wflow.log
2021-09-08 10:04:31,875 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - wflow_ewatercycle.ini 2020.1.2.dev8+gfdb404c5 Case: /home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211 Runid: run_default
2021-09-08 10:04:31,875 - wflow_ewatercycle.ini - wf_DynamicFramework - INFO - Initialise framework version: 2020.1.2.dev8+gfdb404c5
2021-09-08 10:04:31,876 - wflow_ewatercycle.ini - wf_DynamicFramework - WARNING - Ini file made for wflow_sbm but found wflow.wflow_sbm in code.
2021-09-08 10:04:31,877 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - Found following input variables to get from netcdf file: ['pr', 'pet', 'tas', 'IF', 'pr', 'tas', 'pet', 'IF']
2021-09-08 10:04:31,911 - wflow_ewatercycle.ini - wf_netcdfio - INFO - Reading input from netCDF file: /home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211/wflow_ERA5_Merrimack_1990_2001.nc
2021-09-08 10:04:31,957 - wflow_ewatercycle.ini - wf_netcdfio - WARNING - Variable IF not found in netcdf file: /home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211/wflow_ERA5_Merrimack_1990_2001.nc
2021-09-08 10:04:31,957 - wflow_ewatercycle.ini - wf_netcdfio - WARNING - Variable IF not found in netcdf file: /home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211/wflow_ERA5_Merrimack_1990_2001.nc
...
2021-09-08 10:04:33,966 - wflow_ewatercycle.ini - wf_DynamicFramework - DEBUG - State input data (/home/vagrant/fef0db06-2ffd-4792-ab2b-60e31df98766/ewatercycle_output/wflow_20210908_095211/instate/ReservoirVolume.map)  not present, returning 0.0
2021-09-08 10:04:34,009 - wflow_ewatercycle.ini - wf_DynamicFramework - ERROR -   File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/wflow-2020.1.2.dev8+gfdb404c5-py3.7.egg/wflow/wflow_bmi.py", line 792, in initialize
    self.initialize_model()
  File "/opt/conda/lib/python3.7/site-packages/wflow-2020.1.2.dev8+gfdb404c5-py3.7.egg/wflow/wflow_bmi.py", line 676, in initialize_model
    self.dynModel._runResume()
  File "/opt/conda/lib/python3.7/site-packages/pcraster/framework/frameworkBase.py", line 402, in _runResume
    self._userModel().resume()
  File "/opt/conda/lib/python3.7/site-packages/wflow-2020.1.2.dev8+gfdb404c5-py3.7.egg/wflow/wflow_sbm.py", line 2142, in resume
    + self.CanopyStorage
  File "/opt/conda/lib/python3.7/site-packages/wflow-2020.1.2.dev8+gfdb404c5-py3.7.egg/wflow/wflow_lib.py", line 75, in sum_list_cover
    for map in list_of_maps:

2021-09-08 10:04:34,010 - wflow_ewatercycle.ini - wf_DynamicFramework - ERROR - <class 'TypeError'>: 'pcraster._pcraster.Field' object is not iterable
sverhoeven commented 3 years ago

Following stacktrace at https://github.com/eWaterCycle/wflow/blob/docker-fixes-2020.1.1/wflow/wflow_sbm.py#L2141 I applied workaround from other wflow nb:

cd staticmaps
cp wflow_dem.map wflow_demmax.map
cp wflow_dem.map wflow_demmin.map
cd ..

The parameter set has instate_era5/ and instate_era-interim/ dir and no instate/ dir.

Doing ln -s instate_era5 instate make initialize() work.

The station id was incorrect, after fixing got hydrograph.

sverhoeven commented 3 years ago

So to solve need have parameter sets with instate/ dir when reinit = 0. Can solve by doing:

  1. Make copy of each parameter set
    1. Add ERA5 to parameter set name
    2. Move instate_era5 to instate and remove instate_era-interim
  2. Make copy of each parameter set
    1. Add ERA-Interim to parameter set name
    2. Move instate_era-interim to instate and remove instate_era5

@Peter9192 any objection if I do some copying/renaming/moving in /mnt/data/pararmeter-sets/wflow*calibrated dirs?

sverhoeven commented 3 years ago

Also saw that I generated forcings which falls inside calibration window, will regenerate for 2002.

Peter9192 commented 3 years ago

any objection if I do some copying/renaming/moving in /mnt/data/pararmeter-sets/wflow*calibrated dirs?

Fine with me. I think there's a tarball of the original stored somewhere as well. I vaguely recall we did some copying and renaming in the original wflow comparison notebook as well, but not sure if that's the same files..

sverhoeven commented 3 years ago
Generating forcings with ```python import shutil import ewatercycle.forcing import ewatercycle.parameter_sets CATCHMENTS = ["Meuse", "Rhine", "Doring", "Great_Kei", "Merrimack", "Savannah"] def gen(catchment): catchment2 = catchment.lower().replace('_','-') parameter_set_name = f"wflow_{catchment2}_calibrated" print(parameter_set_name) parameter_set = ewatercycle.parameter_sets.get_parameter_set(parameter_set_name) shape = f"/mnt/data/climate-data/aux/Lorentz_Basin_Shapefiles/{catchment}/{catchment}.shp" dataset = 'ERA5' forcing = ewatercycle.forcing.generate( target_model="wflow", dataset=dataset, start_time="2002-01-01T00:00:00Z", end_time="2002-12-31T00:00:00Z", shape=shape, model_specific_options={ "dem_file": str(parameter_set.directory / "staticmaps/wflow_dem.map"), }, ) shutil.copytree(forcing.shape.parent, forcing.directory / catchment) target = f'/home/vagrant/forcing/wflow_{dataset}_2002-2002_{catchment2}' forcing.shape = forcing.directory / catchment / f"{catchment}.shp" forcing.save() shutil.copytree(forcing.directory, target) return target, forcing for c in CATCHMENTS: print(gen(c)) ```
parameter reshuffle commands ``` mkdir parameter-sets cd parameter-sets unzip /mnt/data/parameter-sets/wflow_sbm_models_calibrated.zip mv doring wflow_Doring_ERA5-calibrated mv great_kei wflow_Great-Kei_ERA5-calibrated mv merrimack wflow_Merrimack_ERA5-calibrated mv meuse wflow_Meuse_ERA5-calibrated mv rhine wflow_Rhine_ERA5-calibrated mv savannah wflow_Savannah_ERA5-calibrated cp -r wflow_Doring_ERA5-calibrated wflow_Doring_ERA-Interim-calibrated cp -r wflow_Great-Kei_ERA5-calibrated wflow_Great-Kei_ERA-Interim-calibrated cp -r wflow_Merrimack_ERA5-calibrated wflow_Merrimack_ERA-Interim-calibrated cp -r wflow_Meuse_ERA5-calibrated wflow_Meuse_ERA-Interim-calibrated cp -r wflow_Rhine_ERA5-calibrated wflow_Rhine_ERA-Interim-calibrated cp -r wflow_Savannah_ERA5-calibrated wflow_Savannah_ERA-Interim-calibrated for f in $(ls -1 |grep ERA5); do rmdir $f/instate mv $f/instate_era5 $f/instate rm -r $f/instate_era-interim done for f in $(ls -1 |grep Interim); do rmdir $f/instate mv $f/instate_era-interim $f/instate rm -r $f/instate_era5 done for f in $(ls -1); do echo 'This parameter set was extracted from wflow_sbm_models_calibrated.zip file, renamed to standardized naming scheme and instate was renamed based on forcing dataset.' > $f/README.md done # drop sets without instate as they can not be used without swapping reinit rm -r wflow_Doring_ERA-Interim-calibrated rm -r wflow_Savannah_ERA5-calibrated cd .. rclone copy -P parameter-sets dcache:parameter-sets rclone purge dcache:parameter-sets/wflow_doring_calibrated rclone purge dcache:parameter-sets/wflow_great-kei_calibrated rclone purge dcache:parameter-sets/wflow_merrimack_calibrated rclone purge dcache:parameter-sets/wflow_meuse_calibrated rclone purge dcache:parameter-sets/wflow_rhine_calibrated rclone purge dcache:parameter-sets/wflow_savannah_calibrated ```
sverhoeven commented 3 years ago

Parameter sets and forcing for wflow era5 calibrated are now on dcache. Still to do is review/merge https://github.com/eWaterCycle/infra/pull/98, spin up VM and retry notebooks.

sverhoeven commented 3 years ago

Notebook now runs