search

Deltares / hydromt_wflow

Wflow plugin for HydroMT
https://deltares.github.io/hydromt_wflow/
GNU General Public License v3.0
20 stars 15 forks source link

river roughness mapping table not found #298

Closed shartgring closed 2 months ago

shartgring commented 2 months ago

HydroMT-Wflow version checks

Reproducible Example

The internal mapping table cannot always be found when working on a virtual machine. Sometimes a long path is created which results in hydromt_wflow not finding the csv.

https://github.com/Deltares/hydromt_wflow/blob/main/hydromt_wflow/data/parameters_data.yml

Example code:

2024-09-25 13:51:11,155 - build - log - DEBUG - Writing log messages to new file C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}\SVROOT\Users\cazemier\HydroMT_Vecht\NbS\FinalModel2016\Dry_newlai\hydromt.log.2024-09-25 13:51:11,155 - build - log - INFO - HydroMT version: 0.10.02024-09-25 13:51:11,156 - build - main - INFO - Building instance of wflow model at C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}\SVROOT\Users\cazemier\HydroMT_Vecht\NbS\FinalModel2016\Dry_newlai.2024-09-25 13:51:11,156 - build - main - INFO - User settings:2024-09-25 13:51:12,268 - build - data_catalog - INFO - Reading data catalog deltares_data latest2024-09-25 13:51:12,270 - build - data_catalog - INFO - Parsing data catalog from C:\Users\cazemier\.hydromt_data\deltares_data\v0.6.0\data_catalog.yml2024-09-25 13:51:12,447 - build - data_catalog - INFO - Parsing data catalog from C:/Users/cazemier/HydroMT_Vecht/DataCatalog.yml2024-09-25 13:51:12,474 - build - model_api - INFO - Initializing wflow model from hydromt_wflow (v0.6.2.dev0).2024-09-25 13:51:12,474 - build - data_catalog - INFO - Parsing data catalog from C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}\SVROOT\Users\cazemier\hydromt_wflow\hydromt_wflow\data\parameters_data.yml2024-09-25 13:51:12,488 - build - model_api - DEBUG - Setting model config options.2024-09-25 13:51:12,492 - build - model_api - DEBUG - Default config read from C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}\SVROOT\Users\cazemier\hydromt_wflow\hydromt_wflow\data\wflow\wflow_sbm.toml2024-09-25 13:51:12,493 - build - model_api - INFO - setup_basemaps.region: {'subbasin': [6.202598, 52.522593], 'strord': 6}2024-09-25 13:51:12,493 - build - model_api - INFO - setup_basemaps.hydrography_fn: merit_hydro2024-09-25 13:51:12,493 - build - model_api - INFO - setup_basemaps.basin_index_fn: merit_hydro_index2024-09-25 13:51:12,493 - build - model_api - INFO - setup_basemaps.res: 0.002922024-09-25 13:51:12,493 - build - model_api - INFO - setup_basemaps.upscale_method: ihu2024-09-25 13:51:12,493 - build - wflow - INFO - Preparing base hydrography basemaps.2024-09-25 13:51:12,677 - build - rasterdataset - INFO - Reading merit_hydro raster data from p:\wflow_global\hydromt\topography\merit_hydro\*.vrt2024-09-25 13:51:14,115 - build - basin_mask - DEBUG - Parsed region (kind=subbasin): {'strord': 6, 'xy': [6.202598, 52.522593]}2024-09-25 13:51:14,155 - build - basin_mask - DEBUG - Getting basin IDs at point locations.2024-09-25 13:51:26,987 - build - basin_mask - INFO - subbasin bbox: [6.2017, 51.9650, 7.4600, 52.8767]2024-09-25 13:51:27,212 - build - wflow - DEBUG - Adding basins vector to geoms.2024-09-25 13:51:27,214 - build - basemaps - DEBUG - Mask in dataset assumed to represent subbasins.2024-09-25 13:51:27,216 - build - basemaps - DEBUG - (Sub)basin at original resolution has 799606 cells.2024-09-25 13:51:27,422 - build - basemaps - DEBUG - Burn subbasin outlet in upstream area data.2024-09-25 13:51:27,724 - build - basemaps - INFO - Upscale flow direction data: 4x, ihu method.2024-09-25 13:51:56,664 - build - basemaps - DEBUG - Derive stream order.2024-09-25 13:51:57,085 - build - basemaps - DEBUG - Map shape: (273, 377); active cells: 49972.2024-09-25 13:51:57,085 - build - basemaps - DEBUG - Outlet coordinates (1/1): (6.20250,52.52167).2024-09-25 13:51:57,902 - build - wflow - DEBUG - Adding region vector to geoms.2024-09-25 13:51:57,917 - build - model_api - INFO - setup_rivers.hydrography_fn: merit_hydro2024-09-25 13:51:57,917 - build - model_api - INFO - setup_rivers.river_geom_fn: River_geom_Vecht2024-09-25 13:51:57,918 - build - model_api - INFO - setup_rivers.river_upa: 302024-09-25 13:51:57,918 - build - model_api - INFO - setup_rivers.rivdph_method: powlaw2024-09-25 13:51:57,920 - build - model_api - INFO - setup_rivers.slope_len: 20002024-09-25 13:51:57,920 - build - model_api - INFO - setup_rivers.min_rivlen_ratio: 0.02024-09-25 13:51:57,923 - build - model_api - INFO - setup_rivers.min_rivdph: 12024-09-25 13:51:57,925 - build - model_api - INFO - setup_rivers.min_rivwth: 202024-09-25 13:51:57,925 - build - model_api - INFO - setup_rivers.smooth_len: 50002024-09-25 13:51:57,928 - build - model_api - INFO - setup_rivers.rivman_mapping_fn: roughness_river_mapping_default2024-09-25 13:51:57,929 - build - model_api - INFO - setup_rivers.elevtn_map: wflow_dem2024-09-25 13:51:57,929 - build - model_api - INFO - setup_rivers.river_routing: local-inertial2024-09-25 13:51:57,929 - build - model_api - INFO - setup_rivers.connectivity: 82024-09-25 13:51:57,930 - build - wflow - INFO - Preparing river maps.2024-09-25 13:51:57,942 - build - rasterdataset - INFO - Reading merit_hydro raster data from p:\wflow_global\hydromt\topography\merit_hydro\*.vrt2024-09-25 13:51:59,219 - build - rasterdataset - DEBUG - Clip to [6.201, 51.967, 7.458, 52.877] (epsg:4326))2024-09-25 13:51:59,257 - build - river - DEBUG - Set river mask (min uparea: 30 km2) and prepare flow dirs.2024-09-25 13:52:00,241 - build - river - DEBUG - Derive river length.2024-09-25 13:52:08,964 - build - river - DEBUG - Derive river slope.2024-09-25 13:52:10,568 - build - river - DEBUG - Derive rivwth from hydrography dataset.2024-09-25 13:52:10,804 - build - main - ERROR - No such file found: C:\SnapVolumesTemp\MountPoints\{{45c63495-0000-0000-0000-100000000000}}\{{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}}\SVROOT\Users\cazemier\hydromt_wflow\hydromt_wflow\data\wflow\N_river_mapping.csvTraceback (most recent call last): File "C:\Users\cazemier\AppData\Local\miniforge3\envs\hydromt_wflow\Lib\site-packages\hydromt\cli\main.py", line 224, in build mod.build(region, opt=opt) File "C:\Users\cazemier\AppData\Local\miniforge3\envs\hydromt_wflow\Lib\site-packages\hydromt\models\model_api.py", line 246, in build self._run_log_method(method, **kwargs) File "C:\Users\cazemier\AppData\Local\miniforge3\envs\hydromt_wflow\Lib\site-packages\hydromt\models\model_api.py", line 188, in _run_log_method return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}\SVROOT\Users\cazemier\hydromt_wflow\hydromt_wflow\wflow.py", line 480, in setup_rivers df = self.data_catalog.get_dataframe(rivman_mapping_fn) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\cazemier\AppData\Local\miniforge3\envs\hydromt_wflow\Lib\site-packages\hydromt\data_catalog.py", line 1683, in get_dataframe obj = source.get_data( ^^^^^^^^^^^^^^^^ File "C:\Users\cazemier\AppData\Local\miniforge3\envs\hydromt_wflow\Lib\site-packages\hydromt\data_adapter\dataframe.py", line 224, in get_data fns = self._resolve_paths(variables=variables) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\cazemier\AppData\Local\miniforge3\envs\hydromt_wflow\Lib\site-packages\hydromt\data_adapter\data_adapter.py", line 407, in _resolve_paths raise FileNotFoundError(f"No such file found: {path}{postfix}")FileNotFoundError: No such file found: C:\SnapVolumesTemp\MountPoints\{{45c63495-0000-0000-0000-100000000000}}\{{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}}\SVROOT\Users\cazemier\hydromt_wflow\hydromt_wflow\data\wflow\N_river_mapping.csv

Current behaviour

mapping table is being searched for on C:\SnapVolumesTemp\MountPoints\{{45c63495-0000-0000-0000-100000000000}}\{{db1e65df-ac0c-4981-bb4b-9ce5f13a76f2}}\SVROOT\Users\cazemier\hydromt_wflow\hydromt_wflow\data\wflow\N_river_mapping.csv

Desired behaviour

mapping table should be searched for on a shorter path like C:\Users\cazemier\hydromt_wflow\hydromt_wflow\data\wflow\N_river_mapping.csv

Additional context

No response

shartgring commented 2 months ago

Might also affect other internal mapping tables (lulc)?

shartgring commented 2 months ago

The catalog parameters_data.yml is being read in the definition of the WflowModel where _CATALOGS = join(_DATADIR, "parameters_data.yml") and _DATADIR = DATADIR, where DATADIR comes from hydromt_wflow.utils: DATADIR = join(dirname(abspath(__file__)), "data")

Changing this may fix the path?

shartgring commented 2 months ago

The path is interpreted through the data_catalog of DATADIR as C:\\SnapVolumesTemp\\MountPoints\\{{45c63495-0000-0000-0000-100000000000}}\\{{4EB959A1-DAFA-48F1-96BB-E4B67D6DC4E4}}\\SVROOT\\git\\hydromt_wflow\\hydromt_wflow\\data\\wflow\\N_river_mapping.csv in my case, and reading it in pandas using read_csv() raises FileNotFoundError

shartgring commented 2 months ago 
from hydromt_wflow import WflowModel
model = WflowModel()
model.data_catalog.get_dataframe("roughness_river_mapping_default")

FileNotFoundError: No such file found: C:\SnapVolumesTemp\MountPoints\{{45c63495-0000-0000-0000-100000000000}}\{{4EB959A1-DAFA-48F1-96BB-E4B67D6DC4E4}}\SVROOT\git\hydromt_wflow\hydromt_wflow\data\wflow\N_river_mapping.csv

However, using single brackets does results in correct reading of the csv file. So the solution might need to be searched for in the HydroMT code https://github.com/Deltares/hydromt/blob/a707d44497ef493174b76fafde90e45085d78a6c/hydromt/data_adapter/data_adapter.py#L341

hboisgon commented 2 months ago

Yes this is something for core. I thought the double bracket reading {{}} was solved before but maybe something went wrong later? It's because we use {} for keywords such as variable, year, month. I'll open the issue on core, nothing we can do in hydromt_wflow itself I'm afraid.