search

ArcticSnow / TopoPyScale

TopoPyScale: a Python library to perform simplistic climate downscaling at the hillslope scale
https://topopyscale.readthedocs.io
MIT License
39 stars 9 forks source link

Error when getting to topo parameters extraction stage #55

Closed dvalters closed 7 months ago

dvalters commented 1 year ago

Hi, I'm currently reviewing the software for JOSS here, and have got through the install ok, but am running into problems when getting to the extraction of topographic metrics stage. I'm not entirely sure from the error output message where exactly it is falling over. I'm using the Norway ex1 example dataset.

2023-02-21 21:56:22,639 INFO Download rate 2.5M/s
[USER_DIRS]/TopoPyScale_examples/ex1_norway_finse/inputs/climate/PLEV_202107.nc complete
2023-02-21 22:09:21,200 INFO Request is completed
2023-02-21 22:09:21,200 INFO Downloading https://download-0008-clone.copernicus-climate.eu/cache-compute-0008/cache/data7/adaptor.mars.internal-1677017196.2052243-11103-8-173d15ea-9bf4-42a7-9dd6-1ff637b84745.nc to [USER_DIRS]/TopoPyScale_examples/ex1_norway_finse/inputs/climate/PLEV_202108.nc (2M)
2023-02-21 22:09:21,862 INFO Download rate 3.1M/s
[USER_DIRS]/TopoPyScale_examples/ex1_norway_finse/inputs/climate/PLEV_202108.nc complete

---> Extracting DEM parameters (slope, aspect, svf)
Computing slope and aspect ...
Computing svf ...
---> File [USER_DIRS]/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
---> Scaling data prior to clustering
Traceback (most recent call last):
  File "[USER_DIRS]/TopoPyScale_examples/ex1_norway_finse/pipeline.py", line 9, in <module>
    mp.extract_topo_param()
  File "[USER_DIRS]/TopoPyScale/TopoPyScale/topoclass.py", line 321, in extract_topo_param
    self.extract_topo_cluster_param()
  File "[USER_DIRS]/dev/TopoPyScale/TopoPyScale/topoclass.py", line 269, in extract_topo_cluster_param
    df_scaled, self.toposub.scaler = ts.scale_df(df_param,
  File "[USER_DIRS]/dev/TopoPyScale/TopoPyScale/topo_sub.py", line 57, in scale_df
    df_scaled = pd.DataFrame(scaler.fit_transform(df_param[feature_list].values),
  File "[USER_DIRS]/miniconda3/envs/downscaling/lib/python3.9/site-packages/sklearn/utils/_set_output.py", line 142, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "[USER_DIRS]/miniconda3/envs/downscaling/lib/python3.9/site-packages/sklearn/base.py", line 859, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "[USER_DIRS]/miniconda3/envs/downscaling/lib/python3.9/site-packages/sklearn/preprocessing/_data.py", line 824, in fit
    return self.partial_fit(X, y, sample_weight)
  File "[USER_DIRS]/miniconda3/envs/downscaling/lib/python3.9/site-packages/sklearn/preprocessing/_data.py", line 861, in partial_fit
    X = self._validate_data(
  File "[USER_DIRS]/miniconda3/envs/downscaling/lib/python3.9/site-packages/sklearn/base.py", line 546, in _validate_data
    X = check_array(X, input_name="X", **check_params)
  File "[USER_DIRS]/miniconda3/envs/downscaling/lib/python3.9/site-packages/sklearn/utils/validation.py", line 940, in check_array
    raise ValueError(
ValueError: Found array with 0 feature(s) (shape=(40512, 0)) while a minimum of 1 is required by StandardScaler.
dvalters commented 1 year ago

It may be that I'm a bit confused about the config files...

in the TopoPyScale folder, I have a config.ini file and in the example datasets folders, (TopoPyScale_examples/ex1_norway etc.) I have a config_point.yml and config_spatial.yml

Do the contents of the yaml files need to be copied to the example ini file? Or do I need all three config files present for the analysis to work?

ArcticSnow commented 1 year ago

Hi @dvalters ,

  1. config.ini is deprecated. I will remove it. Thank you for pointing it out.
  2. toposcale can be run in two mode. One is downscaling to specific points, and the other is aggregating the DEM with k-means. ex1_norway has two config files to show how those two modes are being run.
  3. I have never encountered the error you have. I have updated this morning the version of both topopyscale and of the example repository (and therefore of their respective config files).

you may try pulling the latest repo or updating the pip install (depending on how you installed it).

In general, config files follow the syntax described in: https://topopyscale.readthedocs.io/en/latest/3_configurationFile/

I hope this helps you and resolve the problem you have, which may come from incompability between the main branch of topopycale under development and the example repository. As of my test running the example just now, it should fix the problem.

dvalters commented 1 year ago

Hi @ArcticSnow - I've got further along the processing with the latest pull just now, but am getting an error referring to "reference before assignment" now:

---> Downscaling climate to list of points using TopoScale
TopoPyScale_examples/ex1_norway_finse/outputs/tmp cleaned
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing plev for point 0
Preparing plev for point 1
Preparing plev for point 2
Preparing plev for point 3
Preparing plev for point 4
Preparing plev for point 5
Preparing plev for point 6
Preparing plev for point 7
Preparing plev for point 8
Preparing plev for point 9
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing surf for point 0
Preparing surf for point 1
Preparing surf for point 2
Preparing surf for point 3
Preparing surf for point 4
Preparing surf for point 5
Preparing surf for point 6
Preparing surf for point 7
Preparing surf for point 8
Preparing surf for point 9
Downscaling t,q,p,tp,ws,wd for point: 1
Downscaling t,q,p,tp,ws,wd for point: 2
Downscaling t,q,p,tp,ws,wd for point: 3
Downscaling t,q,p,tp,ws,wd for point: 4
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
Downscaling t,q,p,tp,ws,wd for point: 5
---> WARNING: Point 9 is -403.0 m lower than the 1000hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
Downscaling t,q,p,tp,ws,wd for point: 6
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
Downscaling t,q,p,tp,ws,wd for point: 7
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
Downscaling t,q,p,tp,ws,wd for point: 8
---> WARNING: Point 9 is -11.0 m lower than the 1000hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
Downscaling t,q,p,tp,ws,wd for point: 9
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
Downscaling t,q,p,tp,ws,wd for point: 10
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
ERROR: Upper pressure level 600 hPa geopotential is lower than cluster mean elevation
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File ".miniconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "miniconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "TopoPyScale/TopoPyScale/topo_scale.py", line 302, in pt_downscale_interp
    top = plev_interp.isel(level=ind_z_top)
UnboundLocalError: local variable 'ind_z_top' referenced before assignment
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "TopoPyScale_examples/ex1_norway_finse/pipeline.py", line 12, in <module>
    mp.downscale_climate()
  File "/TopoPyScale/TopoPyScale/topoclass.py", line 439, in downscale_climate
    ta.downscale_climate(self.config.project.directory,
  File "/TopoPyScale/TopoPyScale/topo_scale.py", line 349, in downscale_climate
    multicore_pooling(pt_downscale_interp, fun_param, n_core)
  File "/dev/TopoPyScale/TopoPyScale/topo_scale.py", line 77, in multicore_pooling
    pool.starmap(fun, fun_param)
  File "/miniconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/miniconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/miniconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/miniconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/dev/TopoPyScale/TopoPyScale/topo_scale.py", line 302, in pt_downscale_interp
    top = plev_interp.isel(level=ind_z_top)
UnboundLocalError: local variable 'ind_z_top' referenced before assignment
ArcticSnow commented 1 year ago

The problem is because the higher pressure level in the climate forcing data is not high enough in elevation in respect to the point it tries downscaling at. This behavior is as expected in the case of such situation. This is becuase we cannot interpolate vertically variables if the pressure level do not spread higher than the highest point in the topography.

There seem to be a problem either with your config file, or with the input climate dataset, or with the DEM. I am actually surprised your climate forcing for ex1_norway has a 600hPa pressure level when the config file will download climate data up to 700hPa. Moreover, 600 hPa pressure level should be higher than the 700 hPa. So this is really strange.

https://github.com/ArcticSnow/TopoPyScale_examples/blob/3dee762f8a8918b6ecb585ff340d1bce3ebcf25b/ex1_norway_finse/config_spatial.yml#L25

So the pressure levels must be chosen in relation to the topography. The user must make sure the highest pressure level will be above the topography at all time. There could be a pre-check prior to downscaling (a userfriendly feature to add).

dvalters commented 1 year ago

Thanks for the pointers, I will take a look