[JOSS]: KeyError while retreiving ERA5 data using `get_modeldata`

[ashwinvis]

I got as far as this step, https://vergauwenthomas.github.io/MetObs_toolkit/examples/gee_example.html#Extracting-ERA5-timeseries

Input:

#Get the ERA5 data for a single station (to reduce data transfer)
your_station = your_dataset.get_station('vlinder02')

#Extract time series at the location of the station
ERA5_data = your_station.get_modeldata(modelname='ERA5_hourly',
                                      obstype='temp',
                                      startdt=None, #if None, the start of the observations is used
                                      enddt=None, #if None, the end of the observations is used
                                      )

Click below to view:

Traceback

```py --------------------------------------------------------------------------- KeyError Traceback (most recent call last) File ~/test-metobs/lib/python3.11/site-packages/pandas/core/indexes/base.py:3802, in Index.get_loc(self, key, method, tolerance) 3801 try: -> 3802 return self._engine.get_loc(casted_key) 3803 except KeyError as err: File ~/test-metobs/lib/python3.11/site-packages/pandas/_libs/index.pyx:138, in pandas._libs.index.IndexEngine.get_loc() File ~/test-metobs/lib/python3.11/site-packages/pandas/_libs/index.pyx:165, in pandas._libs.index.IndexEngine.get_loc() File pandas/_libs/hashtable_class_helper.pxi:5745, in pandas._libs.hashtable.PyObjectHashTable.get_item() File pandas/_libs/hashtable_class_helper.pxi:5753, in pandas._libs.hashtable.PyObjectHashTable.get_item() KeyError: 'name' The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) Cell In[11], line 5 2 your_station = your_dataset.get_station('vlinder02') 4 #Extract time series at the location of the station ----> 5 ERA5_data = your_station.get_modeldata(modelname='ERA5_hourly', 6 obstype='temp', 7 startdt=None, #if None, the start of the observations is used 8 enddt=None, #if None, the end of the observations is used 9 ) File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/dataset.py:803, in Dataset.get_modeldata(self, modelname, modeldata, obstype, stations, startdt, enddt) 801 # fill modell with data 802 if modelname == "ERA5_hourly": --> 803 Modl.get_ERA5_data(metadf=metadf, 804 startdt_utc=startdt_utc, 805 enddt_utc=enddt_utc, 806 obstype=obstype) 808 else: 809 Modl.get_gee_dataset_data(mapname=modelname, 810 metadf=metadf, 811 startdt_utc=startdt_utc, 812 enddt_utc=enddt_utc, 813 obstype=obstype) File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/modeldata.py:464, in Modeldata.get_ERA5_data(self, metadf, startdt_utc, enddt_utc, obstype) 429 def get_ERA5_data(self, metadf, startdt_utc, enddt_utc, obstype='temp'): 430 """Extract timeseries of the ERA5_hourly dataset. 431 432 The units are converted to the toolkit standard units. (...) 462 463 """ --> 464 self.get_gee_dataset_data(mapname='ERA5_hourly', 465 metadf=metadf, 466 startdt_utc=startdt_utc, 467 enddt_utc=enddt_utc, 468 obstype=obstype) File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/modeldata.py:400, in Modeldata.get_gee_dataset_data(self, mapname, metadf, startdt_utc, enddt_utc, obstype, target_unit_name, conv_expr) 397 connect_to_gee() 399 # Get data using GEE --> 400 df = gee_extract_timeseries(metadf=metadf, 401 mapinfo=geeinfo, 402 startdt=startdt_utc, 403 enddt=enddt_utc, 404 obstype=obstype, 405 latcolname="lat", 406 loncolname="lon", 407 ) 409 if not df.empty: 410 self._df_units[obstype] = geeinfo['band_of_use'][obstype]['units'] File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/landcover_functions.py:472, in gee_extract_timeseries(metadf, mapinfo, startdt, enddt, obstype, latcolname, loncolname) 467 use_drive = True 468 # ============================================================================= 469 # df to featurecollection 470 # ============================================================================= --> 472 ee_fc = _df_to_features_point_collection(metadf) 474 # ============================================================================= 475 # extract raster values 476 # ============================================================================= 478 def rasterExtraction(image): File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/landcover_functions.py:194, in _df_to_features_point_collection(df) 192 poi_geometry = ee.Geometry.Point([row["lon"], row["lat"]]) 193 # construct the attributes (properties) for each point --> 194 poi_properties = poi_properties = {"name": row["name"]} 195 # construct feature combining geometry and properties 196 poi_feature = ee.Feature(poi_geometry, poi_properties) File ~/test-metobs/lib/python3.11/site-packages/pandas/core/series.py:981, in Series.__getitem__(self, key) 978 return self._values[key] 980 elif key_is_scalar: --> 981 return self._get_value(key) 983 if is_hashable(key): 984 # Otherwise index.get_value will raise InvalidIndexError 985 try: 986 # For labels that don't resolve as scalars like tuples and frozensets File ~/test-metobs/lib/python3.11/site-packages/pandas/core/series.py:1089, in Series._get_value(self, label, takeable) 1086 return self._values[label] 1088 # Similar to Index.get_value, but we do not fall back to positional -> 1089 loc = self.index.get_loc(label) 1090 return self.index._get_values_for_loc(self, loc, label) File ~/test-metobs/lib/python3.11/site-packages/pandas/core/indexes/base.py:3804, in Index.get_loc(self, key, method, tolerance) 3802 return self._engine.get_loc(casted_key) 3803 except KeyError as err: -> 3804 raise KeyError(key) from err 3805 except TypeError: 3806 # If we have a listlike key, _check_indexing_error will raise 3807 # InvalidIndexError. Otherwise we fall through and re-raise 3808 # the TypeError. 3809 self._check_indexing_error(key) KeyError: 'name' ```

Debug session:

```py ipdb> u > /home/avmo/test-metobs/lib/python3.11/site-packages/metobs_toolkit/landcover_functions.py(194)_df_to_features_point_collection() 192 poi_geometry = ee.Geometry.Point([row["lon"], row["lat"]]) 193 # construct the attributes (properties) for each point --> 194 poi_properties = poi_properties = {"name": row["name"]} 195 # construct feature combining geometry and properties 196 poi_feature = ee.Feature(poi_geometry, poi_properties) ipdb> print(row) index vlinder02 network Vlinder lat 51.022379 lon 3.709695 call_name Sterre location Gent geometry POINT (3.709695 51.022379) assumed_import_frequency 0 days 00:05:00 dataset_resolution 0 days 00:05:00 lcz Open midrise altitude 7 water_100m 0.0 pervious_100m 0.428769 impervious_100m 0.571231 water_250m 0.0 pervious_250m 0.535944 impervious_250m 0.464056 Name: 0, dtype: object ipdb> q 1 #Get info ```

[vergauwenthomas]

@ashwinvis, I think this bug is already fixed in the master branch. I have tried but could not reproduce it with the master.

Could it be that your /home/avmo/test-metobs/lib/python3.11/site-packages/metobs_toolkit is not the (latest) master ?

(Since the version on PyPi is not synchronized with the github master, the PyPi version is lagging the master)

[vergauwenthomas]

Ref: https://github.com/openjournals/joss-reviews/issues/5916

[ashwinvis]

You are right. This is fixed in the master branch. Closing.