search

bird-house / finch

A Web Processing Service for Climate Indicators
https://finch.readthedocs.io/en/latest/
Apache License 2.0
13 stars 5 forks source link

Finch notebooks failure after Geoserver upgrade #206

Closed tlvu closed 3 years ago

tlvu commented 3 years ago

Description

Since our notebooks are unable to test Geoserver from a dev machine (issue https://github.com/Ouranosinc/pavics-sdi/issues/183), this error is caught only after a go-live on prod of PR https://github.com/bird-house/birdhouse-deploy/pull/136.

Steps to Reproduce

Jenkins build error: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/master/1276/console

KeyError: "['bbox'] not found in axis"

regridding.ipynb:

12:15:43  ______ pavics-sdi-master/docs/source/notebooks/regridding.ipynb::Cell 22 _______
12:15:43  Notebook cell execution failed
12:15:43  Cell 22: Cell execution caused an exception
12:15:43  
12:15:43  Input:
12:15:43  # NBVAL_IGNORE_OUTPUT
12:15:43  
12:15:43  wfs_url = 'http://boreas.ouranos.ca/geoserver/wfs'
12:15:43  # Connect to GeoServer WFS service.
12:15:43  wfs = WebFeatureService(wfs_url, version='2.0.0')
12:15:43  
12:15:43  # Get the json as a binary stream
12:15:43  # Here we select Quebec's MRCs polygons
12:15:43  # We select only a few properties
12:15:43  data = wfs.getfeature(
12:15:43      typename='public:quebec_mrc_boundaries',
12:15:43      #bbox=(-93.1, 41.1, -75.0, 49.6),
12:15:43      outputFormat='json',
12:15:43      propertyname=['the_geom', 'MRS_NM_MRC']
12:15:43  )
12:15:43  # Load into a GeoDataFrame by reading the json on-the-fly
12:15:43  shapes_all = gpd.GeoDataFrame.from_features(json.load(data))
12:15:43  # Just for simplicity, let's take 10 large MRCs
12:15:43  shapes_all['AREA'] = shapes_all.area
12:15:43  shapes = shapes_all.sort_values('AREA').iloc[-20:-10].drop(columns=['bbox']).set_index('MRS_NM_MRC')
12:15:43  shapes
12:15:43  
12:15:43  Traceback:
12:15:43  
12:15:43  ---------------------------------------------------------------------------
12:15:43  KeyError                                  Traceback (most recent call last)
12:15:43  /tmp/ipykernel_438/1389203164.py in <module>
12:15:43       18 # Just for simplicity, let's take 10 large MRCs
12:15:43       19 shapes_all['AREA'] = shapes_all.area
12:15:43  ---> 20 shapes = shapes_all.sort_values('AREA').iloc[-20:-10].drop(columns=['bbox']).set_index('MRS_NM_MRC')
12:15:43       21 shapes
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/frame.py in drop(self, labels, axis, index, columns, level, inplace, errors)
12:15:43     4313             level=level,
12:15:43     4314             inplace=inplace,
12:15:43  -> 4315             errors=errors,
12:15:43     4316         )
12:15:43     4317 
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/generic.py in drop(self, labels, axis, index, columns, level, inplace, errors)
12:15:43     4151         for axis, labels in axes.items():
12:15:43     4152             if labels is not None:
12:15:43  -> 4153                 obj = obj._drop_axis(labels, axis, level=level, errors=errors)
12:15:43     4154 
12:15:43     4155         if inplace:
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/generic.py in _drop_axis(self, labels, axis, level, errors)
12:15:43     4186                 new_axis = axis.drop(labels, level=level, errors=errors)
12:15:43     4187             else:
12:15:43  -> 4188                 new_axis = axis.drop(labels, errors=errors)
12:15:43     4189             result = self.reindex(**{axis_name: new_axis})
12:15:43     4190 
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/indexes/base.py in drop(self, labels, errors)
12:15:43     5590         if mask.any():
12:15:43     5591             if errors != "ignore":
12:15:43  -> 5592                 raise KeyError(f"{labels[mask]} not found in axis")
12:15:43     5593             indexer = indexer[~mask]
12:15:43     5594         return self.delete(indexer)
12:15:43  
12:15:43  KeyError: "['bbox'] not found in axis"

subset.ipynb:

12:15:43  Notebook cell execution failed
12:15:43  Cell 7: Cell execution caused an exception
12:15:43  
12:15:43  Input:
12:15:43  # NBVAL_IGNORE_OUTPUT
12:15:43  # Select 3 regions around the city of Montr��al (drop unused "bbox" dimension)
12:15:43  poly_mtl = all_shapes[all_shapes.RES_NM_REG.isin(['Mont��r��gie', 'Montr��al', 'Laval'])].drop(columns=['bbox'])
12:15:43  poly_mtl
12:15:43  
12:15:43  Traceback:
12:15:43  
12:15:43  ---------------------------------------------------------------------------
12:15:43  KeyError                                  Traceback (most recent call last)
12:15:43  /tmp/ipykernel_719/432195274.py in <module>
12:15:43        1 # NBVAL_IGNORE_OUTPUT
12:15:43        2 # Select 3 regions around the city of Montr��al (drop unused "bbox" dimension)
12:15:43  ----> 3 poly_mtl = all_shapes[all_shapes.RES_NM_REG.isin(['Mont��r��gie', 'Montr��al', 'Laval'])].drop(columns=['bbox'])
12:15:43        4 poly_mtl
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/frame.py in drop(self, labels, axis, index, columns, level, inplace, errors)
12:15:43     4313             level=level,
12:15:43     4314             inplace=inplace,
12:15:43  -> 4315             errors=errors,
12:15:43     4316         )
12:15:43     4317 
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/generic.py in drop(self, labels, axis, index, columns, level, inplace, errors)
12:15:43     4151         for axis, labels in axes.items():
12:15:43     4152             if labels is not None:
12:15:43  -> 4153                 obj = obj._drop_axis(labels, axis, level=level, errors=errors)
12:15:43     4154 
12:15:43     4155         if inplace:
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/generic.py in _drop_axis(self, labels, axis, level, errors)
12:15:43     4186                 new_axis = axis.drop(labels, level=level, errors=errors)
12:15:43     4187             else:
12:15:43  -> 4188                 new_axis = axis.drop(labels, errors=errors)
12:15:43     4189             result = self.reindex(**{axis_name: new_axis})
12:15:43     4190 
12:15:43  
12:15:43  /opt/conda/envs/birdy/lib/python3.7/site-packages/pandas/core/indexes/base.py in drop(self, labels, errors)
12:15:43     5590         if mask.any():
12:15:43     5591             if errors != "ignore":
12:15:43  -> 5592                 raise KeyError(f"{labels[mask]} not found in axis")
12:15:43     5593             indexer = indexer[~mask]
12:15:43     5594         return self.delete(indexer)
12:15:43  
12:15:43  KeyError: "['bbox'] not found in axis"

Additional Information

Similar Raven issue https://github.com/Ouranosinc/raven/issues/410

aulemahal commented 3 years ago

At a quick glance, I find it weird that the returned polygon collection has different attributes with only a server upgrade, this seems quite an easy fix anyway.

Zeitsperre commented 3 years ago

This is exactly the type of error I was expecting post-upgrade. It appears that one version of the quebec_mrc_boundaries had a BBOX field, and we dropped that particular version in the upgrade. By chance, If anything, this might result in cleaner code. I wonder if this field was automatically generated by the previous GeoServer, given that the files are the same...

aulemahal commented 3 years ago

I am guessing the field was automatically generated! In the WFS calls, we have propertyname=['the_geom', 'MRS_NM_MRC'], so it was kinda strange that an additional "bbox" field was also provided.

tlvu commented 3 years ago

@aulemahal I'll test both PR on Jenkins once power is back tomorrow and will merge. Thanks for the fix !