jeffzi
commented
2 years ago Hi @roshcagra, interesting question !
pandera.errors.SchemaError: expected series 'geometry' to have type None, got object
It's a little confusing to trace back the source of the error but here is the gist of it. Your BaseGeometry.type is None by default and your SchemaModel does not provide a type. During validation, pandera will call BaseGeometry.check(), inherited from pandas_engine.DataType, which leverages the type argument. The type is supposed to be something understood by pandas. You can pass arguments to the dtype with the following syntax:
class GeoDataFrameSchema(SchemaModel):
geometry: Series[BaseGeometry] = Field(dtype_kwargs={"type": APPROPRIATE_TYPE}) # APPROPRIATE_TYPE = ?I'm not familiar with geopandas or shapely, and I couldn't see usage of subclasses of shapely.geometry.base.BaseGeometry in geopandas getting started tutorial. Geopandas seems to only have a single dtype GeometryDtype. I would use it to benefit from the work that has already been done by geopandas.
import geopandas
import pandera as pa
from pandera.engines import pandas_engine
@pandas_engine.Engine.register_dtype(
equivalents=[ # Let pandera know how to translate this data type from other objects
"geometry",
geopandas.array.GeometryDtype,
geopandas.array.GeometryDtype(),
]
)
@pa.dtypes.immutable
class Geometry(pandas_engine.DataType):
type = geopandas.array.GeometryDtype()
class GeoDataFrameSchema(pa.SchemaModel):
geometry: pa.typing.Series[Geometry]
BoroCode: pa.typing.Series[Geometry] # should fail (contains int)
BoroName: pa.typing.Series[Geometry] # should fail (contains object)
gdf = geopandas.read_file(geopandas.datasets.get_path("nybb"))
gdf.info()
#> <class 'geopandas.geodataframe.GeoDataFrame'>
#> RangeIndex: 5 entries, 0 to 4
#> Data columns (total 5 columns):
#> # Column Non-Null Count Dtype
#> --- ------ -------------- -----
#> 0 BoroCode 5 non-null int64
#> 1 BoroName 5 non-null object
#> 2 Shape_Leng 5 non-null float64
#> 3 Shape_Area 5 non-null float64
#> 4 geometry 5 non-null geometry
#> dtypes: float64(2), geometry(1), int64(1), object(1)
#> memory usage: 328.0+ bytes
# verify that pandera recognizes geometry
print(repr(pandas_engine.Engine.dtype(gdf["geometry"].dtype)))
#> DataType(geometry)
GeoDataFrameSchema.validate(gdf, lazy=True)
#> Traceback (most recent call last):
#> ...
#> SchemaErrors: A total of 2 schema errors were found.
#> Error Counts
#> - schema_component_check: 2
#> Schema Error Summary
#> failure_cases n_failure_cases
#> schema_context column check
#> Column BoroCode dtype('geometry') [int64] 1
#> BoroName dtype('geometry') [object] 1
#> Usage Tip
#> Directly inspect all errors by catching the exception:
#> ```
#> try:
#> schema.validate(dataframe, lazy=True)
#> except SchemaErrors as err:
#> err.failure_cases # dataframe of schema errors
#> err.data # invalid dataframe
#> ```^ The draft above is not tested but works in this basic example.
@roshcagra Would you be interested in extending this snippet and contributing proper geopandas support? I'm sure other geopandas users would benefit from schema validation !
roshcagra
cosmicBboy
m-richards
How to properly extend pandas_engine.DataType
Hi! I'm trying to use pandera with GeoPandas - I think all I should need to do to make it work is add support for the
geometrycolumn by registeringBaseGeometryas aDataType. However, I'm struggling to get it to work - any suggestions?