jeffzi
commented
3 years ago I'd like to suggest another use case for a refactor of pandera dtypes.
I use pandera to validate pandas DataFrames that are ultimately written as parquet files via pyarrow. Parquet supports date and decimal types which are not natively supported by pandas but can be stored in object columns. I wanted to let pandera coerce to those types during validation. So far, my solution has been to subclass pandera.Column and override coerce_dtype.
Example for date:
from typing import Optional
import pandas as pd
import pandera as pa
class DateColumn(pa.Column):
"""Column containing date values encoded as date types (not datetime)."""
def __init__(
self,
checks: pa.schemas.CheckList = None,
nullable: bool = False,
allow_duplicates: bool = True,
coerce: bool = False,
required: bool = True,
name: Optional[str] = None,
regex: bool = False,
) -> None:
super().__init__(
pa.Object, # <===========
checks=checks,
nullable=nullable,
allow_duplicates=allow_duplicates,
coerce=coerce,
required=required,
name=name,
regex=regex,
)
def coerce_dtype(self, series: pd.Series) -> pd.Series:
"""Coerce a pandas.Series to date types."""
try:
dttms = pd.to_datetime(series, infer_datetime_format=True, utc=True)
except TypeError as err:
msg = f"Error while coercing '{self.name} to type'date'"
raise TypeError(msg) from err
return dttms.dt.date
schema = pa.DataFrameSchema(columns={"dt": DateColumn(coerce=True)})
df = pd.DataFrame({'dt':pd.date_range("2021-01-01", periods=1, freq="H")})
print(df)
#> dt
#> 0 2021-01-01
print(df["dt"].dtype)
#> datetime64[ns]
df = schema.validate(df)
print(df)
#> dt
#> 0 2021-01-01
print(df["dt"].dtype)
#> objectCreated on 2021-01-07 by the reprexpy package
The issue with the above is that custom columns are not compatible with SchemaModel.
I suggest to move the coerce_dtype() to a Dtype class that could be subclassed to add arguments to the __init__ and/or modify coercion. That would help decouple the coercion logic from DataFrameSchema. e.g:
https://github.com/pandera-dev/pandera/blob/bfdb118504f78f7dcf6dcba96f6200194b4f5fbf/pandera/schemas.py#L285
cosmicBboy
ryanhaarmann
Is your feature request related to a problem? Please describe.
Currently,
pandera's type system is strongly coupled to thepandastype system. This works well in pandera's current state since it only supports pandas dataframe validation. However, in order to obtain a broader coverage of dataframe-like data structures in the python ecosystem, I think it makes sense to slowly move towards this goal by abstracting pandera's type system so that it's not so strongly coupled with pandas' type system.The
PandasDtypeenum class needs to be made more flexible such that it supports types with dynamic definitions likeCategoryDtypeandPeriodDtype.Describe the solution you'd like
TBD
Describe alternatives you've considered
TBD
Additional context
TBD