search

unionai-oss / pandera

A light-weight, flexible, and expressive statistical data testing library
https://www.union.ai/pandera
MIT License
3.37k stars 310 forks source link

Spark dataframe support #816

Closed Niccolum closed 2 years ago

Niccolum commented 2 years ago

Is your feature request related to a problem? Please describe. pandera is a great library, but it doesn't support pyspark DataFrame yet

Describe the solution you'd like Schema model for pyspark DataFrame

import pyspark
import pandera as pa
from pandera.typing.pyspark import DataFrame, Series

class Schema(pa.SchemaModel):
    foo: Series[str]

spark_df = spark.read.parquet('foo.parquet')
spark_df = Schema(spark_df)

pip install pandera[pyspark]

cosmicBboy commented 2 years ago

woops, hang tight @Niccolum, I updated the README too soon 😅 0.10.0, which adds support for pyspark.pandas, should be out by EOD today

cosmicBboy commented 2 years ago

hey @Niccolum pandera now supports pyspark.pandas dataframes! https://pandera.readthedocs.io/en/stable/pyspark.html

Out of curiosity: I haven't tested validating OG pyspark dataframes, I suspect that is not supported... is this a feature that you'd want, or is converting pyspark DFs to pyspark.pandas DFs sufficient?

Niccolum commented 2 years ago

In the future, it may be needed, but for now it is enough)