search

unionai-oss / pandera

A light-weight, flexible, and expressive statistical data testing library
https://www.union.ai/pandera
MIT License
3.33k stars 309 forks source link

Look into `returns` library for higher-kinded types #880

Open cosmicBboy opened 2 years ago

cosmicBboy commented 2 years ago

https://returns.readthedocs.io/

Use case: for the model.SchemaModel class, we want to be able to support higher kinded types in methods e.g. the validate method, such that it can return type-varying generics based on the type of the check_obj argument.

https://returns.readthedocs.io/en/latest/pages/hkt.html

ericfeunekes commented 1 year ago

What about, eg, returning a Result from Validate? That would allow the user to do something with the Failure, like log the failed df for examination.

cosmicBboy commented 1 year ago

I'm guessing Success type would be a breaking change? i.e. I get a Success object instead of a DataFrame, right?

I'd hesitate to introduce this breaking change until perhaps version 1.* of pandera, or perhaps introduce a different method that implements this behavior.

ericfeunekes commented 1 year ago

Definitely a breaking change and likely not something you'd want as a default. Most people probably wouldn't be expecting it.

But it could be implemented either as a return_monad: bool argument in validate; or even easier like you said is a separate method that just wraps validate in a try ... except ... and returns a Result in either case. The Failure case could also return the df of failing values for the user to do something with.

I'm not super familiar with the source code so there may be a better way to implement, but that's off the top of my head.