search

great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
9.87k stars 1.53k forks source link

Polars Support? #9649

Closed edgBR closed 1 month ago

edgBR commented 6 months ago

Is your feature request related to a problem? Please describe. Hi, we have been using GE before in our team but we are migrating most of our pandas processing pipelines to polars. However it seems that GE does not have any support for polars and transforming the polars dataset to pandas just to run the GE suite seems extremely subotimal.

Describe the solution you'd like We would like to get polars support in GE.

Describe alternatives you've considered We have considered to use pandera, which is going to offer full polars support very soon:

https://github.com/unionai-oss/pandera/issues/1064

https://pandera--1373.org.readthedocs.build/en/1373/polars.html#polars

Additional context We are in the situation were our data is not big enough to use spark. Therefore GE with pyspark or pydequee have been discarded for data validation.

abekfenn commented 5 months ago

Jumping on the bandwagon here. We are also exploring migrating from pandas to polars in our data processing pipelines. Pandera is working on this and it appears support for polars is on its way there. Polars is blazingly fast compared to pandas and this would be incredibly helpful if GX added support for a polars execution engine.

Bidek56 commented 2 months ago

5914 was created for Polars but closed without a comment. :(

molliemarie commented 1 month ago

Hello @edgBR. With the launch of Great Expectations Core (GX 1.0), we are closing old issues posted regarding previous versions. Moving forward, we will focus our resources on supporting and improving GX Core (version 1.0 and beyond). If you find that an issue you previously reported still exists in GX Core, we encourage you to resubmit it against the new version. With more resources dedicated to community support, we aim to tackle new issues swiftly. For specific details on what is GX-supported vs community-supported, you can reference our integration and support policy.

To get started on your transition to GX Core, check out the GX Core quickstart (click “Full example code” tab to see a code example).

You can also join our upcoming community meeting on August 28th at 9am PT (noon ET / 4pm UTC) for a comprehensive rundown of everything GX Core, plus Q&A as time permits. Go to https://greatexpectations.io/meetup and click “follow calendar” to follow the GX community calendar.

Thank you for being part of the GX community and thank you for submitting this issue. We're excited about this new chapter and look forward to your feedback on GX Core. 🤗