[Feature Request]DuckDB as another analytic engine for Deequ

[chenliu0831]

Is your feature request related to a problem? Please describe. Today, PyDeequ is a PySpark binding for Deequ which is in Scala and Spark only. While it is a good fit for DEs, Spark is not a great fit for many DS use-cases who will have datasets fit in memory and do not want to setup Spark.

See initial ideas here https://youtu.be/fvKFOfaLwBA?t=1393 from @sscdotopen.

Describe the solution you'd like

As discussed in above video, it would be good to create the proper abstractions to support another analytic engine. DuckDB who has gained popularity recently can be another analytic engine. The designs need more thoughts/discussion.

[tdoehmen]

Hi, I am a PhD student who has worked with @sscdotopen on implementing this idea. The main challenge was indeed seperating the business logic from the execution engine. We published a workshop paper about that (https://ssc.io/pdf/duckdq.pdf) and released the source code here: https://github.com/tdoehmen/duckdq

[chenliu0831]

@tdoehmen happy to hear from you! We will take a look at the paper and the repo. Will you be open to evolve/migrate some code to Pydeequ if that's the right thing to do?

In addition to bring another analytic engine, in general we want to improve the ergonomics of Deequ working for Python users.

[chenliu0831]

For the business logic - execution engine separation, I think ibis could be an useful candidate to help. We might add a detailed write-up on design here.

[tdoehmen]

I am a bit limited on time, but happy to look over a more detailed design. When parts of duckdq should make sense to be migrated I'm also happy to help out. Just a few more random thoughts on the challenges involved in extending (py)deequ with another execution engine:

• since pydeequ is mainly a wrapper around the original deequ scala library, in my eyes, changing the execution engine either requires far-reaching changes to the deequ scala codebase, or a (potentially more lightweight) replication of the deequ architecture inside the pydeequ library. In DuckDQ we went for the second approach. One reason that makes it hard to re-use the existing deequ codebase for another execution engine is that it heavily builds on Spark's DataFrame API rather than a more generic language such as SQL (or in the python world maybe something like ibis).
• another thing to keep in mind is that some deequ analyzers need functionality that is not necessarily exposed by the execution engine. E.g. to compute the intermediate state for the stddev analyzer, we need the count, mean and dsquared value (see this DuckDB branch we did for this purpose https://github.com/tdoehmen/duckdb/commit/077129acea4efac936beea26348690f9b9c1a1e6). Same for Sketch-based analyzers like the ApproxQuantile and ApproxCountDistinct. In DuckDQ, we used apache-datasketches to compute these.
• one other key design improvement we did in duckdq was to decouple state merging (engine-indepdent) from state computation (engine-dependent), which is kind of intermingled in the analyzers of the original deequ codebase and makes using different compute backends more complex than necessary because merging states is not computationally intensive (apart from edge cases like exact uniqueness). I dug out a more detailed architecture diagram of duckdq that shows the separtion more clearly. State Engine is the engine-agnostic part that can merge states (pure python, apart from sketch merging), and Base Engine is then quite straightforward to implement for any desired computational backend. E.g. we implemented a generic SQL engine based on SQLAlchemy, and a DuckDB SQL engine that exploits DuckDB-specifics like the stddev-extension and the zero-copy handover of Pandas DataFrames, which makes it very efficient to do data quality validation on them: