delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python

https://delta-io.github.io/delta-rs/

Apache License 2.0

1.98k stars 365 forks source link

Is there a way to see a query plan? #2481

Open geekodour opened 2 months ago

geekodour commented 2 months ago

Description

See this blogpost: https://salim-sayed.medium.com/spark-delta-lake-merge-slowness-due-to-skewed-partitions-f212285bd04f It shows an example of the Physical plan

Use Case I am trying to debug a slow merge while using delta-rs via polars and I am not sure how should I access the query plan. Is it something that's accessible?

geekodour commented 2 months ago

Another example: https://kb.databricks.com/delta/delta-merge-into

Think this is more of a pyarrow question?

ion-elgreco commented 2 months ago

Not really at the moment. You could modify the delta-rs code to return/dbg the plan.

Are you merging on the full table or a set of partitions?

echai58 commented 1 month ago

If you set RUST_LOG=debug datafusion will print out the full optimized/unoptimized query plans, if that's what you're looking for.