datafusion-contrib / datafusion-federation

Allow DataFusion to resolve queries across remote query engines while pushing down as much compute as possible down.
Apache License 2.0
78 stars 18 forks source link

fix: adds more information when an `EXPLAIN <query>` is run on a federated source #34

Closed phillipleblanc closed 6 months ago

phillipleblanc commented 6 months ago

I wanted an easy way to be able to quickly check what SQL is actually being sent to a remote SQLExecutor, as well as the logical plan that was being federated.

Ideally we could have more information about the compute context at the LogicalPlan level, but I'm not 100% sure what makes sense yet.

Sample output: EXPLAIN="true" cargo run --example sqlite

+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                                                                                                                                                                                                                                                                                                     |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Federated                                                                                                                                                                                                                                                                                                                                                                |
|               |  Limit: skip=0, fetch=10                                                                                                                                                                                                                                                                                                                                                 |
|               |   Projection: t.trackid, t.name AS trackname, a.title AS albumtitle, ar.name AS artistname                                                                                                                                                                                                                                                                               |
|               |     Inner Join:  Filter: a.artistid = ar.artistid                                                                                                                                                                                                                                                                                                                        |
|               |       Inner Join:  Filter: t.albumid = a.albumid                                                                                                                                                                                                                                                                                                                         |
|               |         SubqueryAlias: t                                                                                                                                                                                                                                                                                                                                                 |
|               |           TableScan: track                                                                                                                                                                                                                                                                                                                                               |
|               |         SubqueryAlias: a                                                                                                                                                                                                                                                                                                                                                 |
|               |           TableScan: album                                                                                                                                                                                                                                                                                                                                               |
|               |       SubqueryAlias: ar                                                                                                                                                                                                                                                                                                                                                  |
|               |         TableScan: artist                                                                                                                                                                                                                                                                                                                                                |
| physical_plan | VirtualExecutionPlan name=connector_x_executor compute_context=sqlite://./examples/examples/chinook.sqlite sql=SELECT "t"."trackid", "t"."name" AS "trackname", "a"."title" AS "albumtitle", "ar"."name" AS "artistname" FROM "track" AS "t" JOIN "album" AS "a" ON ("t"."albumid" = "a"."albumid") JOIN "artist" AS "ar" ON ("a"."artistid" = "ar"."artistid") LIMIT 10 |
|               |                                                                                                                                                                                                                                                                                                                                                                          |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+