search

apache / incubator-wayang

Apache Wayang(incubating) is the first cross-platform data processing system.
https://wayang.incubator.apache.org/
Apache License 2.0
174 stars 70 forks source link

[Question] How to print/explain plans? #417

Open wangxiaoying opened 3 months ago

wangxiaoying commented 3 months ago

Is there a way to explain (print out) the logical and physical plan for wayang?

kbeedkar commented 3 months ago

A quick and dirty solution as a workaround: https://github.com/apache/incubator-wayang/blob/b134ec4a8b8038486025b47091920d1a85d3f298/wayang-api/wayang-api-sql/src/test/java/org/apache/wayang/api/sql/SqlToWayangRelTest.java#L109

zkaoudi commented 3 months ago

Besides that, @juripetersen is currently implementing an explain() functionality in the plan builder that will print the logical, inflated, and physical plans. It should be ready soon :)

wangxiaoying commented 3 months ago

Thanks @kbeedkar @zkaoudi for the quick reply : )

Besides that, @juripetersen is currently implementing an explain() functionality in the plan builder that will print the logical, inflated, and physical plans. It should be ready soon :)

Looking forward to this feature! I tried the one using PlanTraversal as suggested above, but it seems like the result is hard to read. For example, the Q1 and Q3 plan it printed from the wayang-benchmark TPCH test is as follows:

Q1:
Map[Calculate result fields]LocalCallbackSink[collect()]Map[Post-process line item aggregates]ReduceBy[Aggregate line items]PostgresTableSource[Load LINEITEM table]Filter[Filter line items]Map[Project line items]

Q3:
Map[Project customer-order join product]Filter[Filter line items]LocalCallbackSink[collect()]PostgresTableSource[Load LINEITEM table]Map[Extract customer ID]PostgresTableSource[Load ORDERS table]Join[Join CO with line items]PostgresTableSource[Load CUSTOMER table]Map[Unpack orders]Map[Project orders]Map[Extract line item data]Map[Project CO-line-item join product]Map[Project customers]Filter[Filter customers]Join[Join customers with orders]Map[Project line items]Filter[Filter orders]ReduceBy[Aggregate revenue]

It is a bit confusing how to interpret these outputs.

zkaoudi commented 3 months ago

Hi @wangxiaoying,

I assume you saw the last merged commit adding the explain functionality. Let us know if you have any further questions. More than happy to help out.

wangxiaoying commented 3 months ago

Hi @wangxiaoying,

I assume you saw the last merged commit adding the explain functionality. Let us know if you have any further questions. More than happy to help out.

Thank you, I will definitely try!