search

metatron-app / metatron-discovery

Powerful & Easy way for big data discovery
https://metatron.app
Apache License 2.0
442 stars 112 forks source link

Compress filter before rewriting #4256

Open navis opened 1 year ago

navis commented 1 year ago

propagating filter to be used to make bloom filter compressed twice (tpch3)

05:25:44.404 - -> INNER (orders:731060:0.488 + customer:30142:0.201)
05:25:44.435 - -- customer:30142:0.201 (R) is merged into orders (L) as filter on [O_CUSTKEY]
05:25:44.440 - -> INNER (lineitem:3224085:0.537 + orders+customer:146904:0.241)
05:25:44.440 - -- lineitem:3224085:0.537 (L) (-)
05:25:44.440 - -- orders+customer:146904 (R) (hash)
05:25:44.440 - -- .. with bloom filter $view:orders[O_ORDERKEY]((BoundDimFilter{O_ORDERDATE < 1995-03-22} && InDimFilter{dimension='O_CUSTKEY', values=[1, 1000, 100002, 100003, 100004, 100005, 100008, 100010, 100011, 100023, ..30132 more]})) (R) to lineitem:3224085:0.537 (L)
05:25:44.440 - --- selectivity 0.241 merged into lineitem(3224085:0.537 to 1447109:0.241)
05:25:44.444 - -- compressed in filter [O_CUSTKEY], 101,170 bytes into 52,235 bytes (48% reduction, 3 msec)
05:25:44.492 - -- compressed in filter [O_CUSTKEY], 101,170 bytes into 52,235 bytes (48% reduction, 3 msec)
05:25:44.492 - Running 2-way join processing [lineitem, orders+customer]
05:25:44.601 - >> INNER ([lineitem].[L_ORDERKEY](stream:1447109?) --> [orders+customer].[O_ORDERKEY](hashed:147922/147922)) ([L_DISCOUNT, L_EXTENDEDPRICE, L_ORDERKEY] + [O_ORDERDATE, O_ORDERKEY, O_SHIPPRIORITY])
05:25:44.693 - << INNER ([lineitem].[L_ORDERKEY](stream:199286) + [orders+customer].[O_ORDERKEY](hashed:147922/147922)), resulting 30585 rows ([L_DISCOUNT, L_EXTENDEDPRICE, L_ORDERKEY]+[O_ORDERDATE, O_ORDERKEY, O_SHIPPRIORITY]) : (92 msec)
05:25:44.702 - {"success":true,"query/time":361,"query/rows":10,"query/bytes":992}  "SELECT /* TPCH3 */    L_ORDERKEY,    sum(L_EXTENDEDPRICE * (1 - L_DISCOUNT)) AS revenue,    O_ORDERDATE,    O_SHIPPRIORITY FROM    customer,    orders,    lineitem WHERE    C_MKTSEGMENT = 'BUILDING'    and C_CUSTKEY = O_CUSTKEY    and L_ORDERKEY = O_ORDERKEY    and O_ORDERDATE < '1995-03-22'    and L_SHIPD (..49 more..) M    customer,    orders,    lineitem WHERE    C_MKTSEGMENT = 'BUILDING'    and C_CUSTKEY = O_CUSTKEY    and L_ORDERKEY = O_ORDERKEY    and O_ORDERDATE < '1995-03-22'    and L_SHIPDATE > '1995-03-22' GROUP BY    L_ORDERKEY,    O_ORDERDATE,    O_SHIPPRIORITY ORDER BY    revenue desc,    O_ORDERDATE LIMIT 10"