verdict-project / verdict

Interactive-Speed Analytics: 200x Faster, 200x Fewer Cluster Resources, Approximate Query Processing
http://verdictdb.org
Apache License 2.0
248 stars 66 forks source link

WIP:Joezhong fix 320 #323

Closed Beastjoe closed 5 years ago

Beastjoe commented 5 years ago

Related Issue #320

Sorry for the delay. I tried to solve the case when scramble tables are involved in set operation. Currently, no AsyncAggExecutionNode is created if scramble tables are in set operation. The reason is that VerdictDB will select and group by verditdbtier column. However, in set operation, the number of selected columns should be the same. For instance, select count(something) from (select something from scramble_table UNION select something from normal_table) If we try to use AsyncAggExecutionNode here, the query created by the individual AggExecutionNode will become like select something, verdictdbtier from scramble_table where verdictdbblock>=0 UNION select something from normal_table

codecov-io commented 5 years ago

Codecov Report

Merging #323 into master will increase coverage by 0.33%. The diff coverage is 81.04%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #323      +/-   ##
==========================================
+ Coverage   70.62%   70.94%   +0.33%     
==========================================
  Files         168      168              
  Lines       11492    11696     +204     
  Branches     1874     1935      +61     
==========================================
+ Hits         8115     8297     +182     
- Misses       2891     2904      +13     
- Partials      486      495       +9
Impacted Files Coverage Δ
.../main/java/org/verdictdb/sqlwriter/QueryToSql.java 64% <0%> (-4%) :arrow_down:
...verdictdb/core/querying/ola/InMemoryAggregate.java 83.46% <100%> (+0.12%) :arrow_up:
...ictdb/core/querying/QueryNodeWithPlaceHolders.java 88.28% <100%> (ø) :arrow_up:
...java/org/verdictdb/sqlwriter/SelectQueryToSql.java 83.16% <100%> (+0.45%) :arrow_up:
...org/verdictdb/sqlreader/ScrambleTableReplacer.java 78% <33.34%> (-4.22%) :arrow_down:
...verdictdb/core/sqlobject/SetOperationRelation.java 58.63% <52.64%> (-3.28%) :arrow_down:
...va/org/verdictdb/coordinator/ExecutionContext.java 85.1% <66.67%> (+0.09%) :arrow_up:
...main/java/org/verdictdb/sqlreader/RelationGen.java 89.31% <66.67%> (-2.5%) :arrow_down:
.../org/verdictdb/sqlreader/RelationStandardizer.java 79.88% <85.72%> (+0.4%) :arrow_up:
...tdb/core/querying/ola/AsyncQueryExecutionPlan.java 84.27% <88.6%> (-0.54%) :arrow_down:
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2922390...f16d283. Read the comment docs.

Beastjoe commented 5 years ago

@pyongjoo I think now set operation is supported for async aggregation. When scramble tables have different probability distribution, VerdictDB will not create asyncAggExecutionNode.