verdict-project / verdict

Interactive-Speed Analytics: 200x Faster, 200x Fewer Cluster Resources, Approximate Query Processing
http://verdictdb.org
Apache License 2.0
248 stars 66 forks source link

Create scrambled table that has same partition columns as its original table #326

Closed dongyoungy closed 5 years ago

dongyoungy commented 5 years ago
codecov-io commented 5 years ago

Codecov Report

Merging #326 into master will increase coverage by 2.47%. The diff coverage is 81.14%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #326      +/-   ##
==========================================
+ Coverage   68.24%   70.71%   +2.47%     
==========================================
  Files         167      167              
  Lines       11198    11233      +35     
  Branches     1839     1847       +8     
==========================================
+ Hits         7641     7942     +301     
+ Misses       3095     2831     -264     
+ Partials      462      460       -2
Impacted Files Coverage Δ
.../verdictdb/sqlreader/ScramblingQueryGenerator.java 0% <ø> (ø) :arrow_up:
...va/org/verdictdb/coordinator/ExecutionContext.java 87.46% <100%> (+16.66%) :arrow_up:
...ctdb/core/scrambling/CreateScrambledTableNode.java 73.08% <100%> (+0.53%) :arrow_up:
.../org/verdictdb/core/scrambling/ScramblingNode.java 94.12% <100%> (+0.62%) :arrow_up:
.../verdictdb/core/sqlobject/CreateScrambleQuery.java 45.46% <45.46%> (+3.46%) :arrow_up:
.../java/org/verdictdb/connection/JdbcConnection.java 73.88% <80%> (+8.13%) :arrow_up:
...g/verdictdb/coordinator/ScramblingCoordinator.java 76.38% <84.22%> (+0.94%) :arrow_up:
...ava/org/verdictdb/metastore/ScrambleMetaStore.java 97.39% <0%> (-1.3%) :arrow_down:
...a/org/verdictdb/jdbc41/VerdictStreamResultSet.java 8.29% <0%> (-0.26%) :arrow_down:
.../core/scrambling/FastConvergeScramblingMethod.java 91.08% <0%> (+0.3%) :arrow_up:
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0c32d73...5384909. Read the comment docs.

dongyoungy commented 5 years ago

For 'partition by list' syntax that we use to implement partitioned tables in Postgres and MySQL, Postgres only supports a single column partition while MySQL can support partition with multiple columns using 'LIST COLUMNS'.

For now, I have made it to throw an exception for Postgres and MySQL when one tries to create a scramble for already-partitioned tables as suggested. This check is done by existing checkIfSupported() method in CreateScrambleQuery.

We may implement multi-column partitioned scrambles for MySQL in the future, but I do not think it is necessary at the moment.

pyongjoo commented 5 years ago

Thanks. Please merge to master whenever you think it's OK.