search

bakoe / hyrise

Hyrise is a research in-memory database.
https://hpi.de/plattner/projects/hyrise.html
MIT License
0 stars 0 forks source link

WIP: Add parallelization to replaced sort operator #13

Open bakoe opened 4 years ago

bakoe commented 4 years ago

Benchmarking results on the remote VM running the the old state (e2c746018a2d5c1c19c3ec73c2075ce038c6b8e9):

Bastian.Koenig@vm-appleton:~/hyrise$ cmake-build-release/hyriseMicroBenchmarks --benchmark_filter=BM_Sort*
2020-01-28 23:04:28
Running cmake-build-release/hyriseMicroBenchmarks
Run on (32 X 2500 MHz CPU s)
CPU Caches:
  L1 Data 32K (x32)
  L1 Instruction 32K (x32)
  L2 Unified 256K (x32)
  L3 Unified 25600K (x32)
Load Average: 3.49, 4.71, 2.73
--------------------------------------------------------------------------------------------
Benchmark                                                  Time             CPU   Iterations
--------------------------------------------------------------------------------------------
SortPicoBenchmark/BM_SortPico                           3082 ns         3082 ns       227453
SortSmallBenchmark/BM_SortSmall                       664765 ns       664764 ns         1000
SortBenchmark/BM_Sort                               11260624 ns     11259791 ns           81
SortLargeBenchmark/BM_SortLarge                     74649837 ns     74610777 ns            9
SortReferencePicoBenchmark/BM_SortReferencePico         3834 ns         3834 ns       188811
SortReferenceSmallBenchmark/BM_SortReferenceSmall     809404 ns       809355 ns         1087
SortReferenceBenchmark/BM_SortReference             17714122 ns     17714127 ns           34
SortReferenceLargeBenchmark/BM_SortReferenceLarge  117632771 ns    117627624 ns            5
SortStringSmallBenchmark/BM_SortStringSmall          3284776 ns      3279945 ns          172
SortStringBenchmark/BM_SortString                   26705476 ns     26705487 ns           26
SortStringLargeBenchmark/BM_SortStringLarge        463878036 ns    463872157 ns            2
SortNullBenchmark/BM_SortNullBenchmark               7927768 ns      7927661 ns          102
SortBenchmark/BM_SortSingleColumnSQL                12534447 ns     12532888 ns           53
SortBenchmark/BM_SortMultiColumnSQL                 17271437 ns     17270361 ns           38
bakoe commented 4 years ago

Benchmarking on the remote VM running the naïve parallelisation implementation (99ff033):

Bastian.Koenig@vm-appleton:~/hyrise$ cmake-build-release/hyriseMicroBenchmarks --benchmark_filter=BM_Sort*
2020-01-28 23:12:17
Running cmake-build-release/hyriseMicroBenchmarks
Run on (32 X 2500 MHz CPU s)
CPU Caches:
  L1 Data 32K (x32)
  L1 Instruction 32K (x32)
  L2 Unified 256K (x32)
  L3 Unified 25600K (x32)
Load Average: 0.40, 1.35, 1.85
--------------------------------------------------------------------------------------------
Benchmark                                                  Time             CPU   Iterations
--------------------------------------------------------------------------------------------
SortPicoBenchmark/BM_SortPico                           3663 ns         3663 ns       187498
SortSmallBenchmark/BM_SortSmall                      1056024 ns      1055703 ns          515
SortBenchmark/BM_Sort                               16072711 ns     16062487 ns           37
SortLargeBenchmark/BM_SortLarge                    126499748 ns    126494666 ns            5
SortReferencePicoBenchmark/BM_SortReferencePico         5330 ns         5329 ns       161766
SortReferenceSmallBenchmark/BM_SortReferenceSmall    1357965 ns      1357348 ns          427
SortReferenceBenchmark/BM_SortReference             19685890 ns     19685720 ns           46
SortReferenceLargeBenchmark/BM_SortReferenceLarge  182483435 ns    182472399 ns            4
SortStringSmallBenchmark/BM_SortStringSmall          5162711 ns      5161771 ns          100
SortStringBenchmark/BM_SortString                   56570768 ns     56568191 ns           10
SortStringLargeBenchmark/BM_SortStringLarge        529530764 ns    529489929 ns            1
SortNullBenchmark/BM_SortNullBenchmark              13554956 ns     13554955 ns           43
SortBenchmark/BM_SortSingleColumnSQL                15535304 ns     15530165 ns           42
SortBenchmark/BM_SortMultiColumnSQL                 26829529 ns     26829562 ns           25
bakoe commented 4 years ago

TPC-H performance comparisons of the master branch state (d2571500dc039dd0e7a483ec2ea6e43b5b381f9d) and this PR's current state d675cebcbb20657991a6e421fb323c8b3aaa52d4 are as follows:

 +----------------+--------------------+------+--------------------+------+------------+---------+
 | Benchmark      | prev. iter/s       | runs | new iter/s         | runs | change [%] | p-value |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 | TPC-H 01       | 0.689612090587616  | 42   | 0.6885654330253601 | 42   | -0%        |  0.8214 |
+| TPC-H 02       | 42.83256912231445  | 2570 | 51.07833480834961  | 3065 | +19%       |  0.0000 |
 | TPC-H 03       | 4.662003993988037  | 280  | 4.614461898803711  | 277  | -1%        |  0.0076 |
 | TPC-H 04       | 4.347574234008789  | 261  | 4.441842555999756  | 267  | +2%        |  0.0041 |
 | TPC-H 05       | 2.3762333393096924 | 143  | 2.4757614135742188 | 149  | +4%        |  0.0000 |
 | TPC-H 06       | 120.64415740966797 | 7239 | 122.11299896240234 | 7327 | +1%        |  0.0000 |
 | TPC-H 07       | 5.355220317840576  | 322  | 5.396876811981201  | 324  | +1%        |  0.2429 |
 | TPC-H 08       | 3.5660836696624756 | 214  | 3.6496741771698    | 219  | +2%        |  0.0000 |
 | TPC-H 09       | 1.2240474224090576 | 74   | 1.2318470478057861 | 74   | +1%        |  0.2214 |
 | TPC-H 10       | 2.019357681274414  | 122  | 1.9842945337295532 | 120  | -2%        |  0.0671 |
 | TPC-H 11       | 19.570510864257812 | 1175 | 19.133033752441406 | 1148 | -2%        |  0.0000 |
 | TPC-H 12       | 10.40722370147705  | 625  | 10.097774505615234 | 606  | -3%        |  0.0000 |
 | TPC-H 13       | 1.649210810661316  | 99   | 1.6133252382278442 | 97   | -2%        |  0.0000 |
 | TPC-H 14       | 32.8909797668457   | 1974 | 33.59960174560547  | 2016 | +2%        |  0.0000 |
 | TPC-H 15       | 64.49054718017578  | 3870 | 64.40335845947266  | 3865 | -0%        |  0.3457 |
+| TPC-H 16       | 5.088113784790039  | 306  | 5.609182357788086  | 337  | +10%       |  0.0000 |
 | TPC-H 17       | 2.8013389110565186 | 169  | 2.830157995223999  | 170  | +1%        |  0.0418 |
-| TPC-H 18       | 0.8350475430488586 | 51   | 0.7895979285240173 | 48   | -5%        |  0.0037 |
+| TPC-H 19       | 5.26102876663208   | 316  | 5.543107986450195  | 333  | +5%        |  0.0000 |
 | TPC-H 20       | 16.5688419342041   | 995  | 17.18280029296875  | 1031 | +4%        |  0.0000 |
 | TPC-H 21       | 0.7997806072235107 | 49   | 0.783776044845581  | 48   | -2%        |  0.0972 |
 | TPC-H 22       | 8.714345932006836  | 523  | 8.631366729736328  | 518  | -1%        |  0.0203 |
 | geometric mean |                    |      |                    |      | +1%        |         |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 +----------------+--------------------+------+--------------------+------+------------+---------+
 | Benchmark      | prev. iter/s       | runs | new iter/s         | runs | change [%] | p-value |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 | TPC-H 01       | 0.689612090587616  | 42   | 0.6849160194396973 | 42   | -1%        |  0.3269 |
+| TPC-H 02       | 42.83256912231445  | 2570 | 51.023277282714844 | 3062 | +19%       |  0.0000 |
 | TPC-H 03       | 4.662003993988037  | 280  | 4.763455390930176  | 286  | +2%        |  0.0000 |
 | TPC-H 04       | 4.347574234008789  | 261  | 4.256586074829102  | 256  | -2%        |  0.0081 |
+| TPC-H 05       | 2.3762333393096924 | 143  | 2.6027915477752686 | 157  | +10%       |  0.0000 |
 | TPC-H 06       | 120.64415740966797 | 7239 | 119.70783996582031 | 7183 | -1%        |  0.0023 |
 | TPC-H 07       | 5.355220317840576  | 322  | 5.26931619644165   | 317  | -2%        |  0.0171 |
 | TPC-H 08       | 3.5660836696624756 | 214  | 3.6769168376922607 | 221  | +3%        |  0.0000 |
 | TPC-H 09       | 1.2240474224090576 | 74   | 1.2242646217346191 | 74   | +0%        |  0.9762 |
 | TPC-H 10       | 2.019357681274414  | 122  | 1.9808576107025146 | 119  | -2%        |  0.0398 |
 | TPC-H 11       | 19.570510864257812 | 1175 | 19.42022132873535  | 1166 | -1%        |  0.0002 |
-| TPC-H 12       | 10.40722370147705  | 625  | 9.86573600769043   | 592  | -5%        |  0.0000 |
 | TPC-H 13       | 1.649210810661316  | 99   | 1.660463571548462  | 100  | +1%        |  0.1175 |
 | TPC-H 14       | 32.8909797668457   | 1974 | 33.692501068115234 | 2022 | +2%        |  0.0000 |
 | TPC-H 15       | 64.49054718017578  | 3870 | 63.0474853515625   | 3783 | -2%        |  0.0000 |
+| TPC-H 16       | 5.088113784790039  | 306  | 5.538172245025635  | 333  | +9%        |  0.0000 |
 | TPC-H 17       | 2.8013389110565186 | 169  | 2.8208167552948    | 170  | +1%        |  0.1436 |
 | TPC-H 18       | 0.8350475430488586 | 51   | 0.7954840064048767 | 48   | -5%        |  0.0087 |
 | TPC-H 19       | 5.26102876663208   | 316  | 5.264037609100342  | 316  | +0%        |  0.9287 |
 | TPC-H 20       | 16.5688419342041   | 995  | 17.25289535522461  | 1036 | +4%        |  0.0000 |
 | TPC-H 21       | 0.7997806072235107 | 49   | 0.7862756848335266 | 48   | -2%        |  0.1342 |
 | TPC-H 22       | 8.714345932006836  | 523  | 8.564682006835938  | 514  | -2%        |  0.0000 |
 | geometric mean |                    |      |                    |      | +1%        |         |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 +----------------+--------------------+------+--------------------+------+------------+----------------------+
 | Benchmark      | prev. iter/s       | runs | new iter/s         | runs | change [%] |              p-value |
 +----------------+--------------------+------+--------------------+------+------------+----------------------+
 | TPC-H 01       | 0.8166056871414185 | 49   | 0.8332337737083435 | 50   | +2%        |               0.3839 |
+| TPC-H 02       | 23.73128318786621  | 1424 | 30.18152618408203  | 1811 | +27%       | (run time too short) |
 | TPC-H 03       | 4.8996992111206055 | 294  | 4.933220863342285  | 296  | +1%        | (run time too short) |
 | TPC-H 04       | 3.933244228363037  | 236  | 3.9166526794433594 | 235  | -0%        | (run time too short) |
 | TPC-H 05       | 3.583219289779663  | 215  | 3.516610860824585  | 211  | -2%        | (run time too short) |
 | TPC-H 06       | 85.29890441894531  | 5118 | 84.07007598876953  | 5045 | -1%        | (run time too short) |
 | TPC-H 07       | 5.399720668792725  | 324  | 5.415759563446045  | 325  | +0%        | (run time too short) |
 | TPC-H 08       | 4.199488162994385  | 252  | 4.282773017883301  | 257  | +2%        | (run time too short) |
 | TPC-H 09       | 1.3164951801300049 | 79   | 1.316494345664978  | 79   | -0%        |               0.8929 |
 | TPC-H 10       | 1.6331889629364014 | 98   | 1.616569995880127  | 97   | -1%        |               0.1575 |
 | TPC-H 11       | 12.448585510253906 | 747  | 12.464641571044922 | 748  | +0%        | (run time too short) |
 | TPC-H 12       | 11.466364860534668 | 688  | 11.416545867919922 | 685  | -0%        | (run time too short) |
 | TPC-H 13       | 2.766602039337158  | 166  | 2.6830201148986816 | 161  | -3%        |               0.0001 |
 | TPC-H 14       | 20.614057540893555 | 1237 | 19.981739044189453 | 1199 | -3%        | (run time too short) |
-| TPC-H 15       | 31.14808464050293  | 1869 | 28.795320510864258 | 1728 | -8%        | (run time too short) |
+| TPC-H 16       | 4.099714756011963  | 246  | 4.41655969619751   | 265  | +8%        | (run time too short) |
 | TPC-H 17       | 6.116011619567871  | 367  | 6.182864189147949  | 371  | +1%        | (run time too short) |
 | TPC-H 18       | 0.7665697932243347 | 46   | 0.7999053597450256 | 48   | +4%        | (run time too short) |
 | TPC-H 19       | 6.898996353149414  | 414  | 6.865633964538574  | 412  | -0%        | (run time too short) |
 | TPC-H 20       | 12.916570663452148 | 775  | 12.899374961853027 | 774  | -0%        | (run time too short) |
 | TPC-H 21       | 1.2498197555541992 | 75   | 1.2331945896148682 | 74   | -1%        | (run time too short) |
 | TPC-H 22       | 13.382573127746582 | 803  | 13.131560325622559 | 788  | -2%        | (run time too short) |
 | geometric mean |                    |      |                    |      | +1%        |                      |
 +----------------+--------------------+------+--------------------+------+------------+----------------------+

One can quite clearly see that the performance in the order-heavy TPC-H queries 2 and 16 increased – which makes sense, since those are the TPC-H queries with the largest amount of ORDER BY conditions (both have 4 ordering criteria).

Furthermore, the overall mean performance (see geomertric mean at the bottom of the performance diffs) seemed to increase slightly.

Interestingly, the performance decreased noticeably in one (different) TPC-H query per run (but not always the same one). It decreased, per benchmarking run, either in TPC-H 12, TPC-H 15, or in TPC-H 18.

bakoe commented 4 years ago

TPC-DS performance comparison of the master branch state (d257150) and this PR's current state (d675ceb) are as follows:

 +----------------+--------------------+------+--------------------+------+------------+---------+
 | Benchmark      | prev. iter/s       | runs | new iter/s         | runs | change [%] | p-value |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 | 01             | 11.795251846313477 | 708  | 11.355052947998047 | 682  | -4%        |  0.0000 |
 | 03             | 16.608789443969727 | 997  | 17.129695892333984 | 1028 | +3%        |  0.0000 |
+| 06             | 13.184954643249512 | 792  | 16.931264877319336 | 1017 | +28%       |  0.0000 |
 | 07             | 5.281435012817383  | 317  | 5.257420539855957  | 316  | -0%        |  0.2427 |
 | 09             | 3.9666378498077393 | 238  | 3.9769606590270996 | 239  | +0%        |  0.3187 |
-| 10             | 8.815311431884766  | 529  | 7.4268999099731445 | 446  | -16%       |  0.0000 |
 | 13             | 8.256613731384277  | 496  | 8.396881103515625  | 504  | +2%        |  0.1547 |
+| 15             | 26.107589721679688 | 1567 | 28.38237190246582  | 1703 | +9%        |  0.0000 |
+| 17             | 8.047514915466309  | 483  | 9.702763557434082  | 583  | +21%       |  0.0000 |
+| 19             | 19.471820831298828 | 1169 | 23.667558670043945 | 1421 | +22%       |  0.0000 |
+| 25             | 13.753172874450684 | 826  | 14.50787353515625  | 871  | +5%        |  0.0000 |
 | 26             | 10.479242324829102 | 629  | 10.247549057006836 | 615  | -2%        |  0.0000 |
 | 28             | 10.12134075164795  | 608  | 10.177525520324707 | 611  | +1%        |  0.0000 |
 | 29             | 8.37825870513916   | 503  | 7.960818767547607  | 478  | -5%        |  0.0000 |
+| 31             | 2.816605806350708  | 170  | 2.9621970653533936 | 178  | +5%        |  0.0000 |
+| 34             | 15.111749649047852 | 907  | 16.37285041809082  | 983  | +8%        |  0.0000 |
+| 35             | 2.915496826171875  | 175  | 3.157623291015625  | 190  | +8%        |  0.0000 |
-| 39a            | 1.8884050846099854 | 114  | 1.7054455280303955 | 103  | -10%       |  0.0000 |
 | 39b            | 1.846909999847412  | 111  | 1.8303683996200562 | 110  | -1%        |  0.5529 |
 | 41             | 27.981056213378906 | 1679 | 27.97555160522461  | 1679 | -0%        |  0.8864 |
-| 42             | 25.565645217895508 | 1534 | 23.413909912109375 | 1405 | -8%        |  0.0000 |
 | 43             | 1.9743539094924927 | 119  | 1.954245686531067  | 118  | -1%        |  0.0180 |
 | 45             | 34.42607116699219  | 2066 | 34.756072998046875 | 2086 | +1%        |  0.0000 |
 | 48             | 4.466436862945557  | 268  | 4.471993446350098  | 269  | +0%        |  0.7857 |
+| 50             | 16.914020538330078 | 1015 | 18.004314422607422 | 1081 | +6%        |  0.0000 |
 | 52             | 23.022422790527344 | 1382 | 23.927642822265625 | 1436 | +4%        |  0.0000 |
 | 55             | 26.053987503051758 | 1564 | 26.874469757080078 | 1613 | +3%        |  0.0000 |
 | 62             | 6.062280654907227  | 364  | 6.010075092315674  | 361  | -1%        |  0.0000 |
 | 65             | 4.699222087860107  | 283  | 4.686134338378906  | 282  | -0%        |  0.5330 |
 | 69             | 7.238651752471924  | 435  | 7.3568291664123535 | 442  | +2%        |  0.0013 |
 | 73             | 15.708356857299805 | 943  | 15.098516464233398 | 906  | -4%        |  0.0000 |
+| 79             | 7.192746639251709  | 432  | 7.837674617767334  | 471  | +9%        |  0.0000 |
+| 81             | 18.419893264770508 | 1106 | 22.763187408447266 | 1366 | +24%       |  0.0000 |
 | 83             | 36.578025817871094 | 2195 | 36.29519271850586  | 2178 | -1%        |  0.0000 |
 | 85             | 7.454848766326904  | 448  | 7.411296844482422  | 445  | -1%        |  0.8316 |
+| 88             | 3.3026669025421143 | 199  | 3.619063138961792  | 218  | +10%       |  0.0000 |
-| 91             | 31.038206100463867 | 1863 | 28.769859313964844 | 1727 | -7%        |  0.0000 |
 | 93             | 2.8145997524261475 | 169  | 2.827727794647217  | 170  | +0%        |  0.4361 |
 | 96             | 30.092435836791992 | 1807 | 31.521039962768555 | 1892 | +5%        |  0.0000 |
 | 97             | 1.1559638977050781 | 70   | 1.1571468114852905 | 70   | +0%        |  0.9118 |
 | 99             | 3.0270228385925293 | 182  | 2.9690635204315186 | 179  | -2%        |  0.0000 |
 | geometric mean |                    |      |                    |      | +2%        |         |
 +----------------+--------------------+------+--------------------+------+------------+---------+
bakoe commented 4 years ago

TPC-H performance comparison of the master branch (d2571500dc039dd0e7a483ec2ea6e43b5b381f9d, ran on 2020/02/04) and this PR's current state (34c412fe03ea1545b7b8272f7e40bde4fa9e3300, ran on 2020/02/04):

 +----------------+--------------------+------+--------------------+------+------------+---------+
 | Benchmark      | prev. iter/s       | runs | new iter/s         | runs | change [%] | p-value |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 | TPC-H 01       | 0.6839942932128906 | 42   | 0.6845481395721436 | 42   | +0%        |  0.8517 |
+| TPC-H 02       | 42.264156341552734 | 2536 | 51.48957824707031  | 3090 | +22%       |  0.0000 |
 | TPC-H 03       | 4.846597194671631  | 291  | 4.790519714355469  | 288  | -1%        |  0.0117 |
 | TPC-H 04       | 4.3741912841796875 | 263  | 4.285599708557129  | 258  | -2%        |  0.0083 |
 | TPC-H 05       | 2.389423370361328  | 144  | 2.4488184452056885 | 147  | +2%        |  0.0303 |
 | TPC-H 06       | 119.57794952392578 | 7175 | 121.98258209228516 | 7319 | +2%        |  0.0000 |
 | TPC-H 07       | 5.295278072357178  | 318  | 5.3925323486328125 | 324  | +2%        |  0.0096 |
+| TPC-H 08       | 3.5397117137908936 | 213  | 3.8330276012420654 | 230  | +8%        |  0.0000 |
 | TPC-H 09       | 1.2127244472503662 | 73   | 1.2232056856155396 | 74   | +1%        |  0.1645 |
 | TPC-H 10       | 2.0098278522491455 | 121  | 2.0111286640167236 | 121  | +0%        |  0.9479 |
 | TPC-H 11       | 18.649887084960938 | 1119 | 19.1567325592041   | 1150 | +3%        |  0.0000 |
 | TPC-H 12       | 9.837431907653809  | 591  | 10.014595031738281 | 601  | +2%        |  0.0000 |
+| TPC-H 13       | 1.1097503900527954 | 67   | 1.576829433441162  | 95   | +42%       |  0.0000 |
+| TPC-H 14       | 30.525157928466797 | 1832 | 33.388301849365234 | 2004 | +9%        |  0.0000 |
 | TPC-H 15       | 64.04070281982422  | 3843 | 64.77598571777344  | 3887 | +1%        |  0.0000 |
+| TPC-H 16       | 5.063379764556885  | 304  | 5.85122537612915   | 352  | +16%       |  0.0000 |
 | TPC-H 17       | 2.912839412689209  | 175  | 2.858480215072632  | 172  | -2%        |  0.0003 |
 | TPC-H 18       | 0.8326975703239441 | 50   | 0.7973725199699402 | 48   | -4%        |  0.0799 |
 | TPC-H 19       | 5.010912895202637  | 302  | 5.11836576461792   | 308  | +2%        |  0.1050 |
+| TPC-H 20       | 9.79638671875      | 588  | 17.357471466064453 | 1042 | +77%       |  0.0000 |
 | TPC-H 21       | 0.789459228515625  | 48   | 0.8048768639564514 | 49   | +2%        |  0.2019 |
 | TPC-H 22       | 8.544171333312988  | 513  | 8.898557662963867  | 534  | +4%        |  0.0000 |
+| geometric mean |                    |      |                    |      | +7%        |         |
 +----------------+--------------------+------+--------------------+------+------------+---------+

With parallelization enabled (--cores 8 --scheduler --clients=4):

 +----------------+--------------------+-------+--------------------+-------+------------+---------+
 | Benchmark      | prev. iter/s       | runs  | new iter/s         | runs  | change [%] | p-value |
 +----------------+--------------------+-------+--------------------+-------+------------+---------+
 | TPC-H 01       | 3.132821798324585  | 188   | 3.1832149028778076 | 191   | +2%        |  0.6905 |
+| TPC-H 02       | 116.67940521240234 | 7001  | 132.5027313232422  | 7951  | +14%       |  0.0000 |
+| TPC-H 03       | 14.849773406982422 | 891   | 16.24746322631836  | 975   | +9%        |  0.0000 |
 | TPC-H 04       | 12.365447998046875 | 742   | 12.966375350952148 | 778   | +5%        |  0.0155 |
 | TPC-H 05       | 9.716527938842773  | 583   | 9.565568923950195  | 574   | -2%        |  0.6878 |
 | TPC-H 06       | 385.6679992675781  | 23142 | 386.3264465332031  | 23183 | +0%        |  0.0000 |
 | TPC-H 07       | 17.947097778320312 | 1077  | 17.699832916259766 | 1062  | -1%        |  0.4449 |
+| TPC-H 08       | 12.531888961791992 | 752   | 13.315630912780762 | 799   | +6%        |  0.0072 |
 | TPC-H 09       | 4.066176891326904  | 244   | 4.233008861541748  | 254   | +4%        |  0.2054 |
+| TPC-H 10       | 5.53258752822876   | 332   | 5.883141040802002  | 353   | +6%        |  0.0364 |
 | TPC-H 11       | 40.015506744384766 | 2401  | 40.41328811645508  | 2425  | +1%        |  0.2720 |
 | TPC-H 12       | 35.578269958496094 | 2135  | 36.17770767211914  | 2171  | +2%        |  0.1890 |
+| TPC-H 13       | 6.882422924041748  | 413   | 7.43264627456665   | 446   | +8%        |  0.0197 |
 | TPC-H 14       | 67.68167877197266  | 4061  | 67.90689086914062  | 4075  | +0%        |  0.6384 |
 | TPC-H 15       | 171.3638458251953  | 10283 | 169.5092315673828  | 10171 | -1%        |  0.0004 |
+| TPC-H 16       | 13.932799339294434 | 836   | 15.931422233581543 | 956   | +14%       |  0.0000 |
 | TPC-H 17       | 14.015767097473145 | 841   | 14.298567771911621 | 858   | +2%        |  0.3627 |
 | TPC-H 18       | 2.133197546005249  | 128   | 2.149927854537964  | 129   | +1%        |  0.9865 |
 | TPC-H 19       | 20.16547393798828  | 1210  | 20.04993438720703  | 1203  | -1%        |  0.7291 |
 | TPC-H 20       | 48.383140563964844 | 2903  | 49.54710006713867  | 2973  | +2%        |  0.0361 |
-| TPC-H 21       | 3.416090726852417  | 205   | 3.21634840965271   | 193   | -6%        |  0.2094 |
 | TPC-H 22       | 37.530155181884766 | 2252  | 38.13066101074219  | 2288  | +2%        |  0.2348 |
 | geometric mean |                    |       |                    |       | +3%        |         |
 +----------------+--------------------+-------+--------------------+-------+------------+---------+
bakoe commented 4 years ago

TPC-DS performance comparison of the master branch state (d257150) and this PR's current state (34c412f) are as follows:

 +----------------+--------------------+------+--------------------+------+------------+---------+
 | Benchmark      | prev. iter/s       | runs | new iter/s         | runs | change [%] | p-value |
 +----------------+--------------------+------+--------------------+------+------------+---------+
 | 01             | 11.795251846313477 | 708  | 11.488670349121094 | 690  | -3%        |  0.0000 |
 | 03             | 16.608789443969727 | 997  | 16.768413543701172 | 1007 | +1%        |  0.1093 |
+| 06             | 13.184954643249512 | 792  | 14.366501808166504 | 862  | +9%        |  0.0000 |
 | 07             | 5.281435012817383  | 317  | 5.32116174697876   | 320  | +1%        |  0.0303 |
 | 09             | 3.9666378498077393 | 238  | 3.902395725250244  | 235  | -2%        |  0.0000 |
-| 10             | 8.815311431884766  | 529  | 8.060312271118164  | 484  | -9%        |  0.0000 |
 | 13             | 8.256613731384277  | 496  | 8.389236450195312  | 504  | +2%        |  0.1742 |
+| 15             | 26.107589721679688 | 1567 | 27.92925262451172  | 1676 | +7%        |  0.0000 |
+| 17             | 8.047514915466309  | 483  | 9.979583740234375  | 599  | +24%       |  0.0000 |
 | 19             | 19.471820831298828 | 1169 | 20.169034957885742 | 1211 | +4%        |  0.0000 |
-| 25             | 13.753172874450684 | 826  | 12.861227989196777 | 772  | -6%        |  0.0000 |
 | 26             | 10.479242324829102 | 629  | 10.545248985290527 | 633  | +1%        |  0.0001 |
 | 28             | 10.12134075164795  | 608  | 10.091062545776367 | 606  | -0%        |  0.0257 |
 | 29             | 8.37825870513916   | 503  | 8.007177352905273  | 481  | -4%        |  0.0000 |
 | 31             | 2.816605806350708  | 170  | 2.8334856033325195 | 171  | +1%        |  0.2094 |
+| 34             | 15.111749649047852 | 907  | 15.975574493408203 | 959  | +6%        |  0.0000 |
+| 35             | 2.915496826171875  | 175  | 3.2120261192321777 | 193  | +10%       |  0.0000 |
-| 39a            | 1.8884050846099854 | 114  | 1.768853783607483  | 107  | -6%        |  0.0000 |
+| 39b            | 1.846909999847412  | 111  | 1.9806188344955444 | 119  | +7%        |  0.0000 |
-| 41             | 27.981056213378906 | 1679 | 26.411283493041992 | 1585 | -6%        |  0.0000 |
 | 42             | 25.565645217895508 | 1534 | 24.344820022583008 | 1461 | -5%        |  0.0000 |
 | 43             | 1.9743539094924927 | 119  | 1.9455996751785278 | 117  | -1%        |  0.0066 |
 | 45             | 34.42607116699219  | 2066 | 33.233055114746094 | 1994 | -3%        |  0.0000 |
 | 48             | 4.466436862945557  | 268  | 4.465438365936279  | 268  | -0%        |  0.9593 |
 | 50             | 16.914020538330078 | 1015 | 16.755657196044922 | 1006 | -1%        |  0.0981 |
 | 52             | 23.022422790527344 | 1382 | 23.1141357421875   | 1387 | +0%        |  0.6236 |
 | 55             | 26.053987503051758 | 1564 | 26.928974151611328 | 1616 | +3%        |  0.0000 |
 | 62             | 6.062280654907227  | 364  | 5.9949727058410645 | 360  | -1%        |  0.0000 |
 | 65             | 4.699222087860107  | 283  | 4.628204345703125  | 278  | -2%        |  0.0037 |
+| 69             | 7.238651752471924  | 435  | 7.619361400604248  | 458  | +5%        |  0.0000 |
 | 73             | 15.708356857299805 | 943  | 15.904525756835938 | 955  | +1%        |  0.0001 |
+| 79             | 7.192746639251709  | 432  | 7.670797824859619  | 461  | +7%        |  0.0000 |
+| 81             | 18.419893264770508 | 1106 | 23.251218795776367 | 1396 | +26%       |  0.0000 |
 | 83             | 36.578025817871094 | 2195 | 35.72597885131836  | 2144 | -2%        |  0.0000 |
 | 85             | 7.454848766326904  | 448  | 7.3310041427612305 | 440  | -2%        |  0.5404 |
+| 88             | 3.3026669025421143 | 199  | 3.60959529876709   | 217  | +9%        |  0.0000 |
 | 91             | 31.038206100463867 | 1863 | 30.571197509765625 | 1835 | -2%        |  0.0000 |
 | 93             | 2.8145997524261475 | 169  | 2.937558174133301  | 177  | +4%        |  0.0000 |
+| 96             | 30.092435836791992 | 1807 | 31.87918472290039  | 1914 | +6%        |  0.0000 |
 | 97             | 1.1559638977050781 | 70   | 1.1658796072006226 | 70   | +1%        |  0.3615 |
 | 99             | 3.0270228385925293 | 182  | 2.9930217266082764 | 180  | -1%        |  0.0001 |
 | geometric mean |                    |      |                    |      | +2%        |         |
 +----------------+--------------------+------+--------------------+------+------------+---------+