search

vmware-archive / database-stream-processor

Streaming and Incremental Computation Framework
Other
222 stars 20 forks source link

benchmark: Add scripts and instructions for DBSP, Flink, and Apache Beam #361

Closed blp closed 1 year ago

blp commented 1 year ago

This makes it easier to benchmark them in a way that I intend to be fair.

codecov[bot] commented 1 year ago

Codecov Report

Merging #361 (6eeff02) into main (a93112b) will increase coverage by 0.00%. The diff coverage is n/a.

:exclamation: Current head 6eeff02 differs from pull request most recent head 735f207. Consider uploading reports for the commit 735f207 to get more accurate results

Additional details and impacted files [![Impacted file tree graph](https://codecov.io/gh/vmware/database-stream-processor/pull/361/graphs/tree.svg?width=650&height=150&src=pr&token=0wZcmD11gt&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)](https://codecov.io/gh/vmware/database-stream-processor/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) ```diff @@ Coverage Diff @@ ## main #361 +/- ## ======================================= Coverage 71.74% 71.75% ======================================= Files 234 234 Lines 48834 48835 +1 ======================================= + Hits 35038 35043 +5 + Misses 13796 13792 -4 ``` [see 5 files with indirect coverage changes](https://codecov.io/gh/vmware/database-stream-processor/pull/361/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)
github-actions[bot] commented 1 year ago

Benchmark results

Nexmark

name main~3 [kOp/s] PR [kOp/s] Tput change [%] Assessment Peak RSS diff
q0 5633 5581.44 -1 :heavy_check_mark: 10.0 MB
q1 5671.58 5640.26 -1 :heavy_check_mark: -8.9 MB
q2 5638.51 4501.03 -20 :small_red_triangle_down: 5.2 MB
q3 5594.08 5652.87 1 :heavy_check_mark: -18.6 MB
q4 4975.4 4899.09 -2 :heavy_check_mark: 17.2 MB
q5 5560.18 5538.04 0 :heavy_check_mark: 17.2 MB
q6 4852.51 4363.55 -10 :small_red_triangle_down: 270.0 MB
q7 3009.65 4122.54 37 :evergreen_tree: -2.6 GB
q8 5435.96 5295.93 -3 :heavy_check_mark: -2.6 GB
q9 850.825 859.726 1 :heavy_check_mark: -410.1 MB
q12 4168.13 5162.57 24 :evergreen_tree: -410.1 MB
q13 2781.47 3426.93 23 :evergreen_tree: -410.1 MB
q14 5307.96 5473.39 3 :heavy_check_mark: -410.1 MB
q15 5235.55 5222.67 0 :heavy_check_mark: -410.1 MB
q16 1040.82 1043.89 0 :heavy_check_mark: -410.1 MB
q17 3103.07 3050.77 -2 :heavy_check_mark: -319.1 MB
q18 1445.38 1455.47 1 :heavy_check_mark: -202.4 MB
q19 1437.08 1431.89 0 :heavy_check_mark: -202.4 MB
q20 1525.8 1537.78 1 :heavy_check_mark: -202.4 MB
q21 5177.17 4670.8 -10 :small_red_triangle_down: -202.4 MB
q22 5663.55 4480.88 -21 :small_red_triangle_down: -202.4 MB

Galen

name main~3 [s] PR [s] Runtime change [%] Assessment
galen 28.5243 28.6244 0 :heavy_check_mark:

LDBC

algorithm dataset threads main~3 [kEVPS] PR [kEVPS] Tput change [%] Assessment Peak RSS diff
bfs graph500-22 1 1765.79 1759.38 0 :heavy_check_mark: 20.5 kB
bfs datagen-8_4-fb 6 7584.25 7527.37 -1 :heavy_check_mark: -38.2 MB
pagerank graph500-22 1 688.337 685.254 0 :heavy_check_mark: 237.6 kB
pagerank datagen-8_4-fb 6 1864.02 2035.33 9 :evergreen_tree: -158.4 MB

Nexmark (with Persistence)

name main~3 [kOp/s] PR [kOp/s] Tput change [%] PR DRAM [kOp/s] DRAM diff [%] Assessment
q0 2496.45 2435.08 -2 2450.48 -1 :heavy_check_mark:
q1 1754.03 1732.8 -1 1742.69 -1 :heavy_check_mark:
q2 2453.72 2453.34 0 2441.1 1 :heavy_check_mark:
q3 2052.56 2066.28 1 2286.92 -10 :heavy_check_mark:
q4 362.861 358.763 -1 1404.51 -74 :heavy_check_mark:
q5 2059.54 2051.06 0 2328.59 -12 :heavy_check_mark:
q6 328.975 334.647 2 1397.16 -76 :heavy_check_mark:
q7 618.423 615.696 0 1287.68 -52 :heavy_check_mark:
q8 2190.4 2188.9 0 2302.28 -5 :heavy_check_mark:
q9 79.1681 79.3796 0 385.906 -79 :heavy_check_mark:
q12 858.709 854.275 -1 1845.83 -54 :heavy_check_mark:
q13 434.283 442.586 2 996.728 -56 :heavy_check_mark:
q14 1723.78 1706.25 -1 1737.33 -2 :heavy_check_mark:
q15 195.631 197.852 1 1196.1 -83 :heavy_check_mark:
q16 25.6457 25.6498 0 285.967 -91 :heavy_check_mark:
q17 80.7856 80.6873 0 787.338 -90 :heavy_check_mark:
q18 126.207 124.429 -1 801.258 -84 :heavy_check_mark:
q19 180.901 182.962 1 672.259 -73 :heavy_check_mark:
q20 505.978 503.421 -1 955.027 -47 :heavy_check_mark:
q21 1530.46 1530.95 0 1533.86 0 :heavy_check_mark:
q22 2044.03 1966.27 -4 2035.89 -3 :heavy_check_mark:
blp commented 1 year ago

I pushed a revision that rebases against main and adds some automation.