vmware-archive / database-stream-processor

Streaming and Incremental Computation Framework
Other
225 stars 20 forks source link

Add Postgres support #346

Closed lalithsuresh closed 1 year ago

lalithsuresh commented 1 year ago

This patch includes:

codecov[bot] commented 1 year ago

Codecov Report

Merging #346 (2ad1daa) into main (832ce0c) will decrease coverage by 6.14%. The diff coverage is 60.05%.

Additional details and impacted files [![Impacted file tree graph](https://codecov.io/gh/vmware/database-stream-processor/pull/346/graphs/tree.svg?width=650&height=150&src=pr&token=0wZcmD11gt&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) ```diff @@ Coverage Diff @@ ## main #346 +/- ## ========================================== - Coverage 79.37% 73.24% -6.14% ========================================== Files 167 227 +60 Lines 33521 46249 +12728 ========================================== + Hits 26608 33876 +7268 - Misses 6913 12373 +5460 ``` | [Impacted Files](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) | Coverage Δ | | |---|---|---| | [crates/dataflow-jit/src/dataflow/mod.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2RhdGFmbG93LWppdC9zcmMvZGF0YWZsb3cvbW9kLnJz) | `60.51% <ø> (ø)` | | | [crates/dbsp/src/lib.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL2xpYi5ycw==) | `100.00% <ø> (ø)` | | | [crates/dbsp/src/operator/time\_series/range.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL29wZXJhdG9yL3RpbWVfc2VyaWVzL3JhbmdlLnJz) | `78.28% <0.00%> (-2.07%)` | :arrow_down: | | [crates/dbsp/src/time/nested\_ts32.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RpbWUvbmVzdGVkX3RzMzIucnM=) | `72.52% <ø> (+1.09%)` | :arrow_up: | | [crates/dbsp/src/trace/layers/erased/mod.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy9lcmFzZWQvbW9kLnJz) | `0.00% <0.00%> (ø)` | | | [crates/dbsp/src/trace/layers/erased/vtable.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy9lcmFzZWQvdnRhYmxlLnJz) | `18.66% <ø> (ø)` | | | [crates/dbsp/src/trace/layers/mod.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy9tb2QucnM=) | `35.71% <ø> (ø)` | | | [crates/dbsp/src/trace/layers/ordered/mod.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy9vcmRlcmVkL21vZC5ycw==) | `74.08% <ø> (ø)` | | | [crates/dbsp/src/trace/layers/test.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy90ZXN0LnJz) | `100.00% <ø> (ø)` | | | [crates/dbsp/src/trace/ord/zset\_batch.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL29yZC96c2V0X2JhdGNoLnJz) | `80.32% <ø> (+7.16%)` | :arrow_up: | | ... and [60 more](https://codecov.io/gh/vmware/database-stream-processor/pull/346?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) | | ... and [18 files with indirect coverage changes](https://codecov.io/gh/vmware/database-stream-processor/pull/346/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)
github-actions[bot] commented 1 year ago

Benchmark results

Nexmark

name main~69 [kOp/s] PR [kOp/s] Tput change [%] Assessment Peak RSS diff
q0 5478.79 5423.49 -1 :heavy_check_mark: -8.8 MB
q1 5398.75 5472.79 1 :heavy_check_mark: 16.1 MB
q2 5630.96 5093.66 -10 :small_red_triangle_down: 5.8 MB
q3 5504.96 5516.72 0 :heavy_check_mark: -5.5 MB
q4 3341.37 3412.43 2 :heavy_check_mark: -17.7 MB
q5 5543.82 5425.88 -2 :heavy_check_mark: -17.7 MB
q6 3697.7 3576.17 -3 :heavy_check_mark: 12.5 MB
q7 2908 4037.55 39 :evergreen_tree: -4.0 GB
q8 5260.06 5216.27 -1 :heavy_check_mark: -4.0 GB
q9 627.889 593.382 -5 :small_red_triangle_down: -51.1 MB
q12 4913.17 4887.92 -1 :heavy_check_mark: -51.1 MB
q13 2697.28 3488.47 29 :evergreen_tree: -51.1 MB
q14 5659.87 5426.57 -4 :heavy_check_mark: -51.1 MB
q15 5302.63 5316.23 0 :heavy_check_mark: -51.1 MB
q16 1038.06 1041.78 0 :heavy_check_mark: -51.1 MB
q17 3140.32 3058.65 -3 :heavy_check_mark: -51.1 MB
q18 871.812 872.995 0 :heavy_check_mark: -48.4 MB
q19 1025.09 1033.82 1 :heavy_check_mark: -48.4 MB
q20 1069.96 1057.46 -1 :heavy_check_mark: -48.4 MB
q21 5468.41 5329.35 -3 :heavy_check_mark: -48.4 MB
q22 5234.71 5502.03 5 :evergreen_tree: -48.4 MB

Galen

name main~69 [s] PR [s] Runtime change [%] Assessment
galen 27.7409 28.3286 2 :heavy_check_mark:

LDBC

algorithm dataset threads main~69 [kEVPS] PR [kEVPS] Tput change [%] Assessment Peak RSS diff
bfs graph500-22 1 1755.93 1758.48 0 :heavy_check_mark: 32.8 kB
bfs datagen-8_4-fb 6 7705.23 7860.25 2 :heavy_check_mark: 116.5 MB
pagerank graph500-22 1 688.825 687.847 0 :heavy_check_mark: 122.9 kB
pagerank datagen-8_4-fb 6 2050.74 2026.43 -1 :heavy_check_mark: 203.9 MB

Nexmark (with Persistence)

name main~69 [kOp/s] PR [kOp/s] Tput change [%] PR DRAM [kOp/s] DRAM diff [%] Assessment
q0 2374.54 2400.44 1 2405.05 0 :heavy_check_mark:
q1 1682.87 1680.01 0 1722.01 -2 :heavy_check_mark:
q2 2357.63 2427.89 3 2394.18 1 :heavy_check_mark:
q3 1989.03 2009.91 1 2301.37 -13 :heavy_check_mark:
q4 360.36 365.522 1 1386.66 -74 :heavy_check_mark:
q5 2006.19 2006.74 0 2284.6 -12 :heavy_check_mark:
q6 333.035 340.373 2 1398.54 -76 :heavy_check_mark:
q7 258.051 254.223 -1 1270.78 -80 :heavy_check_mark:
q8 2067 2113.83 2 2227.09 -5 :heavy_check_mark:
q9 32.9563 33.3701 1 388.776 -91 :heavy_check_mark:
q12 829.501 848.793 2 1798.87 -53 :heavy_check_mark:
q13 440.688 443.307 1 983.104 -55 :heavy_check_mark:
q14 1701.72 1729.34 2 1707.84 1 :heavy_check_mark:
q15 196.648 196.718 0 1189.38 -83 :heavy_check_mark:
q16 26.8666 27.1151 1 288.721 -91 :heavy_check_mark:
q17 80.9823 80.9857 0 800.009 -90 :heavy_check_mark:
q18 60.4819 61.3366 1 803.282 -92 :heavy_check_mark:
q19 67.9337 68.4015 1 660.944 -90 :heavy_check_mark:
q20 169.217 168.369 -1 941.201 -82 :heavy_check_mark:
q21 1479.42 1533.24 4 1506.66 2 :heavy_check_mark:
q22 2034.39 2016.39 -1 2018.11 0 :heavy_check_mark: