search

cmu-db / noisepage

Self-Driving Database Management System from Carnegie Mellon University
https://noise.page
MIT License
1.74k stars 502 forks source link

Switch to the new version of OLTPBench. #1632

Open lmwnshn opened 3 years ago

lmwnshn commented 3 years ago

Heading

Switch to the new version of OLTPBench.

Description

Blocked on merging:

Does NOT need OLTPBench itself to merge oltpbench_tim into master.

noisepage-checks[bot] commented 3 years ago

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change) benchmark_type wal_device details
-6.16% tpcc RAM disk
Detailsmaster tps=22847.0, commit tps=21440.41, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-2.19% tpcc None
Detailsmaster tps=29796.25, commit tps=29142.8, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-1.94% tpcc HDD
Detailsmaster tps=21697.7, commit tps=21276.38, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
2.64% tatp RAM disk
Detailsmaster tps=6574.02, commit tps=6747.38, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-7.81% tatp None
Detailsmaster tps=7537.73, commit tps=6949.0, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
5.52% tatp HDD
Detailsmaster tps=6546.74, commit tps=6907.92, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32
noisepage-checks[bot] commented 3 years ago

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

lmwnshn commented 3 years ago

benchbase

noisepage-checks[bot] commented 3 years ago

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change) benchmark_type wal_device details
-54.65% tpcc RAM disk
Detailsmaster tps=22928.11, commit tps=10398.17, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-60.3% tpcc None
Detailsmaster tps=29712.44, commit tps=11794.9, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-57.62% tpcc HDD
Detailsmaster tps=21952.81, commit tps=9302.72, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-30.74% tatp RAM disk
Detailsmaster tps=6435.38, commit tps=4456.95, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-34.53% tatp None
Detailsmaster tps=7211.2, commit tps=4720.9, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-32.15% tatp HDD
Detailsmaster tps=6349.03, commit tps=4308.11, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32
codecov[bot] commented 3 years ago

Codecov Report

Merging #1632 (af0592c) into master (25b264e) will increase coverage by 0.01%. The diff coverage is n/a.

:exclamation: Current head af0592c differs from pull request most recent head 714253a. Consider uploading reports for the commit 714253a to get more accurate results Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1632      +/-   ##
==========================================
+ Coverage   81.11%   81.13%   +0.01%     
==========================================
  Files         761      761              
  Lines       55258    55258              
==========================================
+ Hits        44825    44835      +10     
+ Misses      10433    10423      -10
Impacted Files Coverage Δ
src/include/storage/block_access_controller.h 88.23% <0.00%> (-5.89%) :arrow_down:
src/execution/sql/vector.cpp 85.76% <0.00%> (-0.34%) :arrow_down:
src/include/storage/index/bplustree.h 90.98% <0.00%> (-0.11%) :arrow_down:
src/execution/sema/sema_builtin.cpp 60.54% <0.00%> (-0.09%) :arrow_down:
src/execution/vm/bytecode_generator.cpp 84.00% <0.00%> (+0.08%) :arrow_up:
src/parser/postgresparser.cpp 84.99% <0.00%> (+0.14%) :arrow_up:
src/storage/arrow_serializer.cpp 86.87% <0.00%> (+0.62%) :arrow_up:
src/execution/sema/sema_expr.cpp 77.27% <0.00%> (+1.13%) :arrow_up:
...self_driving/modeling/data/grouped_op_unit_data.py 46.85% <0.00%> (+1.14%) :arrow_up:
src/storage/index/hash_index.cpp 91.13% <0.00%> (+1.26%) :arrow_up:
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 25b264e...714253a. Read the comment docs.

noisepage-checks[bot] commented 3 years ago

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change) benchmark_type wal_device details
-51.43% tpcc RAM disk
Detailsmaster tps=23246.39, commit tps=11291.67, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-57.08% tpcc None
Detailsmaster tps=29097.48, commit tps=12488.65, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-54.54% tpcc HDD
Detailsmaster tps=21521.21, commit tps=9783.1, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-26.29% tatp RAM disk
Detailsmaster tps=6454.45, commit tps=4757.84, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-33.08% tatp None
Detailsmaster tps=7294.7, commit tps=4881.72, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-25.31% tatp HDD
Detailsmaster tps=6349.03, commit tps=4741.83, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32
noisepage-checks[bot] commented 3 years ago

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change) benchmark_type wal_device details
-10.48% tpcc RAM disk
Detailsmaster tps=23070.67, commit tps=20653.66, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-6.35% tpcc None
Detailsmaster tps=30175.57, commit tps=28259.21, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-5.71% tpcc HDD
Detailsmaster tps=21604.76, commit tps=20370.56, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-4.84% tatp RAM disk
Detailsmaster tps=6501.3, commit tps=6186.63, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-14.14% tatp None
Detailsmaster tps=7233.86, commit tps=6210.83, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-3.28% tatp HDD
Detailsmaster tps=6262.88, commit tps=6057.52, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32
noisepage-checks[bot] commented 3 years ago

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change) benchmark_type wal_device details
-9.47% tpcc RAM disk
Detailsmaster tps=23070.67, commit tps=20886.88, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-7.8% tpcc None
Detailsmaster tps=30175.57, commit tps=27821.55, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-7.08% tpcc HDD
Detailsmaster tps=21604.76, commit tps=20074.48, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-7.12% tatp RAM disk
Detailsmaster tps=6501.3, commit tps=6038.61, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-12.44% tatp None
Detailsmaster tps=7233.86, commit tps=6333.97, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.39% tatp HDD
Detailsmaster tps=6262.88, commit tps=6113.22, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32
lmwnshn commented 3 years ago

I don't foresee a risk of merge conflicts, so this can sit out here for a while. The plan is for me and @jkosh44 to poke at BenchBase next week to do a little light profiling work.