pingcap / go-ycsb

A Go port of Yahoo! Cloud Serving Benchmark (YCSB)
Apache License 2.0
602 stars 244 forks source link

support batch for mysql #211

Closed aylei closed 2 years ago

aylei commented 2 years ago

This PR add batch support for mysql, which was motivated by using batch insert to accelerate the load process during benchmarks.

This commit was tested in benchmarks with 3 billions rows. Also, I did some smoke tests locally:

> tiup playground

> ./bin/go-ycsb load mysql -P workloads/workloada -p mysql.host=127.0.0.1 -p mysql.port=4000
***************** properties *****************
"readproportion"="0.5"
"batch.size"="50"
"recordcount"="100000"
"dotransactions"="false"
"scanproportion"="0"
"workload"="core"
"operationcount"="100000"
"mysql.host"="127.0.0.1"
"updateproportion"="0.5"
"insertproportion"="0"
"requestdistribution"="uniform"
"readallfields"="true"
"mysql.port"="4000"
**********************************************
BATCH_INSERT - Takes(s): 9.8, Count: 1807, OPS: 183.7, Avg(us): 4842, Min(us): 4011, Max(us): 63910, 99th(us): 8000, 99.9th(us): 33000, 99.99th(us): 57000
Run finished, takes 17.490709788s
BATCH_INSERT - Takes(s): 17.3, Count: 2000, OPS: 115.4, Avg(us): 4827, Min(us): 4011, Max(us): 63910, 99th(us): 8000, 99.9th(us): 33000, 99.99th(us): 64000

> ./bin/go-ycsb run mysql -P workloads/workloada -p mysql.host=127.0.0.1 -p mysql.port=4000
***************** properties *****************
"requestdistribution"="uniform"
"workload"="core"
"scanproportion"="0"
"updateproportion"="0.5"
"dotransactions"="true"
"operationcount"="100000"
"readallfields"="true"
"recordcount"="100000"
"batch.size"="50"
"insertproportion"="0"
"mysql.port"="4000"
"readproportion"="0.5"
"mysql.host"="127.0.0.1"
**********************************************
BATCH_READ - Takes(s): 9.9, Count: 215, OPS: 21.7, Avg(us): 1709, Min(us): 1404, Max(us): 3159, 99th(us): 3000, 99.9th(us): 4000, 99.99th(us): 4000
BATCH_UPDATE - Takes(s): 9.9, Count: 148, OPS: 14.9, Avg(us): 64719, Min(us): 60583, Max(us): 117298, 99th(us): 116000, 99.9th(us): 118000, 99.99th(us): 118000
........

Signed-off-by: Aylei rayingecho@gmail.com

dbsid commented 2 years ago

what's the default value of batch.size? can we keep the default behaviors consistent with previous version?

aylei commented 2 years ago

what's the default value of batch.size?

No batch by default.

can we keep the default behaviors consistent with previous version?

Yes, the default behavior would be consistent

dbsid commented 2 years ago

/lgtm