search

brianfrankcooper / YCSB

Yahoo! Cloud Serving Benchmark
Apache License 2.0
4.94k stars 2.24k forks source link

Release 0.14.0 #1117

Closed busbey closed 6 years ago

busbey commented 6 years ago

follow up from #981. There's already a fair bit of churn on master in the ~9 months since the 0.13.0-staging branch started.

Additionally, we have a good deal of testing work left over that never happened in the 0.13.0 RC process.

twblamer commented 6 years ago

Just curious, what testing work is needed? Is this a matter of manually testing the bindings, the individual features/PRs, or both?

busbey commented 6 years ago

yep! we'll need to test a number of bindings and any specific fixes that are supposed to be present.

The first step is to get a list up of which bindings / features /etc. It's on my todo list, but I've been a bit swamped on a different project lately.

busbey commented 6 years ago

generally speaking, testing a binding should be straight forward. It's essentially:

  1. Download RC and build binary bits
  2. stand up an instance of whatever backing datastore (cluster deploy optimal, but single-node is fine)
  3. follow the README for the specific binding for configuration
  4. run the recommended sequence of workloads
  5. post results here, with details about versions / sizes / etc.

using either the binding specific tarball or the omnibus is fine, but which one got used should probably be noted in the results.

busbey commented 6 years ago

waiting on these merges:

busbey commented 6 years ago

merged #908

busbey commented 6 years ago

merged #1098 and #1147 .

looks like folks are in progress on #1051

busbey commented 6 years ago

would like to get #1148 "Update HBase 2 version to 2.0.0 GA" in for polish on the hbase addition

busbey commented 6 years ago

As a reminder, this release needs to cover essentially everything since 0.12.0. So the release notes will end up copying some things from the 0.13.0 release notes (like known incompatibilities)

This list of things to check presumes both #1148 and #1051 will land before I cut the feature branch.

New Topics

Needs a review because they'll get pushed in the release notes:

Datastore bindings that changed and will need to be tested to stay in the "verified" category

Newly added datastore bindings

Issues with core

busbey commented 6 years ago

Another modified store I just merged:

busbey commented 6 years ago

found some headers missing before making the staging branch. put up #1150 to fix it.

twblamer commented 6 years ago

Looks like the two output fixes that were done for 0.13.0 aren't present in master: 88ffdbbb8f4efd49a710c3602d16cdb9f8b37a14 and 46933b1d1cf9253d532c087d8d58e0046d4d0ebd

busbey commented 6 years ago

excellent catch!

busbey commented 6 years ago

okay I cherry-pick #1060, #1084, and #1085 to the master branch.

busbey commented 6 years ago

Release candidate 1 is up: https://github.com/brianfrankcooper/YCSB/releases/tag/0.14.0-RC1

robertpang commented 6 years ago

@busbey Just tested 0.14.0-RC1 with Apache Cassandra and @YugaByte datastores with no issue. Here are the test config and results:

Datastore setup:

YCBS client setup:

YCSB workload parameters:

Apache Cassandra config:

Results: workloada-transaction.dat:[OVERALL], Throughput(ops/sec), 66747.20829801293 workloadb-transaction.dat:[OVERALL], Throughput(ops/sec), 51192.273091312025 workloadc-transaction.dat:[OVERALL], Throughput(ops/sec), 54417.28508822448 workloadd-transaction.dat:[OVERALL], Throughput(ops/sec), 66660.44502513099 workloade-transaction.dat:[OVERALL], Throughput(ops/sec), 8650.39383899774 workloadf-transaction.dat:[OVERALL], Throughput(ops/sec), 35613.0058469268

YugaByte DB config:

Results: workloada-transaction.dat:[OVERALL], Throughput(ops/sec), 72913.93239420188 workloadb-transaction.dat:[OVERALL], Throughput(ops/sec), 75755.85403362045 workloadc-transaction.dat:[OVERALL], Throughput(ops/sec), 72795.03829019015 workloadd-transaction.dat:[OVERALL], Throughput(ops/sec), 70898.85569246912 workloade-transaction.dat:[OVERALL], Throughput(ops/sec), 14612.108483944552 workloadf-transaction.dat:[OVERALL], Throughput(ops/sec), 50118.351472330716

Please let me know if you need more details or the full output log.

busbey commented 6 years ago

Thanks @robertpang! that'll do nicely.

busbey commented 6 years ago

I sent pings for testing to the user@ lists for:

busbey commented 6 years ago

known issues for this release should include #1155

metatype commented 6 years ago

@upthewaterspout and I tested the Geode datastore versions 1.2.0, 1.3.0, and 1.6.0 (latest) against the Geode driver. Everything looks good!

busbey commented 6 years ago

thanks geode folks!

fwang29 commented 6 years ago

I tested 0.14.0-RC1 with Kudu (single node) on CentOS 6.6 and tweaked with some workload parameters as below: kudu_buffer_num_ops=1000 kudu_block_size=2048 kudu_table_num_replicas=1

Nothing looks wrong!

busbey commented 6 years ago

thanks @fwang29! what version of Kudu did you use?

fwang29 commented 6 years ago

@busbey NP! Sorry, I forgot to mention it. It was version 1.8.0.

twblamer commented 6 years ago

I tested the mongodb binding and had no issues.

YCSB build:

Host details:

Client details:

Database setup:

YCSB workload parameters:

Disclaimer: this was and a "fire and forget" run with no tuning

Results for mongodb client

./load1.result.txt:         [OVERALL], Throughput(ops/sec), 188692.0621023315
./workloada.result.txt:     [OVERALL], Throughput(ops/sec), 48469.972609618475
./workloadb.result.txt:     [OVERALL], Throughput(ops/sec), 178248.0357066465
./workloadc.result.txt:     [OVERALL], Throughput(ops/sec), 250913.95407772812
./workloadf.result.txt:     [OVERALL], Throughput(ops/sec), 45878.48349590374
./workloadd.result.txt:     [OVERALL], Throughput(ops/sec), 247723.42175408002
./load2.result.txt:         [OVERALL], Throughput(ops/sec), 119994.28827187826
./workloade.result.txt:     [OVERALL], Throughput(ops/sec), 29943.45477999346

Results for mongodb-async client

./load1.result.txt:         [OVERALL], Throughput(ops/sec), 45912.81798643827
./workloada.result.txt:     [OVERALL], Throughput(ops/sec), 24325.67407050991
./workloadb.result.txt:     [OVERALL], Throughput(ops/sec), 56964.15245885764
./workloadc.result.txt:     [OVERALL], Throughput(ops/sec), 72597.38939787725
./workloadf.result.txt:     [OVERALL], Throughput(ops/sec), 24995.731978764627
./workloadd.result.txt:     [OVERALL], Throughput(ops/sec), 70591.75657703396
./load2.result.txt:         [OVERALL], Throughput(ops/sec), 43336.09962102581
./workloade.result.txt:     [OVERALL], Throughput(ops/sec), 5611.603314190471
busbey commented 6 years ago

I'd like to wrap up this RC by the end of the week. Thanks so much to the folks who have tested things so far!

IMHO we already have enough tested datastore bindings to close things out. Since I have most of the testing for the Apache HBase related changes done, I think I'll make a go of finishing that up before closing things up.

I can test out that basics work on windows as well, unless someone else wants to take it on.

Bindings currently slated to be listed as "untested / your milage may vary", with a ping to folks who showed up on related PRs:

haihuang-ml commented 6 years ago

Google Datastore

I have this continuously running in our test environment and see no problem.

busbey commented 6 years ago

@haih-g could you have it test specifically against the 0.14.0-RC1 tag?

diogojfernandes commented 6 years ago

ArangoDB3

Only worked on 0.13 binaries, still can't load and run workload E.

isuntsov-gridgain commented 6 years ago

@busbey where I can get the actual list of untested bindings?

busbey commented 6 years ago

My comment from ~3 days ago is still fairly up to date. IIRC, only Google Datastore has since been tested.

I'm probably only going to test the HBase 1.2+ bindings, which would mean HBase 0.98 and HBase 1.0 will also go into the list.

Is that what you're looking for @isuntsov-gridgain?

ctubbsii commented 6 years ago

Although I was tagged, I won't be able to test the Accumulo bindings without detailed step-by-step instructions... I'm not a YCSB user and know next to nothing about YCSB. However, with detailed step-by-step testing instructions, I'd happily launch an Accumulo instance to test against.

joshelser commented 6 years ago

@ctubbsii see the README, e.g https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.8. It's very straightforward. Just "load" and then "test" for a workload. Testing one is likely "good enough" (you don't need to do workloads A through F).

ctubbsii commented 6 years ago

@joshelser Thanks. I didn't realize there was a README in the subdirectory with detailed instructions. Will use those. :smiley_cat:

bosher commented 6 years ago

Hi Sean,

I have tested my s3 changes with 0.14.0 RC1.

My testing have not revealed any problems.

Best regards, Boris Osher

(408) 368-9089

On Mon, Jun 4, 2018 at 10:26 AM, Sean Busbey notifications@github.com wrote:

I'd like to wrap up this RC by the end of the week. Thanks so much to the folks who have tested things so far!

IMHO we already have enough tested datastore bindings to close things out. Since I have most of the testing for the Apache HBase related changes done, I think I'll make a go of finishing that up before closing things up.

I can test out that basics work on windows as well, unless someone else wants to take it on.

Bindings currently slated to be listed as "untested / your milage may vary", with a ping to folks who showed up on related PRs:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brianfrankcooper/YCSB/issues/1117#issuecomment-394433864, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXzlYhNxXlK2OXQSM0uy9QHa-_Kgn4rks5t5W3DgaJpZM4S-Bxn .

ctubbsii commented 6 years ago

Issues I found testing Accumulo 1.9.1:

  1. The build has an error: no version specified for apache-rat-plugin (see PR #1168)
  2. accumulo.1.7.version should be 1.7.4 (see PR #1167)
  3. accumulo.1.8.version should be 1.9.1 (see PR #1167) (yes, 1.9.1, not 1.8.1; 1.8.0 and 1.8.1 had critical data loss issues; 1.9.x replaces the 1.8 series)

I'm not sure what to look for in the testing output, but everything seemed to work fine if I set accumulo.1.8.version to 1.9.1 and ran load, followed by run, according to the accumulo1.8 README.

I tested using https://github.com/apache/fluo-uno using workload A on a single machine, with Apache Accumulo 1.9.1, Apache Hadoop 2.7.6, and Apache ZooKeeper 3.4.12

busbey commented 6 years ago

are the Accumulo version numbers needed for the release or would they be fine to update for next time?

ctubbsii commented 6 years ago

I think the main risk with releasing without the updated Accumulo version numbers is possible confusion. I would rate the risk low.

isuntsov-gridgain commented 6 years ago

@busbey actually I'm not familiar with any distributed systems from this list so it will be the random choice. I've started to work with Aerospike.

isuntsov-gridgain commented 6 years ago

Guys,

I see in YCSB/pom.xml (0.14.0-staging):

3.1.2 It is a version that was released in 2014. With the 4.1+ I have compilation problems. Is it OK?
busbey commented 6 years ago

I see in YCSB/pom.xml (0.14.0-staging):

3.1.2 It is a version that was released in 2014. With the 4.1+ I have compilation problems. Is it OK?

If it works with the version advertised in YCSB then it's okay for the release. if things are tested then we expressly call out what version(s) were tested for the release notes. ("advertised" here means either a) mentioned in the README, b) mentioned in prior release notes, c) version in the pom

For this sort of thing usually I'd suggest making sure there's a ticket about needing to update. Or if you want to and have time to fix it now just start with a PR. We can include an update in the next release. I'd look for feedback from whoever maintains Aerospike about wether it's worth keeping compatibility with both v3 and v4.

rohanjayaraj commented 6 years ago

@busbey Tested 0.14.0-RC1 with MapR-DB and MapR-JSONDB datastores and found no issues. Below are the configs and results :

Datastore setup

YCBS client setup

YCSB workload parameters

MapR Database Setup

Results for maprdb client

load1:       [OVERALL], Throughput(ops/sec), 121410.793419535
workloada:     [OVERALL], Throughput(ops/sec), 135895.41488870166
workloadb:     [OVERALL], Throughput(ops/sec), 81240.70809401174   
workloadc:     [OVERALL], Throughput(ops/sec), 79145.23149980213
workloadf:      [OVERALL], Throughput(ops/sec), 68762.54916522265
workloadd:     [OVERALL], Throughput(ops/sec), 83941.91219675985
load2:       [OVERALL], Throughput(ops/sec), 133481.05236461683
workloade:     [OVERALL], Throughput(ops/sec), 8598.895901766213

Results for maprjsondb client

load1:       [OVERALL], Throughput(ops/sec), 147240.70911125507
workloada:      [OVERALL], Throughput(ops/sec), 87088.29010851201
workloadb:      [OVERALL], Throughput(ops/sec), 70698.35838411831
workloadc:      [OVERALL], Throughput(ops/sec), 76678.29620825825
workloadf:      [OVERALL], Throughput(ops/sec), 37459.216278277025
workloadd:     [OVERALL], Throughput(ops/sec), 68926.5380956976
load2:       [OVERALL], Throughput(ops/sec), 162593.6946165228
workloade:     [OVERALL], Throughput(ops/sec), 9972.575417601596
isuntsov-gridgain commented 6 years ago

I've finished with Aerospike. Environment:

YCBS client setup

Aerospike server version: 4.2.0.3

Results for Aerospike client: load: [OVERALL], Throughput(ops/sec), 4651.162790697675 workloadb: [OVERALL], Throughput(ops/sec), 4739.336492890995 workloadc: [OVERALL], Throughput(ops/sec), 5464.48087431694 workloadf: [OVERALL], Throughput(ops/sec), 4310.3448275862065 workloadd: [OVERALL], Throughput(ops/sec), 4926.108374384236

isuntsov-gridgain commented 6 years ago

Redis - done! Version: 4.0.9 Environment:

YCBS client setup

load [OVERALL], Throughput(ops/sec), 427.53313381787086

run a [OVERALL], Throughput(ops/sec), 4901.9607843137255 b [OVERALL], Throughput(ops/sec), 4975.124378109453 c [OVERALL], Throughput(ops/sec), 5128.205128205128 d [OVERALL], Throughput(ops/sec), 4739.336492890995 f [OVERALL], Throughput(ops/sec), 3968.253968253968

busbey commented 6 years ago

in evaluating the RC I found that the big tarball doesn't have a README.md included, filed #1171. It's been broken for ~4 years so I don't think it needs to be fixed for this release. someone shout if they'd prefer otherwise.

busbey commented 6 years ago

the hbase098 binding instructions refer to a binding that doesn't exist instead of hbase098. filed #1172. I don't think this should be a blocker.

busbey commented 6 years ago

tested against HBase 1.2 (CDH 5.14.2) (via hbase098, hbase10, hbase12, hbase14, and hbase20 bindings)

found #1173 [hbase20] no slf4j implementation; hbase client doesn't log. it doesn't seem like a blocker, since none of the information we miss from logging is critical and it can be worked around by adding an slf4j logging-binding to the classpath.

busbey commented 6 years ago

tested against HBase 2.0 (CDH 6.0.0-beta1) (via hbase10, hbase12, hbase14, and hbase20 bindings)

1173 applies there too.

busbey commented 6 years ago

draft release notes: https://gist.github.com/busbey/9170b4d5f82b7be215d846f4b0a5d398

petersomogyi commented 6 years ago

@busbey: release notes is great! I have 2 minor comments:

busbey commented 6 years ago

Thanks @petersomogyi! I've corrected both of those now.