add oltpbenchmark

[robert-s-lee]

oltpbenchmark has about 15 OLTP workloads that can be used for demo.

install java with `sudo apt-get -y update; sudo apt-get -y install openjdk-8-jdk maven'

install oltpbenchmark `git clone https://github.com/robert-s-lee/oltpbench.git --branch cockroachdb --single-branch oltpbenchmark; cd oltpbenchmark; ant'

run cd oltpbenchmark;./oltpbehch.sh -w tpcc -t "1 2 4 8 16"

-w can have the following options auctionmark linkbench seats tpcc tatp twitter voter epinions sibench smallbank ycsb wikipedia resourcestresser

-t is the concurrency. use 1 as default, but with an option to change

-s is the size of the database. use 1 as the default, but with an option to change

There are other options but we don't need to expose at this time.

[nstewart]

Personal note: docker version is here: https://hub.docker.com/r/rijalati/oltpbench/

[nstewart]

Currently testing and it looks like some benchmarks end prematurely.

Working

• [x] tpcc (via cockroach workload)
• [x] kv (via cockroach workload)
• [x] auctionmark
• [x] linkbench
• [x] tatp
• [x] voter
• [x] epinions
• [x] sibench
• [x] smallbank
• [x] ycsb - works
• [x] wikipedia
• [x] resourcestresser

Not Working

• [ ] seats - errors before starting
• [ ] twitter - errors after running for a bit

[robert-s-lee]

The default runs for 1 minute. Maybe put -m 1000 for 1,000 seconds.

[nstewart]

@robert-s-lee -- I was using a long-running -m value, maybe its ignored for that one.

The `seats` benchmark erros out when run `docker run -it --rm natestewart/cloudformation-oltpbench -w seats -l -t 1 -m 2592000 -i ipaddress`

``` 13:12:34,262 (DBWorkload.java:818) INFO - Launching the SEATS Benchmark with 1 Phase... java.lang.RuntimeException: Unexpected error when initializing PENDING_INSERTS: 0 PENDING_DELETES: 0 PENDING_UPDATES: 0 CACHE_CUSTOMER_BOOKED_FLIGHTS: 0 CACHE_BOOKED_SEATS: 0 PROFILE: Scale Factor: 0.0 Data Directory: /oltpbenchmark/build/com/oltpbenchmark/benchmarks/seats/data # of Reservations: 0 Flight Start Date: 1970-01-01 00:00:00.0 Flight Upcoming Date: null Flight Past Days: 0 Flight Future Days: 0 Flight Upcoming Offset: null Reservation Upcoming Offset: null at com.oltpbenchmark.api.Worker.run(Worker.java:224) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: relation "config_profile" does not exist at com.oltpbenchmark.benchmarks.seats.SEATSWorker.initialize(SEATSWorker.java:284) at com.oltpbenchmark.api.Worker.run(Worker.java:222) ... 1 more Caused by: org.postgresql.util.PSQLException: ERROR: relation "config_profile" does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:150) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:113) at com.oltpbenchmark.benchmarks.seats.procedures.LoadConfig.run(LoadConfig.java:65) at com.oltpbenchmark.benchmarks.seats.SEATSProfile.loadProfile(SEATSProfile.java:316) at com.oltpbenchmark.benchmarks.seats.SEATSWorker.initialize(SEATSWorker.java:282) ... 2 more ```

[nstewart]

twitter crashes after some time

13:28:02,432 (ThreadBench.java:283) INFO  - Throughput: 1.6 txn/sec
13:28:07,433 (ThreadBench.java:283) INFO  - Throughput: 1.4 txn/sec
java.lang.RuntimeException: Unexpected fatal, error in 'TwitterWorker<000>' when executing 'GetFollowers/03' [COCKROACHDB]
    at com.oltpbenchmark.api.Worker.doWork(Worker.java:544)
    at com.oltpbenchmark.api.Worker.run(Worker.java:296)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:783)
    at org.postgresql.jdbc.PgConnection.rollback(PgConnection.java:791)
    at com.oltpbenchmark.api.Worker.doWork(Worker.java:428)
    ... 2 more

[robert-s-lee]

Will have to debug those. The Twitter TPS seem very low. So 9 of the ones do work so far?

[nstewart]

I wouldn't worry about the TPS -- I'm testing on two clusters to parallelize things. One cluster is running with the bare minimum resources (1 shared crdb + loadgen node), which is why tps is so slow. I'm just making sure each loadgen actually works.

Auctionmark is working now. So so far we have 10 generators that work. I'm almost done testing the rest

[nstewart]

oltpbench's tpcc is too slow.

I'll use our cockroach workload version instead

[robert-s-lee]

@nstewart just tried smallbank

[nstewart]

You have to select a key when you use the template @robert-s-lee

Cloudformation doesnt have required fields so it fails with the message you saw

[nstewart]

The other thing to watch out for is resource limits @robert-s-lee. If many SEs are running templates in the same region, you may bump into the 15 ec2 instance limit. If this becomes an issue we can see how to get these limits lifted.

If you hit an EC2 instance the autoscaling group will hang and eventually, the template will time out. if you hit a limit on VPCs/internet gateways/etc the template will fail immediately with an error.

https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html

[robert-s-lee]

made sure to fill in all of the items this time. nice sub millisecond response times on linkbench.

[nstewart]

@robert-s-lee any update on seats and twitter issues?