Cobra Bench

Cobra bench is a component of the Cobra project. It includes benchmarks to generate histories for Cobra verifier and has tools to measure client-side overheads.

This tutorial introduces how to build Cobra bench and run it with three different databases: RocksDB, PostgreSQL, and Google Cloud Datastore.

The following commands have been tested under Ubuntu 18.04.

Build Cobra bench

Install maven:

$ sudo apt install maven

Compile the code:

$ mvn install

Now you can run a simple test:

$ mkdir -p /tmp/cobra/log
$ java -ea -jar target/txnTest-1-jar-with-dependencies.jar local config-historysize.yaml

The history will be stored in the folder /tmp/cobra/log.

You can specify workload parameters in Cobra's config file (config-historysize.yaml in the above case). See Cobra bench configuration for more information.

Run Cobra bench with RocksDB (single machine)

You can run Cobra bench with RocksDB on a single machine -- both RocksDB and its clients will run on the same machine. (One can also run RocksDB and its clients on different machines, see Step 3 in the next section for more information.)

First, specify using RocksDB as the backend in the config file:

$ cd $COBRA_HOME/CobraBench
$ cp config.yaml.default config.yaml
# set config.yaml line 5 to: "LIB_TYPE: 2"

Next, clear the existing logs and databases, and run Cobra bench:

$ rm -r /tmp/cobra/ /tmp/rocksdb/
$ mkdir -p /tmp/cobra/log/
$ java -ea -jar target/txnTest-1-jar-with-dependencies.jar local config.yaml

The history will be stored in the folder /tmp/cobra/log/.

Run Cobra bench with PostgreSQL (multiple machines)

This section describes how to run experiments with PostgreSQL on multiple machines using Cobra's auto-scripts. Running them requires at least two machines: one for clients and one for the database.

Note: running the auto-scripts will create temporary files under the home folder of clients' machines.

Step 1: Environment for auto-scripts

On the machine that you want to host the database and control the experiments:

(1) config SSH

Host client1
    Hostname [hostname]
    User [username] 
    IdentityFile [path_id_rsa]

$ ssh client1

Install Anaconda:

$ cd $COBRA_HOME/CobraBench/
$ wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
$ bash Anaconda3-2020.07-Linux-x86_64.sh
$ source ~/.bashrc # or any shell you are using

Then, create Anaconda environment and install required packages:

$ conda create --name txn python=3.7.5
$ conda activate txn
$ which python
# You should see something like "/home/ubuntu/anaconda/envs/txn/bin/python"

$ cd $COBRA_HOME/CorbraBench/eval/
$ pip install -r requirements.txt

(3) Install Redis

We use Redis to start multiple clients at the same time.

$ sudo apt install redis
$ pip install redis
$ sudo vim /etc/redis/redis.conf
# comment out the line "bind 127.0.0.1 ::1"
# change the line "protected-mode yes" to "protected-mode no"

$ sudo service redis-server restart

Config Redis:

$ cd $COBRA_HOME/CobraBench/
$ vim config.yaml.default
# line 38 (REDIS_ADDRESS: "redis://[ip]/0"): replace [ip] with this machine's IP address

$ vim eval/main.py
# line 17 (redis_ip = [ip]): change [ip] to this machine's IP address

(4) Setup Docker images

Install Docker packages:

$ cd $COBRA_HOME/CobraBench/
$ fab -r eval/fabfile.py -H localhost install-docker --uname="[username]"
$ fab -r eval/fabfile.py -H client1 install-docker --uname="[username]"

Note that the above [username]s are Linux usernames for localhost and client1.

Build Docker image:

$ cp config.yaml.default config.yaml
$ fab -r eval/fabfile.py -H localhost rebuild
$ fab -r eval/fabfile.py -H client1 rebuild

Step 2: Setup PostgreSQL

(1) Install PostgreSQL

$ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -sc)-pgdg main" > /etc/apt/sources.list.d/PostgreSQL.list'
$ sudo apt update
$ sudo apt-get install postgresql-10

(2) config PostgreSQL

Modify the file /etc/postgresql/10/main/postgresql.conf:

# add this line to the end of the file.
listen_addresses = '*'
# find max_connections, set it to 500.
max_connections = 500

Add the two lines below at the end of the file /etc/postgresql/10/main/pg_hba.conf:

host    all             all              0.0.0.0/0                       md5
host    all             all              ::/0                            md5

Restart PostgreSQL:

$ sudo service postgresql restart

Set the database's IP address config.yaml.default in the config file:

$ vim $COBRA_HOME/CobraBench/config.yaml.default
# line 6 (DB_URL: "jdbc:postgresql://[ip]:5432/testdb"): replace [ip] with the database's IP address

(3) create a PostgreSQL user

$ sudo -u postgres psql
$ create user cobra with password 'Cobra<318';
$ create user [username];
$ alter user [username] with superuser;
$ \q

Note that the above [username] should be the username associated with the private key mentioned in config SSH.

Step 3: Run experiments with auto-scripts

$ vim $COBRA_HOME/CobraBench/eval/main.py
# line 20 (client_machine = ['client1']): include the clients' hostnames here

run_one_series(database, workload, contention, inst_level)
# database: one of ['rocksdb', 'postgres', 'google']
# workload: one of ['cheng', 'twitter', 'rubis', 'tpcc', 'ycsb'] (cheng stands for BWrite-RW in the paper)
# contention: one of ['low', 'high']
# inst_level: one of ['no', 'local']

$ python eval/main.py recompile

$ cd $COBRA_HOME/CobraBench/eval
$ python report.py

1. Create a Cloud Datastore account.
2. Download the credentials and save it in "cobra_key.json" ($COBRA_HOME/CobraBench/cobra_key.json in the repository is a dummy file).
3. Please refer to this link for further instructions.
4. Follow Step 3: Run experiments with auto scripts in the PostgreSQL chapter, and choose database to be google in the file main.py.

Reproduce results

The following instructions describe how to reproduce the results in the Cobra paper [1], Section 6.3.

Latency and throughput overheads (Figure 10)

You can reproduce Figure 10 by running auto-scripts as described above. In particular, follow Step 3: Run experiments with auto-scripts: choose a database, set the workload to twitter, set inst_level to no for legacy systems and local for Cobra, and get throughput latency results by running the script report.py.

Network cost and history sizes (Figure 11)

• Collect network traffic:

  • First make sure you can run ifconfig on the database machine, then change eval/main.py (line 19), set nic_device to your NIC name (for example, eno1).
  • Follow Step 3: Run experiments with auto-scripts: set the database to postgres, choose the workload you'd like to experiment, and and set inst_level to no for legacy systems and local for Cobra.
  • Run python eval/main.py recompile
  • Each experiment's network costs will be stored as a file under folder ~/netstats/ in the DB's machine.

• Calculate history size: you can measure the history size of a workload as follows.

  $ rm -r /tmp/cobra/ /tmp/rocksdb/
  $ mkdir -p /tmp/cobra/log
  $ cp config-historysize.yaml config.yaml
  $ vim config.yaml
  # update config.yaml and choose a workload you'd like to experiment
  
  $ java -ea -jar target/txnTest-1-jar-with-dependencies.jar local config.yaml

See Cobra bench configuration for how to update config.yaml and specify workload parameters. The history is stored under /tmp/cobra/log/, and you can sum file sizes of /tmp/cobra/log/*.log to calculate the total history size.

The above two experiments reproduce Figure 11 in Section 6.3.

Cobra bench configuration

Reference