search

amytai / cinterrupts-osdi

8 stars 5 forks source link

readme

Calibrated Interrupts

This repository contains instructions and source code for compiling and reproducing results in the Calibrated Interrupts (cinterrupts) paper, to appear in OSDI '21 (see the paper PDF in the top-level of this repository).

Evaluation instructions

At the end of this document, we describe how to compile and install the evaluation environment, should the evaluator choose to do so. However, due to needing the cinterrupts custom kernel, we have set up an environment for the evaluators on our machine.

Accessing the evaluation environment

Our system works closely with real hardware and reproduction of our results requires a low latency Intel Optane NVMe SSD (or similar). In addition, we wrote our scripts with an assumption that underlying SSD is connected to a NUMA node #1 which hosts cores 1,3,5,7. Different configuration will require updating our scripts accordingly. Furthermore, in our experience, different CPUs and machine setups require different macrobenchmark (application) configurations to saturate the CPU. This is why we provide evaluators with an access to our setup with Intel Optane NVMe SSD installed and preconfigured building environment.

Please contact authors how to access this setup remotely. (Account username/password and machine IPs are privileged information that we prefer to send out-of-band.)

Content of the repository

Compilation instructions

We highly recommend that you build on Ubuntu 16.04. To build the custom cint kernel, you will need any dependencies required for the Linux kernel. These include libssl-dev, bison, flex, and optionally dh-exec. If there is a compilation error, it is likely because one of these packages is missing.

Run build-kernel.sh in the top-level directory of this repository. This will build and install our custom kernels for micro and macro benchmarks. You will then need to run this script once. To simplify artifact testing we already ran this script which extracted, compiled and installed our kernels into linux-kernel/linux-kernel-5.0.0-16.17-nvmecint and linux-kernel/linux-kernel-5.0.0-16.17-nvmecint-rocks directories.

We install two kernels:

To boot into 5.0.8-nvmecint kernel run:

$> sudo grub-reboot "Ubuntu, with Linux 5.0.8-nvmecint"
$> sudo reboot

To boot into 5.0.8-nvmecint-rocks kernel run:

$> sudo grub-reboot "Ubuntu, with Linux 5.0.8-nvmecint-rocks"
$> sudo reboot

When kernel is loaded the driver is ready. If you modify the driver and need to compile it then run:

$> cd linux-kernel/linux-kernel-5.0.0-16.17-nvmecint
$> sh nvme-make.sh

After that, to switch between different NVMe interrupt emulations and the original driver, you simply need to unload and load the correct nvme driver with relevant parameters:

$> cd linux-kernel/linux-kernel-5.0.0-16.17-nvmecint
$> sh ./nvme-reload.sh our-sol
$>
$> sh ./nvme-reload.sh
Usage: ./nvme-reload.sh {orig|emul|our-sol}

     orig    -- original nvme driver, for-bare-metal tests
     emul    -- emulation of the original nvme driver on a side core
     emul-100-32 -- emulation of the original nvme driver with 100 usec and 32 thr aggregation params
     our-sol -- side-core emulation of our nvme prototype with URGENT and BARRIER flags
     alpha   -- side-core emulation of our nvme prototype, only adaptive coalescing
     alpha0  -- side-core emulation of our nvme prototype, without any thresholds (new baseline0)

To change the parameters edit config files in form nvme-$(hostname)-$(mode).conf, for example:

$> cd linux-kernel/linux-kernel-5.0.0-16.17-nvmecint
$> vim nvme-$(hostname).conf            # params for the cinterrupts driver
$> vim nvme-$(hostname)-clean.conf      # params for the original nvme driver
$> vim nvme-$(hostname)-emul.conf       # params for the emulated nvme device driver

Installation and Setup Instructions

After booting into this custom kernel, compile fio benchmark. Run build-fio.sh in the top-level directory of this repository. Path to fio from the top-level directory: fio/fio-3.12/fio If you can successfully run fio, you are ready!

Running benchmarks

You should compile the following applications, which are applications we modified for cinterrupts.

Reproducing each figure

In the figX/ subdirectories, we have scripts and instructions for reproducing the key figures in our paper, e.g., fig5 directory contain all scripts needed to reproduce Figure 5 in the paper. Enter to a figX directory and run make-all.sh. See figX.pdf with test results, but please check README for each directory to confirm output. For example, results for tables in the paper are stored directly in *.out files.

Pay attention, for microbenchmarks, our scripts run each benchmark 10 times, 60 seconds each run. Since there are multiple flavours of each test the total runtime can be very long. To reduce total runtime evaluators can change runtime and runs variables in the test scripts.

For macrobenchmarks, experiments can also take a while to run (up to 45 min) as they run each benchmark 5 times. Consider using tmux to make sure the benchmark continues to run even if ssh connection is broken.