Ocolos is the first online code layout optimization system for unmodified applications written in unmanaged languages. Ocolos allows profile-guided optimization to be performed on a running process, instead of being performed offline and requiring the application to be re-launched. A description of how we implemented Ocolos and experimental results on MySQL-sysbench workloads are in MICRO'22 paper.
For the demonstration purpose, we integrate MySQL
and sysbench
to Ocolos, so this version of Ocolos ONLY works with MySQL
.
Please refer instructions from links or directly run commands listed below to install prerequisites:
sudo apt-get install linux-tools-common linux-tools-generic
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt-get install libboost-all-dev
[^1]:If your objdump
version is older than 2.27, please download the latest version of binutils
. > git clone git@github.com:upenn-acg/ocolos-public.git
To use llvm-bolt
and perf2bolt
utilities, BOLT
needs to be installed. \
Please follow the commands below to install BOLT
> mkdir BOLT && cd BOLT
> git clone git@github.com:upenn-acg/BOLT.git llvm-bolt
> cd llvm-bolt
> git checkout ocolos/cont-opt
> cd ..
> mkdir build && cd build
> cmake -G "Unix Makefiles" ../llvm-bolt/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="clang;lld;bolt"
> make -j
> git clone https://github.com/mysql/mysql-server.git
> cd mysql-server
> git checkout 6846e6b2f72931991cc9fd589dc9946ea2ab58c9
In CMakeList.txt
, at line 580, please add [^2]:
[^2]: If the mysqld binary compiled by gcc generates callq
instructions rather than call
instructions, please refer to the solution discussed in this page.
STRING_APPEND(CMAKE_C_FLAGS " -fno-jump-tables")
STRING_APPEND(CMAKE_CXX_FLAGS " -fno-jump-tables")
STRING_APPEND(CMAKE_C_FLAGS " -no-pie")
STRING_APPEND(CMAKE_CXX_FLAGS " -no-pie")
Also, in CMakeList.txt
, turn off ld.gold
linker: \
change OPTION(USE_LD_GOLD "Use GNU gold linker" ON)
to be OPTION(USE_LD_GOLD "Use GNU gold linker" OFF)
Then build mysqld
from source:
> export CC=gcc
> export CXX=g++
> mkdir build && cd build
> cmake .. -DWITH_BOOST={path of the boost_1_73_0 directory} -DCMAKE_CXX_LINK_FLAGS=-Wl,--emit-relocs -DCMAKE_C_LINK_FLAGS=-Wl,--emit-relocs -DBUILD_CONFIG=mysql_release
> make -j
> make install
To initialize MySQL, run:
> chown -R {user} {path to MySQL directory}
> {path to MySQL directory}/bin/mysqld --initialize-insecure --user=root --datadir={your data dir path of MySQL}
> {path to MySQL directory}/bin/mysqld --user=root --port=3306 --datadir={your data dir path of MySQL}
In another terminal, run:
> mysql -u root
> CREATE USER 'ocolos'@'localhost';
> GRANT ALL PRIVILEGES ON *.* TO 'ocolos'@'localhost' WITH GRANT OPTION;
> CREATE DATABASE ocolos_db;
> QUIT;
> mysqladmin -u root shutdown
Note:
/usr/local/mysql
unless otherwise specified during MySQL server's installation. > sudo apt-get update
> sudo apt-get install sysbench
Or if you prefer to build sysbench from source, please refer instructions in the following webpage:\ https://github.com/akopytov/sysbench
ocolos-public
directory. config
, specify the absolute path for nm
,perf
,objdump
,llvm-bolt
,perf2bolt
[^3]
[^3]: if nm
,objdump
and perf
are already in shell, it's OK that their paths are not specified in config
. This can be checked by which nm
, which objdump
and which perf
.config
, please also specify the commands to run MySQL server
and sysbench
. The example commands are given in the config file.
ocolos-public
's directory.
> export OCOLOS_PATH=/your/path/to/ocolos-public
> make
> ./extract_call_sites
> ./tracer
make
will produce 2 executables (tracer
& extract_call_sites
)+ 1 shared library (replace_function.so
).
/usr/local/lib
, you also need to edit Makefile and update it to the corresponding path./usr/local/include
, you also need to edit Makefile and update it to the corresponding path. ./extract_call_sites
will produce 2 files which store all call sites information extracted from the target binary (a.k.a. mysqld
) to the tmp_data_dir
you specified in the config file. ./tracer
will invoke both MySQL
server process and sysbench workloads oltp_read_only
, and then perform code layout optimization during runtime.
sysbench_output.txt
. At about the 130th second, you will see a significant throughput improvement, since Ocolos has replace the code layout to be the optimized one at that time.mysqladmin -u root shutdown
command to shutdown the current MySQL
server process. BOLT
to make it support converting perf.data
collected from C1 to be the perf.fdata
that llvm-bolt
can use.
C0
& C1
C0
: The duration before Ocolos
's code replacement C1
: The duration after Ocolos
's code replacement BOLT
's code supports continuous optimization can be found here.C0
, the perf2bolt
and llvm-bolt
commands are changed to :
> perf2bolt -p perf_c0.data -o perf_c0.fdata mysqld
> llvm-bolt mysqld -o mysqld_c0.bolt --enable-bat --enable-func-map-table -data=perf_c0.fdata -reorder-blocks=cache+ -reorder-functions=hfsort
C1
, to make profile collected from C1
work with perf2bolt
, and then to produce C1's mysqld.bolt
for next round of code replacement, the perf2bolt
and llvm-bolt
commands are changed to :
perf2bolt
command, callstack_func.bin
& BOLTed_bin_info.txt
are produced by Ocolos
during C0's code replacement, since we need to pass some essential information from Ocolos
to BOLT
> perf2bolt --ignore-build-id --cont-opt --call-stack-func=callstack_func.bin --bin-path-info=BOLTed_bin_info.txt -p perf_c1.data -o perf_c1.fdata mysqld_c0.bolt
> llvm-bolt mysqld -o mysqld_c1.bolt --enable-bat --enable-func-map-table -data=perf_c1.fdata -reorder-blocks=cache+ -reorder-functions=hfsort
mysqld.bolt
produced from Ocolos' C0 to build a newly BOLTed binaryoltp_read_only
to show the throughput-DCONT_OPT
to CXXFLAGS
in Makefile, and compile Ocolos
again. sh scripts/C1_BOLTed_performance_test.sh
In Makefile
's CXXFLAGS
,
-DTIME_MEASUREMENT
flag is added, Ocolos will print the execution time of code replacement;-DMEASUREMENT
flag is added, Ocolos will print metrics such as:
BOLT
,BOLT
and original functions.-DDEBUG_INFO
flag is added, Ocolos will print debug information such as:
-DDEBUG_INFO
can also be defined in src/replace_function.hpp
. In this way, the ld_preload library will store all machine code per function it inserted to the target process as a uint8_t
format array into a file. The file can be found in the tmp_data_path
you defined in the config file. -DDEBUG
flag is added, after code replacement, Ocolos will first send SIGSTOP
signal to target process and then resume the target process by PTRACE_DETACH
. In this way, it allows debugging tools such as GDB
to attach to the target process and observe what goes wrong after code replacement.If the code replacement runs into a failure, you may want to do the following things to fix this problem
-DDEBUG_INFO
to CXXFLAGS
in Makefile and compile again../tracer
tracer
run into an error, you can check {tmp_data_dir}/machine_code.txt
's last few linesmmap failed. file exists.
mysqld
process. replace_function.cpp
, and then compile Ocolos again.