Popcorn Linux Compiler Toolchain, Copyright Systems Software Research Group at Virginia Tech, 2017.
For more information, please visit http://popcornlinux.org or e-mail Rob Lyerly (rlyerly@vt.edu).
The goal of the heterogeneous compiler toolchain is to prepare multi-ISA binaries for migration through a series of analyses and transformations. We utilize and extend clang/LLVM in order to prepare heterogeneous binaries. We also use a python-based tool to prepare custom linker scripts to align program objects in the generated binaries. Finally, there are a number of additional libraries needed for migration and runtime state transformation.
We need to prepare the binary so that before and after migrating between architectures, the application is able to find the required code and data to seamlessly continue execution. This is done by both using a common layout (where it is possible without significant performance overhead) and state transformation (where state is dictated by the ISA or where a common format would be too costly in performance).
The toolchain operates as follows:
Parsing/LLVM bitcode generation (clang) - clang frontend, mostly unmodified
Middle-end analysis, refactoring & optimization - in addition to standard optimizations, run several passes which adjust the linkage of some variables, insert migration points into the application and notify architecture-specific backends to generate stack frame metadata
Backend - modified LLVM backend(s) generate custom data and function location information and compile bitcode to object-code
Linker - modified gold linker generates detailed data and function linking information, aligns thread-local storage for all architectures
Alignment - python tool uses information provided by gold linker to generate linker scripts that align data & function symbols across binaries
Metadata Generation - parse information generated by LLVM backends and add stack transformation metadata to binaries (must be run post-alignment as it requires final symbol layouts)
In order to have functionally-identical implementations for all compiled source code, the binary must be compiled into a single IR representation, which is then used by each of the backends to generate architecture-specific code:
----------------
| Orig. Source |
----------------
|
| (clang)
V
----------------
| LLVM IR |
----------------
|
| (opt)
V
----------------
| Optimized IR |
----------------
|
--------------------------------- (arch-specific backend)
| | |
V V V
--------------- ---------------
| aarch64 bin | ... | x86 bin |
--------------- ---------------clang is set up to automatically implement this process -- generate a single set of instrumented IR and lower the IR to each target's machine code.
The toolchain's installation folder is organized as shown below, along with a brief introduction about important sub-folders:
| root \ common - source/headers common between components in different folders | lib - libraries needed by the compiler and/or the compiled application \ libelf - library for parsing & reading ELF objects |
|---|
libopenpop - Popcorn Distributed OpenMP library
|
migration - functionality for migrating between architectures
|
musl-1.1.18 - standard C library
|
stack_transformation - runtime for transforming stacks between ISA-specific
layouts| patches - patches for compiler components \ binutils-gold - patch for GNU gold linker |
|---|
llvm - patches clang/LLVM| tool - post-compilation binary tools for alignment & metadata-generation \ alignment - python tool for generating linker scripts to align binaries |
|---|
stack_metadata - tools for post-processing binaries in preparation for
stack transformation| util - various utilities, including a Makefile template, for heterogeneous applications \ scripts - scripts for patch generation, testing and running binaries
See the README file in each subdirectory for more information, and the INSTALL file for installation instructions.
Hardware requirements:
It is highly recommended that the toolchain be built & installed on a machine with at least 4 cores and 8GB of RAM. This is because LLVM is a very large codebase and is built with debugging information by default, making the compile & linking phases taxing in terms of memory consumption.
Software requirements:
Note: the toolchain has been tested on x86-64 with Ubuntu 16.04 and up. It should work on other architectures and distributions (in particular, Debian 8), but may require installing alternate packages and some extra hacking. It is highly recommended that applications be built on x86-64.
The current version of the toolchain only works for aarch64 + x86-64. Applications must be written in C (although anything for which LLVM IR can be generated should be supported), and inline assembly is not supported. All optimizations except auto-vectorization and frame-pointer elimination (required by stackmap intrinsic) are supported.