PARSEC Benchmark Suite 3.0

Patch Info

This patch provides the following enhancements:

Ability to build on x86_64 Arch Linux to overcome minor bugs.
- Updated openssl, see Abhishek Sagar's blog post
- Corrected uctcpip, see Yungang Bao's response
Generalized builds. Instead of calling gcc tools and overwriting CFLAGS, CC, CFLAGS, and related environment variables set in a global bldconf are preserved while building packages.

Overview

The Princeton Application Repository for Shared-Memory Computers (PARSEC) is a collection of parallel programs which can be used for performance studies of multiprocessor machines.

The PARSEC distribution is composed of packages and the PARSEC framework. Packages correspond to benchmark programs, libraries and other essential components. Each package can be compiled in a number of ways as determined by a build configuration. Build configurations contain information such as which features of the package should be enabled, which compilers to use and how the package should be optimized. PARSEC ships with predefined inputs that can be used to run the benchmarks. The inputs for each program exhibit different characteristics such as execution time and working set size.

Licenses

Before you start using, modifying or distributing PARSEC, its programs or the supplied inputs in any way, make sure you understand all licenses involved. The PARSEC framework itself is available under a liberal open source license, as explained in the file LICENSE which is in the same directory as this README file. Each program uses its own license, which is different in some cases. Some of the inputs have their own license, too. Licenses for source code can be found in the 'src/' directory of each package. Licenses for inputs can be found in the 'inputs/' directory of the package.

We distribute the programs and their workloads bundled with PARSEC merely to allow PARSEC users a convenient access to them and because the license terms allow us do so in each case. You have to take appropriate steps yourself to make sure you don't violate any license terms.

Requirements

PARSEC requires at least 8 GB, but we recommend 12 GB or more. The disk usage can be broken down as follows:

PARSEC occupies about 7 GB with a raw installation. Additional 500 MB are needed for each set of binaries. To build packages and run them extra space is required for temporary files, up to several GB if the whole suite is to be built and executed in one run without cleaning up intermittently.

The PARSEC benchmarks have been parallelized with pthreads, OpenMP, TBB and atomic instructions. Many workloads support more than one parallelization. Each parallelization has its own requirements that have to be fulfilled in order to be able to build and run workloads that use it. By default only support for pthreads and OpenMP are needed. Gcc supports OpenMP since version 4.2.0.

PARSEC has been successfully tested on the following systems:

Linux/i386
Linux/x86_64
Linux/Itanium
Solaris/Sparc

Limited support exists for the following platforms, but not all benchmark programs might be available:

Darwin/PowerPC

Usage

PARSEC ships with several tools which are installed in the 'bin/' directory. You can use them to customize and manage your installation of the benchmark suite. 'parsecmgmt' is the main tool. Its purpose is to build and run packages as well as perform other management operations. 'bldconfadd' and 'bldconfdel' can be used to create and delete your own build configurations.

When you build and run PARSEC with 'parsecmgmt', it will create a log file which will contain all output in the 'log/' directory. You can get a help summary for each tool by invoking it with option '-h'.

A full set of man pages documenting PARSEC, its tools and the most important parts of the source code is given in the man/' directory. If you add the bin/' directory to the PATH environment variable and the man/' directory to the MANPATH variable then all tools and man pages are accessible at the command line without having to specify the full path every time. A bash shell scriptenv.sh' is provided in the root directory of the PARSEC distribution which modifies the environment in that way. If you work with the bash shell you can make use of it before you start working with PARSEC by executing source env.sh'. You can then start browsing the documentation by running man parsec'. Support for other shells is currently not available.

The following examples assume that the 'bin/' directory of PARSEC is in your path.

How to Build PARSEC

Before compiling the PARSEC benchmarks, please change some variables in the file "config/gcc.bldconf" such that the PARSEC command can locate the compiler path correctly.

To compile all programs of the benchmark suite with the default configuration, simply run:

parsecmgmt -a build

Building the whole benchmark suite takes a lot of time, usually 30-60 min depending on your system It is possible to selectively build packages (option '-p') and to choose different build configurations (option '-c'), read Section 4 for a more comprehensive explanation of build configurations and Section 6 for more complex usage examples.

How to Run PARSEC

After you have built the suite, you can use the following command to run all benchmarks with the minimal test input:

parsecmgmt -a run

The test should finish within 5 seconds. Its purpose is to quickly verify that all benchmarks have been built successfully and are executable. Do not use it for performance measurements. You can also choose a different input (option '-i') and alter the number of threads (option '-n'). Section 5 explains the different inputs and Section 6 gives more detailed examples.

How to Get More Information

To query PARSEC about all available packages and features, you can use the following command:

parsecmgmt -a info

If you would like to see information about dynamically created files such as the available builds, you can run:

parsecmgmt -a status

Build Configurations

Besides the regular configuration which can be altered to customize the benchmark suite, PARSEC also uses build configurations. A build configuration is a specific way to compile a program. It determines, for example, what compiler to use and which features of the package to enable. Build configurations should be your first approach to alter how a benchmark program is created. The only limitation is that the interface and output of a benchmark must remain the same, otherwise the program will become incompatible with the PARSEC framework. However, you can adapt the run configuration and reference outputs to eliminate this problem.

PARSEC ships with the following preinstalled build configurations:

'gcc' Build parallel version of suite with gcc
'gcc-serial' Build serial version of suite with gcc
'gcc-hooks' Build parallel version of suite with PARSEC hooks enabled with gcc
'icc' Build parallel version of suite with Intel compiler
'gcc-pthreads' Build with pthreads parallelization (if supported)
'gcc-openmp' Build with OpenMP parallelization (if supported)
'gcc-tbb' Build with TBB parallelization (if supported)

For example, to build PARSEC with enabled hooks, you can use:

parsecmgmt -a build -c gcc-hooks

The three build configurations 'gcc-pthreads', 'gcc-openmp' and 'gcc-tbb' can be used to compile a workload with one of these three parallelization models if it is supported by the program. The build configuration 'gcc' implicitly defines a standard parallelization for each workload by falling back to one of these three configurations for each benchmark program.

We also defined an alias for each parallelization model that will be resolved resolved to a complete list of all workloads that support this parallelization model. They have the same name as the parallelization. For example, to build all workloads which support OpenMP with exactly that parallelization, you can use:

parsecmgmt -a build -p openmp -c gcc-openmp

Additional build configurations can be created with the tool 'bldconfadd'. For example, to add a new configuration 'myconfig' that is based on gcc-serial, you can use the following command:

bldconfadd -n myconfig -s gcc-serial

To remove this configuration, you can use 'bldconfdel' as follows:

bldconfdel -n myconfig

Performance Measurement & Research

For each benchmark, we define a Region-Of-Interest (ROI) which includes the computationally intensive, parallelized phase of the benchmark, but not the initialization or shutdown phase. Instead of measuring the total program runtime, you can use and report the execution time of the ROI for any kind of analysis and comparisons. We provide six inputs for each benchmark program:

'test' Minimal input to verify that programs are executable.
'simdev' Very small input which causes code execution comparable to a typical input for this program. Intended for microarchitectural simulator development.
'simsmall' Small input for performance measurements with microarchitectural simulators
'simmedium' Medium-sized input for performance measurements with microarchitectural simulators
'simlarge' Large-sized input for performance measurements with microarchitectural simulators
'native' Very large input intended for large-scale experiments on real machines

All inputs except 'test' and 'simdev' can be used for performance analysis. As a rough guideline, on a Pentium 4 processor with 3.0 GHz you can expect approximately the following execution times:

'test' almost instantaneous
'simdev' almost instantaneous
'simsmall' ~1s
'simsmall' ~3-5s
'simlarge' ~12-20s
'native' ~10-30min

The exact runtime depends on the program and its inputs and deviates in some cases from the described rule of thumb. Different build configurations and system parameters may also result in a deviation.

PARSEC Hooks

All benchmark programs of the benchmark suite support PARSEC hooks, a library which allows rapid instrumentation of all benchmark. If the benchmarks are compiled with hooks enabled, at various locations in the code the programs will call the corresponding hook function. Among other things, the ROI has been instrumented in such a way, and by default the PARSEC hooks library measures the time spent in the ROI.

Documentation and code of the PARSEC hooks is available in the 'hooks' package in group 'libs'. PARSEC hooks are enabled in the build configuration 'gcc-hooks' and disabled in all other ones.

More Examples

Compatible to PARSEC 2.1

Build 'x264' and 'blackscholes' benchmarks with Intel compilers:

parsecmgmt -a build -p x264 blackscholes -c icc

Build serial version of all kernels:

parsecmgmt -a build -p kernels -c gcc-serial

Run a self-test with all applications:

parsecmgmt -a run -p apps -i test

Use version of 'vips' and all kernels with enabled hooks to measure the performance on a real machine with 32 threads:

parsecmgmt -a run -p vips kernels -c gcc-hooks -n 32

Use SimpleScalar simulator to run a simulation with the 'gcc-hooks' build configuration of the 'freqmine' application with small simulator inputs and 4 threads:

parsecmgmt -a run -c gcc-hooks -s sim-outorder -p freqmine \
+ i simsmall -n 4

Use an executable 'qsub' to submit runs with all kernels with 16 threads and large simulator inputs to a batch system (this requires that a program 'qsub' is installed and in the path which handles the job submission):

parsecmgmt -a run -s qsub -p kernels -i simlarge -n 16

Clean up after a build or benchmarking run:

parsecmgmt -a fullclean -p all

Uninstall 'gcc-serial' build of package 'gsl' and build a new version:

parsecmgmt -a uninstall -p gsl -c gcc-serial
parsecmgmt -a build -p gsl -c gcc-serial

Network benchmarks

Check the status of all components involved in network benchmarks:

    parsecmgmt -a status -p netapps

Build network benchmark 'netstreamcluster':

    parsecmgmt -a build -p raytrace

Build all network benchmark:

    parsecmgmt -a build -p netapps

Run network benchmark 'netdedup' w/ input 'native' and 2 server threads:

    parsecmgmt -a run -p netdedup -i native -n 2

Run network benchmark 'netferret' w/ input 'simlarge', 4 server threads and 2 client connections:

    parsecmgmt -a run -p netferret -i simlarge -n 4 -t 2

For simulation, run 'netdedup' server on a simulator w/ 4 threads and run client on a real machine:

    parsecmgmt -a run -p netdedup -i simlarge -n 4 -m server
    parsecmgmt -a run -p netdedup -i simlarge -m client

Do a full cleanup for network benchmarks:

    parsecmgmt -a fullclean -p netapps

SPLASH-2 Suite and SPLASH-2x Suite

Check the status of SPLASH-2 suite and SPLASH-2x suite:

    parsecmgmt -a status -p splash2
    parsecmgmt -a status -p splash2x

Build benchmark 'raytrace' from SPLASH-2x suite other than PARSEC: parsecmgmt -a build -p splash2x.raytrace parsecmgmt -a build -p raytrace ## defaultly from PARSEC (for comparison)

Build benchmark 'fft' from SPLASH-2 suite with 'gcc-serial' build configuration:

    parsecmgmt -a build -c gcc-serial -p splash2.fft

Build all benchmarks from SPLASH-2 suite and SPLASH-2x suite:

    parsecmgmt -a build -p splash2
    parsecmgmt -a build -p splash2x

Run benchmark 'fft' from SPLASH-2x w/ input 'simsmall' and 4 threads:

    parsecmgmt -a run -p splash2x.fft -i simsmall -n 4

Do a full cleanup for SPLASH-2 suite

    parsecmgmt -a fullclean -p splash2

Structure

The PARSEC suite is composed of the software packages, which are the benchmark programs and their required libraries and tools, and the framework, which is everything else. Software packages are located in the 'pkgs/' directory. Each software package is part of exactly one package group, which has its own subdirectory. For example, a package named 'foo' which belongs to group 'bar' would be located in the directory 'pkgs/bar/foo/'.

PARSEC has the following structure:

bin/        directory with PARSEC tools
config/     global configuration
log/        log files of builds and runs (dynamically created)
man/        man pages of the PARSEC distribution
pkgs/       package groups which contain the packages
version     file with version number of PARSEC distribution

A package has the following directory structure:

inputs/     directory with input archives (optional)
inst/       installation directory (dynamically created)
obj/        build directory (dynamically created)
outputs/    directory with reference outputs (optional)
parsec/     PARSEC configuration files
run/        directory for program execution (dynamically created)
src/        source code of the package
version     file with package version

Some of these directories will be auto-generated by 'parsecmgmt' on the fly.

Each package can have multiple builds and installations, and 'parsecmgmt' will use separate subdirectories for them. PARSEC uses the name of the build configuration and the platform for which the program is compiled to form a key that is used to distinguish different builds and installations of a package.

Configuration Files

PARSEC distinguishes between global and local configuration. Global configuration files are located in the 'config/' directory in the root of the benchmark suite. Local configuration files are only valid for a single software package and are stored in the 'parsec/' directory of the package they belong to.

The following types of configuration files exist: 'parsec.conf' is a global file which defines the general structure of the benchmark suite, such as aliases and which software packages exist. Files named '.sysconf' define the basic programs which 'parsecmgmt' is to use on each operating system platform. To port 'parsecmgmt' to a new operating system, a corresponding file has to be created. '.runconf' are configuration files which determine how benchmark programs are to be executed. Finally, files named '*.bldconf' describe the build configuration of a package. Run and build configurations are composed of both global and local configuration files.

Manual Usage

It is possible to build and run PARSEC benchmark programs manually. To do so, the correct build configuration respectively run configuration should be used as defined in the PARSEC configuration files.

The following steps are executed by 'parsecmgmt' to build a package:

Set variable PARSECDIR to the installation root of PARSEC
Set variable PARSECPLAT to the build key used to identify the platform
Source system configuration
Source global build configuration
Source local build configuration
Create build directory and change to it
If it exists, modify build environment and call configure script
Modify build environment and execute 'make'
Modify build environment and execute 'make install'

The following steps are executed by 'parsecmgmt' to run a benchmark:

Set variable PARSECDIR to the installation root of PARSEC
Set variable PARSECPLAT to the build key used to identify the platform
Source system configuration
Source global run configuration
- Source local run configuration
Create run directory and cd to it
Unpack desired input
Execute benchmark with parameters from run configuration

Contact

You can reach the PARSEC team as follows:

<http://parsec.cs.princeton.edu/>
parsec@lists.cs.princeton.edu

bamos / parsec-benchmark

readme