This is a simple and easy extensible benchmark system to answer the question, which FFT library performs best under which conditions. Conditions are given by compute architecture, inplace or outplace as well as real or complex transforms, data precision, and so on. This project is still in development.
If you want to just browse our results, see the raw benchmark data or our online visualisation and comparison tool.
gearshifft
was published at ISC2017. The respective preprint is available on the arxiv.
-DGEARSHIFFT_CXX11_ABI=OFF
cmake option¹) The respective FFTW-style wrapper library is required for the integration in gearshifft.
Go to the gearshifft
directory (created by git clone
):
mkdir release && cd release
cmake ..
make -j 4
CMake tries to find the libraries and enables the corresponding make targets.
After make
is finished, you can run e.g. ./gearshifft/gearshifft_cufft
.
If the FFT library paths cannot be found, CMAKE_PREFIX_PATH
has to be used, e.g.:
export CMAKE_PREFIX_PATH=${HOME}/software/clFFT-cuda8.0-gcc5.4/:/opt/cuda:$CMAKE_PREFIX_PATH
Enable float16 support for cuFFT:
cmake -DGEARSHIFFT_FLOAT16_SUPPORT=1 ..
make gearshifft_cufft # automatically downloads half library
gearshifft's dependencies can be build automatically by using the cmake's superbuild options.
Superbuild mode is currently OFF by default, so you have to enable it via cmake option GEARSHIFFT_USE_SUPERBUILD=ON
.
An example for downloading and building Boost, clFFT and FFTW::
cmake -DGEARSHIFFT_USE_SUPERBUILD=ON \
-DGEARSHIFFT_SUPERBUILD_EXT_INBUILD=OFF \
-DGEARSHIFFT_SUPERBUILD_EXT_DIR=$HOME/sources/deps \
-DGEARSHIFFT_SUPERBUILD_EXT_DOWNLOAD_Boost=ON \
-DGEARSHIFFT_SUPERBUILD_EXT_DOWNLOAD_CLFFT=ON \
-DGEARSHIFFT_SUPERBUILD_EXT_DOWNLOAD_FFTW=ON \
-DGEARSHIFFT_SUPERBUILD_EXT_DOWNLOAD_ROCFFT=OFF \
-DGEARSHIFFT_USE_STATIC_LIBS=ON \
..
Notes:
GEARSHIFFT_SUPERBUILD_EXT_INBUILD=OFF
installs the dependency builts into a separate directory ($GEARSHIFFT_SUPERBUILD_EXT_DIR
), otherwise builts are located in the binary directory ($CMAKE_BINARY_DIR}
).GEARSHIFFT_USE_STATIC_LIBS=OFF
then you have to take care of environment setting such as LD_LIBRARY_PATH
GEARSHIFFT_USE_STATIC_LIBS=ON
probably requires to build Boost manually
BOOST_TEST_DYN_LINK
definition (automatically done in our cmake)GEARSHIFFT_VERBOSE=ON
to get more information during the cmake's build generation processIf the libraries have been build separately, just provide the correct path. An example:
BOOST_VER=1.67.0
FFTW_VER=3.3.8
BOOST_ROOT=${HOME}/sw/boost-${BOOST_VER}
FFTW_ROOT=${HOME}/sw/fftw-${FFTW_VER}/
export CMAKE_PREFIX_PATH==${BOOST_ROOT}/lib:${BOOST_ROOT}/include:${FFTW_ROOT}:${CMAKE_PREFIX_PATH}
if [ -d "CMakeFiles" ]; then
make clean
rm -rf CMakeFiles CMakeCache.txt cmake_install.cmake Makefile
fi
cmake ..
make
Use cmake superbuild or follwing compile script.
Use cmake superbuild or follwing compile script.
The following section illustrates how to build the benchmark with Score-P user instrumentation and how to make sense of the measurements. Note that you need a working Score-P installation available in your shell session.
The benchmark can also be configured to flush the cashes before each plan and execution step.
You can enable this option by passing -DGEARSHIFFT_FLUSH_CACHE=On
to cmake
.
The attempt to flush the caches is made by writing/reading to/from a volatile char
array four
times the size of the last level cache.
You can set the cache and cache line sizes of your hardware by defining the variables
GEARSHIFFT_FLUSH_LLC_SIZE_MEBIBYTES
(last level cache size in MiB, default: 32) and
GEARSHIFFT_FLUSH_CL_SIZE_BYTES
(cache line size in B, default: 64).
This feature is still experimental.
Set CMAKE_INSTALL_PREFIX
as you wish, otherwise defaults are used.
mkdir release && cd release
cmake -DCMAKE_INSTALL_PREFIX=${HOME}/gearshifft ..
make -j 4 install
Superbuild mode in cmake can be used to automatically download and compile the dependencies. It is also intended for creating packages together with static linking of the dependencies. It also supports multi-arch packaging of gearshifft targets, which have been compiled with different compiler.
cmake -DGEARSHIFFT_USE_SUPERBUILD=ON ..
make -j
make gearshifft-package
The tests can be executed by make test
after you have compiled the binaries.
cmake -DBUILD_TESTING=ON ..
make -j
cd gearshifft-build/ # because superbuild generates a build subfolder for gearshifft
make test
See help message (pass --help|-h
) for the command line options.
-h [ --help ] Print help messages
-e [ --extent ] arg Specific extent (eg. 1024x1024) [>=1 nr. of
args possible]
-f [ --file ] arg File with extents (row-wise csv) [>=1 nr.
of args possible]
-o [ --output ] arg (=result.csv) output csv file, will be overwritten!
-t [ --add-tag ] arg Add custom tag to header of output file
-v [ --verbose ] Prints benchmark statistics
-V [ --version ] Prints gearshifft version
-d [ --device ] arg (=gpu) Compute device = (gpu|cpu|acc|<ID>). If
device is not supported by FFT lib, then it
is ignored and default is used.
-n [ --ndevices ] arg (=0) Number of devices (0=all), if supported by
FFT lib (e.g. clfft and fftw with n CPU
threads).
-l [ --list-devices ] List of available compute devices with IDs,
if supported.
-b [ --list-benchmarks ] Show registered benchmarks
-r [ --run-benchmarks ] arg Run specific benchmarks (wildcards
possible, e.g. ClFFT/float/*/Inplace_Real)
... further backend specific options ...
gearshifft
comes with preset files which contain a broad range of FFT extents and are located in:
/gearshifft/share/gearshifft/*.conf
For FFTW benchmarks with, e.g., FFTW_MEASURE
runtimes grow very quickly. You can delete or comment out the unwanted lines in the configuration file with #
.
The extents configurations are loaded by the gearshifft
command-line interface:
./gearshifft_fftw -f myextents.conf
Runs complete benchmark for clFFT (also applies for cuFFT, FFTW, ..)
./gearshifft_clfft
# equals
./gearshifft_clfft -f myextents.conf -o result.csv -d gpu -n 0
List compute devices
./gearshifft_clfft -l
$ "ID",0:0,"ClFFT Informations","Device","Tesla K20Xm","Hardware","OpenCL 1.2 CUDA" <snip>
$ "ID",1:0,"ClFFT Informations","Device","Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz" <snip>
Select compute devices by id returned by --list-devices|-l
./gearshifft_clfft -d 1:0
512x512-point FFT, single and double precision, all transforms, print result to console.
./gearshifft_clfft -e 512x512 -v
1048576-point FFT, only real outplace transform in single precision.
./gearshifft_clfft -e 1048576 -r */float/*/Outplace_Real
1024-point and 128x256x64-point FFTs, on CPU with 1 thread.
./gearshifft_clfft -e 1024 128,256,64 -d cpu -n 1
1024x1024-point FFT, double precision inplace transforms.
--list-benchmarks|-b
gives a list of available extents read in --file|-f
(default is ../config/extents.csv)
./gearshifft_clfft -r */double/1024x1024/Inplace_*
gearshifft_fftw
offers some additional options, e.g., for the plan rigor or the time limit of plans, just check ./gearshifft_fftw -h
.
FFTW itself allows you to pregenerate FFT plans by the so-called wisdom files. These can be generated by FFTW tools, e.g.:
./fftw-3.3.5_single/tools/fftwf-wisdom -v -c -n -T 24 -o wisdomf # single precision
./fftw-3.3.5_double/tools/fftw-wisdom -v -c -n -T 24 -o wisdom # double precision
It is recommended to compile double-precision (fftw-...) and single-precision (fftwf-...) to separate directories to avoid linker issues.
The wisdoms settings must match the gearshifft_fftw
configuration (number of cores, precision, extents).
Most of the times you do not benefit from a multi-core setting, because the FFT is already computed almost in no time.
FFTW spends a lot of time in planning, except you use FFTW_ESTIMATE
or time limits, which are also only lower borders.
The FFT scenario is a roundtrip FFT, i.e. forward and backward transformation. The result is compared with the original input data and an error is shown, if there was a mismatch. If a benchmark cannot be completed due to an error, it proceeds with the next benchmark. The library dependent FFT steps are abstracted, where following steps are wrapped by timers.
Furthermore, the required buffer sizes to run the FFT are recorded.
The results of the benchmark runs are stored into a comma-separated values file (.csv), after the last run has been completed. To ease evaluation, the entries are sorted by columns
See CSV header for column titles and meta-information (memory, number of runs, error-bound, hostname, timestamp, ...).
During runtime, the results are stored in a backup file (suffixed with "~") in the order in which
they occur.
The benchmark will accumulate a number of results before writing them to disk.
This number can be set at compile time by defining GEARSHIFFT_DUMP_FREQUENCY
.
The default value is 1 which means every result will be written right away.
cmake
(say 2.8.12.2), use cmake -DBoost_ADDITIONAL_VERSIONS=1.62.0 -DBOOST_ROOT=/path/to/boost/1.62.0 <more flags>
@InProceedings{10.1007/978-3-319-58667-0_11,
author="Steinbach, Peter
and Werner, Matthias",
editor="Kunkel, Julian M.
and Yokota, Rio
and Balaji, Pavan
and Keyes, David",
title="gearshifft -- The FFT Benchmark Suite for Heterogeneous Platforms",
booktitle="High Performance Computing",
year="2017",
publisher="Springer International Publishing",
address="Cham",
pages="199--216",
abstract="Fast Fourier Transforms (FFTs) are exploited in a wide variety of fields ranging from computer science to natural sciences and engineering. With the rising data production bandwidths of modern FFT applications, judging best which algorithmic tool to apply, can be vital to any scientific endeavor. As tailored FFT implementations exist for an ever increasing variety of high performance computer hardware, choosing the best performing FFT implementation has strong implications for future hardware purchase decisions, for resources FFTs consume and for possibly decisive financial and time savings ahead of the competition. This paper therefor presents gearshifft, which is an open-source and vendor agnostic benchmark suite to process a wide variety of problem sizes and types with state-of-the-art FFT implementations (fftw, clFFT and cuFFT). gearshifft provides a reproducible, unbiased and fair comparison on a wide variety of hardware to explore which FFT variant is best for a given problem size.",
isbn="978-3-319-58667-0"
}