pLiner

pLiner is a framework that helps programmers identify locations in the source of numerical code that are highly affected by floating-point compiler optimizations.

Compiler optimizations can alter significantly the numerical results of scientific computing applications. When numerical results differ significantly between compilers, optimization levels, and floating-point hardware, these numerical inconsistencies can impact programming productivity. pLiner is a framework that helps programmers identify locations in the source code that are highly affected by compiler optimizations. pLiner uses a novel approach to identify such code locations by enhancing the floating-point precision of variables and expressions. Using a guided search to locate the most significant code regions, pLiner can report to users such locations at different granularities, file, function, and line of code. pLiner is implemented as a clang tool. Currently pLiner only supports C/C++.

Getting Started

Requirements and Setup

There are 4 options to start using pLiner:

Download the docker image from DockerHub and run the container
Building the docker image through dockerfile and running the container
Building pLiner as a standalone tool
Building pLiner in the source tree of clang/LLVM

Requirements to run pLiner as a docker container

Docker client needs to be installed in the host machine before starting to setup pLiner using option 1 and 2.

Option 1: Download the docker image from DockerHub and run the container.

Download the docker image from DockerHub:

docker pull ucdavisplse/llnl_pliner:latest

Run the docker container

docker run -it ucdavisplse/llnl_pliner:latest /bin/bash

Option 2: Building the docker image through dockerfile and running the container

Clone the pLiner github repo

git clone https://github.com/llnl/pLiner.git

Building the docker image from dockerfile:
```
cd pLiner
docker build -t <tag-name> .
```
Run the docker container
```
docker run -it <tag-name> /bin/bash
```

Requirements to use pLiner through manual setup(Option3 or Option 4).

Installing clang/LLVM compiler is a prerequisite to use pLiner. So far, we have tested pLiner on clang/LLVM 9.0.1.
pLiner uses nlohmann::json to parse json files in C/C++. Download file json.hpp from https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json.hpp (version 3.5.0) and place it in the directory pLiner/clang-tool before using pLiner.

Option 3: building pLiner as a standalone tool

Clone pLiner and build it:

git clone https://github.com/llnl/pLiner.git
cd pLiner/clang-tool
mkdir build; cd build
cmake ..
make

Install pLiner
```
make install
```
Or, export path to pLiner (this command may differ depending on shell):
```
export PATH=$PATH-TO-pLiner/clang-tool/build:$PATH
```
Option 4: building pLiner in the source tree of clang/LLVM

Building clang/LLVM 9.0.1:

git clone https://github.com/llvm/llvm-project.git clang-llvm
git checkout llvmorg-9.0.1
cd ~/clang-llvm
mkdir build && cd build
cmake -G Ninja ../llvm -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" -DLLVM_BUILD_TESTS=ON
ninja
ninja check       # Test LLVM only.
ninja clang-test  # Test Clang only.
ninja install

Note: Refer to https://clang.llvm.org/docs/LibASTMatchersTutorial.html in case you need instructions for installing cmake and/or ninja.

Clone pLiner in the clang-tools-extra directory and build it:

cd ../clang-tools-extra
git clone https://github.com/llnl/pLiner.git
echo "add_subdirectory(pLiner/clang-tool)" >> CMakeLists.txt
cp pLiner/clang-tool/CMakeLists.txt-insource pLiner/CMakeLists.txt
cd ../build
ninja

Export path to pLiner (this command may differ depending on shell):
```
export PATH=$PATH-TO-CLANG-LLVM/build/bin:$PATH
```

Run optional unit tests

  cd pLiner/tests
  ./test.sh

If pliner works as expected, all unit tests would pass.

Running pLiner

Running pLiner on a test program

We use a simple C program vtest.c to show how to use pLiner. This program was generated by a floating-point random program generator, Varity, and it produces inconsistent results when compiled with gcc -O3 -ffast-math compared to gcc -O0. pLiner isolated a line of the code in the source (Line 25) as the origin of the compiler-induced inconsistency. pLiner also provided a transformed version of the program vtest_trans.c, in which the isolated line of the code has been transformed to higher precision, and the transformed program produces consistent results between gcc -O3 -ffast-math and gcc -O0.

Change to the example directory:
```
cd pLiner/example
```

Compile the original vtest.c program with both gcc -O3 -ffast-math and gcc -O0, and compare the results:

make
./cmp.sh vtest

vtest_O0 corresponds to the executable generated by compiling vtest.c with gcc -O0, and vtest_O3 corresponds to the executable generated by compiling vtest.c with gcc -O3 -ffast-math.

The expected output when comparing the results should be:

./vtest_O0 +1.8768E-306 5 -1.3896E-307 +1.2460E-307 -1.4722E306 -0.0 -1.7470E-322 +1.7072E-307 -1.9009E-307 +1.6022E137 +1.3969E306 +1.8813E34 -1.9422E-99 -1.2666E305 +0.0 -1.2316E-314 +1.5805E-323 +1.7072E208 +1.9220E-307 +1.6811E-306
1.7071999999999999e+208
./vtest_O3 +1.8768E-306 5 -1.3896E-307 +1.2460E-307 -1.4722E306 -0.0 -1.7470E-322 +1.7072E-307 -1.9009E-307 +1.6022E137 +1.3969E306 +1.8813E34 -1.9422E-99 -1.2666E305 +0.0 -1.2316E-314 +1.5805E-323 +1.7072E208 +1.9220E-307 +1.6811E-306
-1.8508999968058596e-316

Note that the difference between vtest_O0 and vtest_O3 is very large. We will use pLiner to diagnose the root cause of the numerical difference.

Run pLiner:

python ../scripts/search.py vtest.c "--"

The first argument vtest.c is the input program; the second argument "--" indicates that to compile the input program there are no header files/librares to specify in the compilation command. Additionally, if there are any such compilation options such as "-I $PATH-TO-HEADERS", specify them following "--" in the second argument, e.g., "-- -I $PATH-TO-HEADERS". Use --help to check for the details of the arguments.

Following is the output.

...
The following areas are transformed to high precision:
compute :
20 -> 22
23 -> 23
failed
The following areas are transformed to high precision:
compute :
25 -> 25
26 -> 26
28 -> 28
success
The following areas are transformed to high precision:
compute :
25 -> 25
success
Search for lines:
The following areas are transformed to high precision:
compute :
25 -> 25
success

Bug area:
compute :
line  25

pLiner found the root cause of the inconsistency (function compute, line 25):
compute :
line 25
pLiner generated a transformed program vtest_trans.c.
```
ls *.c
vtest.c vtest_trans.c
```

Compile the transformed vtest_trans.c program with both gcc -O3 -ffast-math and gcc -O0, and compare the results:

make
./cmp.sh vtest_trans

The expected output when comparing the results should be:

./vtest_trans_O0 +1.8768E-306 5 -1.3896E-307 +1.2460E-307 -1.4722E306 -0.0 -1.7470E-322 +1.7072E-307 -1.9009E-307 +1.6022E137 +1.3969E306 +1.8813E34 -1.9422E-99 -1.2666E305 +0.0 -1.2316E-314 +1.5805E-323 +1.7072E208 +1.9220E-307 +1.6811E-306
1.7071999999999999e+208
./vtest_trans_O3 +1.8768E-306 5 -1.3896E-307 +1.2460E-307 -1.4722E306 -0.0 -1.7470E-322 +1.7072E-307 -1.9009E-307 +1.6022E137 +1.3969E306 +1.8813E34 -1.9422E-99 -1.2666E305 +0.0 -1.2316E-314 +1.5805E-323 +1.7072E208 +1.9220E-307 +1.6811E-306
1.7071999999999999e+208

Both vtest_trans_O0 and vtest_trans_O3 produce 1.7071999999999999e+208. The results are consistent with vtest_O0.

Running pLiner on all Varity programs and comparing results with reference (approx 3 mins).

Navigate to pLiner/benchmarks/Varity/Varity-intel and run the below command to run pLiner on all 50 Varity programs.

python3 run_varity_programs.py

Run the below command to compare the results from pLiner with the reference Varity results and to get a summary for all 50 programs.

python3 compare_varity_results.py

The expected results summary for Varity programs

Results summary
Total Number of programs matched: 50
Total Number of programs which did not match: 0
Total Number of programs for which pLiner was unable to remove inconsistency: 4
List of programs for which pLiner was unable to remove inconsistency
[12, 32, 46, 47]

Running pLiner on NPB programs and comparing results with reference (approx 1 hour).

Navigate to pLiner/benchmarks/NPB/npb-debug and run the below command to run pLiner for three NPB programs CG.B, SP.A and SP.B.

python3 run_npb_programs.py

Run the below command to compare the results from pLiner with the reference NPB results.

python3 compare_npb_programs.py

The expected results summary for NPB programs

CG.B results:
    Results match
         Isolated region granularity: functions
         Function Name: sparse
SP.A results:
    Results match
         File Name: y_solve.c 
         Isolated region granularity: lines
         Function Name: y_solve
         Lines isolated: [[68, 68]]
SP.B results:
    Results match
         File Name: exact_solution.c 
         Isolated region granularity: lines
         Function Name: exact_solution
         Lines isolated: [[44, 44]]

Note

For SP program with input class A, the results are different compared to what is mentioned in the pLiner paper. Please refer to results inside SP.A folder for the updated results.

Use pLiner for your programs

You can follow the instructions as shown in the example to run pLiner for your own programs. Specifically,

Specify the compiler, compilation options that induce inconsistent results, and an error threshold in run.sh
In the example above, those are specified in run.sh as
```
CC="/usr/bin/gcc"
CFLAGS=" -O0 -g -std=c99"
CFLAGS_trouble=" -O3 -ffast-math -g -std=c99"
THRESHOLD=8 
```
CC specifies the compiler; CFLAGS specifies the compilation options that are used to produce ground-truth results and CFLAGS_trouble specifies the compilation options that induce inconsistent results; lastly, THRESHOLD specifies the number of digits that are required to be same with the ground truth for consistency check.
Specify your program file in the first argument, and the compilation options needed to compile the program in the second argument such as python pLiner/scripts/search.py vtest.c "--" in the example.

License

pLiner is distributed under the terms of the Apache-2.0 with LLVM-exception license. All new contributions must be made under this license.

See LICENSE and NOTICE for details.

LLNL-CODE-812209

LLNL / pLiner

readme