Benchmark reproduction issues

[vincedani]

Hello,

I would like to run the benchmark as it is described in the README.md:

cd benchmark
./start_docker.sh

(inside docker) cd perses
(inside docker) bazel test benchmark/...

After these commands, I got the following output:

Executed 19 out of 56 tests: 13 tests pass, 6 fail locally and 37 were skipped.
FAILED: Build did NOT complete successfully

After this test, I tried steps described in benchmark/README.md and run the benchmark.py:

 (inside docker)  ./benchmark.py --benchmark-all

With this command, the following tests returned with error:

• clang-26350
• clang-31259
• gcc-59903
• gcc-60116
• gcc-61383
• gcc-61917
• gcc-64990
• gcc-65383
• gcc-66186
• gcc-66375
• gcc-70127
• gcc-70586

The tests cases failed with this error message:

[INFO   ] Reduction algorithm is org.perses.reduction.reducer.PersesNodePrioritizedDfsReducer
[SEVERE ] The initial sanity check failed. Folder: /tmp/...
[SEVERE ] The files have been saved to /tmp/...
Exception in thread "main" java.lang.IllegalStateException
        at org.perses.reduction.ReductionDriver.sanityCheck(ReductionDriver.kt:277)
        at org.perses.reduction.ReductionDriver.reduce(ReductionDriver.kt:107)
        at org.perses.Main.main(Main.java:65)

The clang-27137 failed with out of memory error. (16 GB)

[chengniansun]

The benchmark scripts are work in progress. The docker container is provided with all required dependencies so that people can run each subject used in our Perses paper.

Sorry that bazel test benchmark/... failed, because some of my students added go and rust subjects which do not have required dependencies inside the docker. If you want to reproduce the results in the original paper, you need to run each individual folder yourself.

[chengniansun]

If you want to run clang-26350, you can cd into this folder, and run ./r.sh to test whether the script works. Then you can use the r.sh and small.c for benchmarking.

Sorry for the late reply. The notification of this repo seems not working for me, and I did not see the notification message in my email box.

[chengniansun]

I filed an internal bug #84 to track this issue. Thank you for reporting this.

[chengniansun]

@XueyanZhang fixed this one.