Detected non-deterministic results under various configurations

[AnnabellaM]

Hi, I have recently been using Soot for an empirical study to detect non-deterministic behaviors in static analyzers. The experiments resulted in discovering some nondeterministic analysis results across multiple runs under various configurations of Soot.

The details of the experimental setup are as below:

• The experiments were conducted on the micro-benchmark CATS and a real-world benchmark DaCapo-2006.

• The experiments were conducted under 30 sample configurations which were generated using a 2-way covering array from the configuration space.

• The timeout set for Soot running on each CATS program was 15 minutes. For running on each DaCapo-2006 program, the timeout was set to 2 hours.

• We ran Soot on each program-configuration combination 5 times and compared the results across 5 runs for detecting non-deterministic behaviors.

• All experiments were conducted in docker containers. The hardware environment is a server with 128GB of RAM and 24 Intel Xeon Silver 4116 CPUs@2.10GHz running Ubuntu 16.04.

In the end, the experiments detected non-deterministic results on 75 programs. 68 out of these 75 programs were from the CATS, and 7 out of 75 were from the DaCapo-2006 programs. These results were detected under 19 configurations out of a total of 30 sample configurations.

The attached data is the detected nondeterministic results from CATS and DaCapo-2006 and configuration files (note1: the configurations are hash-coded in the detected results, but the actual configuration options and values that each hash code stands for can be found in the attached configuration files.) (note2: Since the entire dataset for detected nondeterministic results amounts to 3.1 GB, the attached data is only a fraction of the total dataset.)

[bkushigian]

Interesting, I also just ran Soot a few times on some programs and detected some nondeterminism. I figured it might be timeout-relate?