Hi, I have recently been using Code2flow for an empirical study to detect non-deterministic behaviors in static analyzers. The experiments resulted in discovering some nondeterministic analysis results across multiple runs under various configurations of Code2flow.
The details of the experimental setup are as below:
The experiments were conducted on a micro-benchmark and a real-world benchmark used in the ICSE 2021 paper
The experiments were conducted under 4 sample configurations which were generated using a 2-way covering array from the configuration space.
The timeout set for Code2flow running on both micro-benchmark and macro-benchmark was 5 minutes/program.
We ran Code2flow on each program-configuration combination 10 times and compared the results across 10 runs for detecting non-deterministic behaviors.
All experiments were conducted in docker containers. The hardware environment is a server with 128GB of RAM and 24 Intel Xeon Silver 4116 CPUs@2.10GHz running Ubuntu 16.04.
In the end, the experiments detected non-deterministic results on 2 programs, which were all from micro-benchmarks. These results were observed under all the sampled configurations.
Hi, I have recently been using Code2flow for an empirical study to detect non-deterministic behaviors in static analyzers. The experiments resulted in discovering some nondeterministic analysis results across multiple runs under various configurations of Code2flow.
The details of the experimental setup are as below:
The experiments were conducted on a micro-benchmark and a real-world benchmark used in the ICSE 2021 paper
The experiments were conducted under 4 sample configurations which were generated using a 2-way covering array from the configuration space.
The timeout set for Code2flow running on both micro-benchmark and macro-benchmark was 5 minutes/program.
We ran Code2flow on each program-configuration combination 10 times and compared the results across 10 runs for detecting non-deterministic behaviors.
All experiments were conducted in docker containers. The hardware environment is a server with 128GB of RAM and 24 Intel Xeon Silver 4116 CPUs@2.10GHz running Ubuntu 16.04.
In the end, the experiments detected non-deterministic results on 2 programs, which were all from micro-benchmarks. These results were observed under all the sampled configurations.
The attached data is the detected nondeterministic results from micro-benchmark and macro-benchmark and configuration files (note1: the configurations are hash-coded in the detected results, but the actual configuration options and values that each hash code stands for can be found in the attached configuration files.)