Implementation of the Test Automation for Incremental Analysis - TAIA as part of the bachelor's thesis "Automatic Generation of Test Cases for Incremental Static Analysis". The script enables the user to create and run incremental tests for Goblint based on mutations generated on a single c file provided by the user. With this, you can create with ease thousands of incremental tests based on the .c files from the normal regression tests. This script requires some changes to the analyzer repository, which are implemented in this pull request.
Project Structure
TAIA-test-automation-for-incremental-analysis Contains all components of the "Test Automation for Incremental Analysis - TAIA"
├── README.md Start point for using the "Test Automation for Incremental Analysis - TAIA"
├── RUN.sh Script for creating and running incremental tests based on a .c file
├── RUN_BATCH.sh Script for creating and running incremental tests on a directory
├── clang-mutations Directory containing the clang-tidy checks needed for the mutation generation
│ ├── MUTATIONS.md│ ├── README.md├── sample-files Directory containing files you can use as input files for getting started
│ ├── batch-ignore-files Directory with ignore-files you want to use when running on all regression tests
├── scripts Directory with all scripts for the Mutation Generator
│ ├── README.md Information about the scripts
Implementation
For creating and running incremental tests with the "Test Automation for Incremental Analysis - TAIA" the following steps are executed:
Processes the options from the terminal and asks the user interactively for the inputs if the information was not yet provided by the terminal options (run_cli.py)
Generate one mutation using the clang-tidy checks (generate_mutations.py)
Generate all possible mutations and manage them (generate_programs.py)
Add __goblint_check()s to the programs (add_check.py)
Annotate the generated checks (add_check_annotations.py)
Based on the programs with the annotated check create the test files (generate_tests.py)
Run the tests with the update_suite.rb script from the analyzer repository (run_tests.py)
Introduction
Implementation of the Test Automation for Incremental Analysis - TAIA as part of the bachelor's thesis "Automatic Generation of Test Cases for Incremental Static Analysis". The script enables the user to create and run incremental tests for Goblint based on mutations generated on a single c file provided by the user. With this, you can create with ease thousands of incremental tests based on the
.cfiles from the normal regression tests. This script requires some changes to the analyzer repository, which are implemented in this pull request.Project Structure
TAIA-test-automation-for-incremental-analysis Contains all components of the "Test Automation for Incremental Analysis - TAIA" ├── README.md Start point for using the "Test Automation for Incremental Analysis - TAIA" ├── RUN.sh Script for creating and running incremental tests based on a
.cfile ├── RUN_BATCH.sh Script for creating and running incremental tests on a directory ├── clang-mutations Directory containing the clang-tidy checks needed for the mutation generation │ ├── MUTATIONS.md │ ├── README.md ├── sample-files Directory containing files you can use as input files for getting started │ ├── batch-ignore-files Directory with ignore-files you want to use when running on all regression tests ├── scripts Directory with all scripts for the Mutation Generator │ ├── README.md Information about the scriptsImplementation
For creating and running incremental tests with the "Test Automation for Incremental Analysis - TAIA" the following steps are executed:
__goblint_check()s to the programs (add_check.py)update_suite.rbscript from the analyzer repository (run_tests.py)