Test Frameworks

Test frameworks are used to automate the execution of tests and provide additional utilities for testing. They can be used to write and run unit tests, integration tests, and end-to-end tests.

1. Category

1. Category

1.1. GTest

1.1.1. xUnit

Googletest is based on the xUnit testing framework, a popular architecture for unit testing.

Files and Folders

test.cc

Table-Driven Tests

#include "gtest/gtest.h"

TEST(TestSuiteName, TestName) {
EXPECT_EQ(1, 1);
EXPECT_TRUE(true);
}

.gitlab-ci.yml

Generate unit test reports in JUnit XML format using gtest in C/C++.

cpp:
stage: test
script:
  - gtest.exe --gtest_output="xml:report.xml"
artifacts:
  when: always
  reports:
    junit: report.xml

1.1.2. CMake

Google Test, often referred to as GTest, is a C++ testing framework. It provides a powerful and extensible platform for writing and running Unit Tests.

Layout and Structure

Project Layout

Example Project Structure for in C language with CMake and GTest.

/bit
├── /src
│   ├── bit.h
│   ├── bit.c
│   ├── bit_test.cc
│   └── CMakeLists.txt
│
├── CMakeLists.txt
└── CMakePresets.json

Files and Folders

CMakeLists.txt

The top-level CMakeLists.txt file for the project.

cmake_minimum_required(VERSION 3.10)

# set the project name
project(Bit C)

# add the executable
add_executable(BitProject src/bit.c)

add_executable(runTests src/bit_test.cc)
target_link_libraries(runTests gtest_main)

src/CMakeLists.txt

The CMakeLists.txt file for the source directory.
CMakePresets.json

The CMakePresets.json file for the project.

Commands and Operations
- List Presets
  
  List all available presets.
```
cmake --list-presets=all .
```
- Configure Presets
  
  Configure the project with the given preset.
```
cmake --preset <configurePreset-name>
```
- Build Presets
  
  Build the project with the given preset.
```
cmake --build --preset <buildPreset-name>
```
- Execute Tests
  
  Execute the tests using the specified preset.
```
ctest --preset <testPreset-name>
```

1.1.3. Sanitizer

Files and Folders
- CMakePresets.json
  
  TODO

1.1.4. Coverage

Files and Folders

CMakePresets.json

TODO

.gitlab-ci.yml

Generate the coverage output file in Cobertura XML format using gcovr in C/C++.

run tests:
stage: test
script:
  - cd build
  - make test
  - gcovr --xml-pretty --exclude-unreachable-branches --print-summary -o coverage.xml --root ${CI_PROJECT_DIR}
coverage: /^\s*lines:\s*\d+.\d+\%/
artifacts:
  name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHA}
  expire_in: 2 days
  reports:
    coverage_report:
      coverage_format: cobertura
      path: build/coverage.xml

1.1.5. VS Code

Files and Folders
- settings.json
```
"testMate.cpp.test.executables": "{build,Build,BUILD,out,Out,OUT,cmd,bin}/**/*{test,Test,TEST}*",
```
- extension.json
  
  Extension file located in .vscode for unit test execution in VS Code.

1.2. BATS

BATS (Bash Automated Testing System) is a testing framework designed specifically for Bash scripts. BATS adheres to the Test Anything Protocol (TAP), a widely recognized standard for test reporting. TAP enables seamless integration with various testing tools and frameworks.

Concepts and Benefits
- Simplicity
  
  BATS leverages Bash itself, eliminating the need to learn a new language for testing. This makes it particularly accessible for users already familiar with Bash scripting.
- Readability
  
  Tests written in BATS are clear and concise, enhancing their maintainability and understanding for both developers and testers.
- Compliance
  
  BATS adheres to the Test Anything Protocol (TAP), a widely recognized standard for test reporting. This enables seamless integration with various testing tools and frameworks.
- Flexibility
  
  While primarily intended for Bash scripts, BATS can be used to test any Unix program through the execution of standard shell commands within the test cases.
Install and Setup
- Debian/Ubuntu
```
sudo apt update
sudo apt install bats
```
Layout and Structure
- Project Layout
  
  Example Project Structure for in BASH scripts with BATS.
```
/shell
└── /scripts
  ├── util.sh
  └── util_test.sh
```
Examples and Explanations
- util.sh
  
  TODO
```
#!/bin/bash

function greet() {
echo "Hello, $1!"
}

function add() {
local num1="$1"
local num2="$2"
echo $((num1 + num2))
}
```
- util_test.sh
  - Shebang Line: The first line (#!/usr/bin/env bats) specifies the interpreter to be used for running the script, which is the bats executable.
  - Test Cases: Each test case is defined using the @test annotation followed by a descriptive name in quotes.
  - Command Execution: Inside the test case block, commands are used to perform actions and capture their output.
  - Assertions: BATS provides various assertion functions like assert_output and conditional statements like [[ ]] to verify expected outcomes.
```
#!/usr/bin/env bats

source ./util.sh

@test "greet" {
greeting=$(greet "Alice")
assert_output "Hello, Alice!" "$greeting"
}

@test "add" {
result=$(add 2 3)
[[ $result -eq 5 ]]
}
```

1.3. Terraform

1.3.1. Test

Terraform search for .tftest.hcl files within the current configuration and testing directories. Terraform execute the testing run blocks within any testing files in order, and verify conditional checks and assertions against the created infrastructure.

Layout and Structure

tests/
├── integration/
│   └── main.tftest.hcl
└── unit/
    └── variables.tftest.hcl

Commands and Operations
- terraform test
  
  Executes automated integration tests against the current Terraform configuration.
  
  NOTE terraform test command creates real infrastructure and attempt to teardown the testing infrastructure on completion. Monitor the output carefully to ensure this cleanup process is successful.
```
terraform test -test-directory="tests/unit"
```

1.3.2. Sentinel

HashiCorp Sentinel is an Policy as Code (PaC) framework to for fine-grained, logic-based policy decisions.

NOTE Test cases under tests/policy/ are expected to be in test/<policy>/*.[hcl|json] files where <policy> is the name of the policy filename without an extension.

Layout and Structure

.
├── tests/
│   └── policy/
│       ├── deny-public-rdp-acl-rules.sentinel
│       ├── deny-public-ssh-acl-rules.sentinel
│       ├── restrict-all-vpc-traffic-acl-rules.sentinel
│       └── test/
│           ├── deny-public-rdp-acl-rules/
│           │   ├── failure.hcl
│           │   ├── success.hcl
│           │   └── mock/
│           │       ├── mock-tfplan-failure.json
│           │       └── mock-tfplan-success.json
│           │
│           ├── deny-public-ssh-acl-rules/
│           │   ├── failure.hcl
│           │   ├── success.hcl
│           │   └── mock
│           │       ├── mock-tfplan-failure.json
│           │       └── mock-tfplan-success.json
│           │
│           └── restrict-all-vpc-traffic-acl-rules/
│               ├── failure.hcl
│               ├── success.hcl
│               └── mock
│                   ├── mock-tfplan-failure.json
│                   └── mock-tfplan-success.json
│
└── sentinel.hcl

Commands and Operations
- sentinel test
  
  Execute and verify behavior of policies.
```
sentinel test $(find . -name "*.sentinel" -type f)
```
- sentinel apply
  
  Execute the policy file specified by sentinel.hcl configuration file.
```
sentinel apply
```

1.4. JUnit

JUnit is a widely used testing framework for Java. It provides annotations to identify test methods, assertions for verifying expected outcomes, and support for test fixtures.

1.5. NUnit

NUnit is a testing framework for .NET languages like C# and F#. It offers a range of attributes for test setup, teardown, and categorization.

1.6. TestN

TestN is a testing framework for .NET, providing a versatile set of features for unit testing. It supports parallel execution and various testing attributes.

1.7. TestNG

TestNG is a testing framework inspired by JUnit and NUnit but designed for more flexibility and additional features, especially in the context of automated testing.

1.8. PyTest

PyTest is a testing framework for Python. It simplifies test writing and allows for expressive and modular test setups. PyTest can be used for various types of testing, including unit and functional testing.

1.9. Mocha

Mocha is a flexible testing framework for JavaScript and Node.js. It supports asynchronous testing, various reporters, and can be used with assertion libraries like Chai.

1.10. RSpec

RSpec is a testing framework for Ruby. It follows a Behavior-Driven Development (BDD) approach, allowing developers to express expectations about the behavior of their code.

1.11. Selenium

Selenium is a powerful tool for automating web browsers. It provides a framework for writing automated tests of web applications across various browsers and platforms.

Selenium is commonly used in conjunction with testing frameworks like JUnit and TestNG.

Test Frameworks

1. Category
- 1.1. GTest

1. Category

1.1. GTest

Example of Unit Tests in CMake:

Create a directory structure for your project.

Project
├── CMakeLists.txt
└── src
  ├── CMakeLists.txt
  ├── your_test_files.cpp
  └── your_source_files.cpp

The CMakeLists.txt files that include a feature flag to selectively build the tests.

Root CMakeLists.txt:

cmake_minimum_required(VERSION 3.21.0)

# Use vcpkg to manage dependencies (note: should be placed before project()
# call)
set(CMAKE_TOOLCHAIN_FILE
  <PathTo>/vcpkg/scripts/buildsystems/vcpkg.cmake
  CACHE STRING "Vcpkg toolchain file")

# Set project name
project(app)

# Feature flag to build the test source files
option(BUILD_TESTS "Build the test source files" ON)

# Add the GTest library
if(BUILD_TESTS)
find_package(GTest CONFIG REQUIRED)
endif() # BUILD_TESTS

# Include the source directory
add_subdirectory(src)

Use find_package() to locate the Google Test library. This assumes that Google Test is installed using a dependency manager like vcpkg or is built from source and installed on the system.

src/CMakeLists.txt:
```
# Create a library target
add_library(${PROJECT_NAME})

# Set the C++ standard required for the library target
target_compile_features(${PROJECT_NAME} PUBLIC cxx_std_11)

# Add the source files to the library target
target_sources(${PROJECT_NAME} PRIVATE your_source_files.cpp)

# Set the compiler options required for the library target
target_compile_options(${PROJECT_NAME} PRIVATE -Wall -Wextra -pedantic)

# Include directories for the library target
target_include_directories(${PROJECT_NAME} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})

# Optional: Link any additional libraries if needed
# target_link_libraries(${ProjectName} PRIVATE <additionalLibraries>)

# Build and link the tests separately
if(BUILD_TESTS)
# Create a test target
add_executable(${PROJECT_NAME}-test)

# Add the source files to the test target
target_sources(${PROJECT_NAME}-test PRIVATE your_test_files.cpp)

# Link the library and any additional libraries required for the tests
target_link_libraries(
  ${PROJECT_NAME}-test PRIVATE ${PROJECT_NAME} GTest::gmock GTest::gtest
                              GTest::gmock_main GTest::gtest_main)
endif() # BUILD_TESTS
```
The src/CMakeLists.txt file defines a library target and a test target.
- The library target is created using the add_library command, and is given the name ${PROJECT_NAME}. The ${PROJECT_NAME} variable is typically set in the root CMakeLists.txt file, and is used to give the project a name.
- target_compile_features command is used to set the C++ standard required for the library target
- target_sources command is used to add the source files to the library target.
- target_compile_options command is used to set the compiler options required for the library target
- target_include_directories command is used to specify the include directories for the library target. Finally
- target_link_libraries command is used to link any additional libraries if needed.
- The test target is created using the add_executable command, and is given the name ${PROJECT_NAME}-test.
- target_sources command is used to add the source files to the test target
- target_link_libraries command is used to link the library and any additional libraries required for the tests. In this case, the test target is linked with the ${PROJECT_NAME} library, as well as the GTest and GMock libraries.
The tests are conditionally built based on the value of the BUILD_TESTS option. If BUILD_TESTS is enabled (ON), the your_test_files.cpp is used as sources to create a test target (${PROJECT_NAME}-test). The test target is linked with the library target (${PROJECT_NAME}) and the GTest libraries.

By toggling the feature flagBUILD_TESTS when running CMake, you can control whether to build the test executable.

NOTE Adjust the version 3.21 in the cmake_minimum_required command according to your requirements. You can specify a range or a specific version based on the minimum CMake version you want to support.
Build the project using CMake, passing the appropriate flags:

After building the project with BUILD_TESTS set to ON, you can execute the test executable using the following steps:
```
cmake -S . -B build -DBUILD_TESTS=ON
cmake --build build
```
Execute the test executable:
```
./build/src/app-test
```
This command will run the executable generated from the test source files. Ensure that you are running these commands from the root directory of your project. By following these steps, you can first execute the application and then the test executable to verify the functionality and correctness of your code.

NOTE The exact names of the executable (app-test) might vary depending on the specific project. Replace them with the appropriate names based on your project's configuration.

sentenz / convention

Create an article about `Test Frameworks` #222

Test Frameworks

1. Category

1.1. GTest

1.1.1. xUnit

1.1.2. CMake

1.1.3. Sanitizer

1.1.4. Coverage

1.1.5. VS Code

1.2. BATS

1.3. Terraform

1.3.1. Test

1.3.2. Sentinel

1.4. JUnit

1.5. NUnit

1.6. TestN

1.7. TestNG

1.8. PyTest

1.9. Mocha

1.10. RSpec

1.11. Selenium

Test Frameworks

1. Category

1.1. GTest