Not sure if parametric testing is very popular or not. I'd like to use all these w/ the ceedling test explorer extension.
Issues:
When using the test explorer having tests using the TEST_RANGE macro causes the explorer to freeze and not allow the panel to run tests or update.
The test test_parametric_arguments_inclusive_range() is not showing in panel. The panel is frozen in this picture, and the spinner is constantly reloading.
If I comment out TEST_RANGE tests and enable TEST_MATRIX() tests, the explorer will show the tests and allow the tests to run, but the Test Status will be gray and never update to pass or fail. Also it will only show as a single test, not a list of the matrix tests.
In ceedling, these tests do all pass via the command line when all enabled.
Expected:
At least be able to run all tests with the test explorer even with test using the TEST_RANGE() macro
At least be able to see pass/fail status of the tests w/ TEST_RANGE and TEST_MATRIX tests. All or nothing pass/fail.
showing all the sub-parametric tests would be a plus and beneficial. But secondary.
To Reproduce:
Ceedling 0.31.0 -- ruby Gem install
Unity v2.6.0 -- manually updated by grabbing it from the latest release.
windows -> wsl / ubuntu w/ docker dev container running
# Notes:
# Sample project C code is not presently written to produce a release artifact.
# As such, release build options are disabled.
# This sample, therefore, only demonstrates running a collection of unit tests.
:project:
:use_exceptions: FALSE
:use_test_preprocessor: TRUE
:use_preprocessor_directives: TRUE
:use_auxiliary_dependencies: TRUE
:build_root: build
# :release_build: TRUE
:test_file_prefix: test_
:which_ceedling: vendor/ceedling
:ceedling_version: 0.31.1
:default_tasks:
- test:all
:unity:
:includes:
- parametric_macros.h
:use_param_tests: true
:test_runner:
:use_param_tests: true
:test_build:
:use_assembly: TRUE
:release_build:
:output: MyApp.out
:use_assembly: FALSE
:environment:
:extension:
:executable: .out
:paths:
:test:
- +:test/unit_tests/**
- -:test/support
:source:
- src/**
:support:
- test/support
:libraries: []
:defines:
# in order to add common defines:
# 1) remove the trailing [] from the :common: section
# 2) add entries to the :common: section (e.g. :test: has TEST defined)
:common: &common_defines []
:test:
- *common_defines
- CEEDLING_UNIT_TEST
- UNITY_SUPPORT_TEST_CASES
:test_preprocess:
- *common_defines
- CEEDLING_UNIT_TEST
- UNITY_SUPPORT_TEST_CASES
:cmock:
:mock_prefix: mock_
:when_no_prototypes: :warn
:enforce_strict_ordering: TRUE
:plugins:
- :ignore
- :callback
:treat_as:
uint8: HEX8
uint16: HEX16
uint32: UINT32
int8: INT8
bool: UINT8
# Add -gcov to the plugins list to make sure of the gcov plugin is found
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
:utilities:
- gcovr
:reports:
- HtmlDetailed
:gcovr:
:html_medium_threshold: 75
:html_high_threshold: 90
#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use
# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
:placement: :end
:flag: "-l${1}"
:path_flag: "-L ${1}"
:system: [] # for example, you might list 'm' to grab the math library
:test: []
:release: []
# :module_generator:
# :project_root: ./
# :source_root: src/
# :test_root: test/unit_tests/
# :boilerplate_files:
# :src: '<template_folder>\src_boilerplate.c'
# :inc: '<template_folder>\inc_boilerplate.h'
# :tst: '<template_folder>\tst_boilerplate.c'
# :xml_tests_report:
# :artifact_filename: xml_report.xml
:plugins:
:load_paths:
- vendor/ceedling/plugins
:enabled:
- stdout_pretty_tests_report
- raw_output_report
- xml_tests_report
# - gcov
# - module_generator
My test file:
// /****************************************
// File Name: Tests
// *************************************** */
#include <stdlib.h>
#include "unity.h"
#include "parametric_macros.h"
#include "stdbool.h"
void setUp(void) {
}
void tearDown(void) {
}
void test_hello_world(void)
{
bool myvar = true;
TEST_ASSERT_EQUAL(myvar, true);
}
TEST_CASE(1)
TEST_CASE(3)
void test_parametric_arguments(uint8_t my_param)
{
printf("testing parametric arguments: %d", my_param);
TEST_ASSERT_TRUE_MESSAGE(my_param, "I print if assert is false");
}
TEST_RANGE([0, 2, 1])
void test_parametric_arguments_inclusive_range(uint8_t my_param)
{
printf("testing parametric arguments: %d", my_param);
// make a little debug message:
char *debug_msg = (char*)malloc(128 * sizeof(char));
sprintf(debug_msg, "my_param: %d", my_param);
// assert:
TEST_ASSERT_TRUE_MESSAGE(my_param < 5, debug_msg);
}
TEST_RANGE(<0,1,1>)
void test_parametric_arguments_exclusive_range(uint8_t my_param)
{
// make a little debug message:
char *debug_msg = (char*)malloc(128 * sizeof(char));
sprintf(debug_msg, "my_param: %d", my_param);
TEST_ASSERT_TRUE_MESSAGE(my_param < 1, debug_msg);
}
TEST_MATRIX([3, 4, 7])
void test_parametric_arguments_with_test_matrix(uint8_t my_param)
{
// make a little debug message:
char *debug_msg = (char*)malloc(128 * sizeof(char));
sprintf(debug_msg, "my_param: %d", my_param);
printf("I AM HERE AND RUNNING");
TEST_ASSERT_TRUE_MESSAGE(my_param < 8, debug_msg);
}
I installed ceedling 0.31.0 and upgraded the unity version to 2.6.0 to support the TEST_CASE, TEST_RANGE, and TEST_MATRIX parametric testing features. https://github.com/ThrowTheSwitch/Ceedling/issues/241#issuecomment-2049916878
Not sure if parametric testing is very popular or not. I'd like to use all these w/ the ceedling test explorer extension.
Issues:
When using the test explorer having tests using the TEST_RANGE macro causes the explorer to freeze and not allow the panel to run tests or update.
The test test_parametric_arguments_inclusive_range() is not showing in panel. The panel is frozen in this picture, and the spinner is constantly reloading.
If I comment out TEST_RANGE tests and enable TEST_MATRIX() tests, the explorer will show the tests and allow the tests to run, but the Test Status will be gray and never update to pass or fail. Also it will only show as a single test, not a list of the matrix tests.![image](https://github.com/numaru/vscode-ceedling-test-adapter/assets/48290683/88c3fe91-8bc3-43f5-a0fa-922ca820d3b7)
In ceedling, these tests do all pass via the command line when all enabled.
Expected:
To Reproduce:
My project.yml
My test file: