search

catchorg / Catch2

A modern, C++-native, test framework for unit-tests, TDD and BDD - using C++14, C++17 and later (C++11 support is in v2.x branch, and C++03 on the Catch1.x branch)
https://discord.gg/4CWS9zD
Boost Software License 1.0
18.54k stars 3.05k forks source link

Implement Property Based Testing/ Data Generators #850

Open philsquared opened 7 years ago

philsquared commented 7 years ago

This has been a long-standing promise/ point of discussion and a number of issues have been raised that amount to this, so I thought I'd write up the definite feature request to direct them all to.

There are two strands to this:

  1. Data generators, or parametrised tests: i.e. you want to re-use the same core set of assertions with a range (possible a wide range) or different inputs - including the cross product of multiple possible inputs. In some cases you want to specifically control the inputs, in others generate it as an increasing range, or randomly. There's a variation where different types are used to parameterise. While strictly speaking that's a different feature I suspect it's all going to get tied up in them same solution, so I'll include that here too. We can always unbundle it later, if necessary.

  2. Building on (1) is the idea of Property-Based Testing. This is a more formally structured approach to working with generated ranges of inputs and also includes features such as shrinking (where failing inputs are automatically reduced to the simplest/ smallest possible failure). The tests themselves are for properties which are usually invariants that should always hold - although sometimes an alternate (e.g. simple or reference) implementation is compared against.

Support for generators was experimentally added in the very early days of Catch, but were never progressed. In fact quite a lot of work was done in support of them, and a few alternate implementations started. The main obstacle was interaction with the SECTION tracking infrastructure. That code was reworked with generators in mind a couple of years ago, and a new proof-of-concept generators implementation was successfully written against it. However by that time the goal was full property-based testing and the path there seemed tortuous in a C++03 constrained context, so the effort was deferred to such a time as we could at least rebase on C++11 (and ideally would leverage C++14 and beyond to allow for things like Ranges v3 to be supported in the generators part). C++11 rebasing was decided for Catch2 - with generators/ PBT being one of the main motivators. At time of this writing a proof-of-concept implementation of Catch2 exists which consists of a mostly rewritten core. That work has paused while the "Catch Classic" backlog is tamed but will resume again soon - with generators and PBT being one of the first big features to be worked on next.

I'll keep this issue open until the feature is ready so others can be closed in favour of it.

horenmar commented 3 years ago

@matthew-limbinar Everything that makes it actually property testing. There is data generation, but no case reduction, and no property testing loop support.

tsondergaard commented 1 year ago

@horenmar: Can you identify what is left on this feature request? Seems like it's potentially closable.

You cannot address a particular combination that you would like to run. Lets say that I break a particular case in a large set of generated combinations.

  1. The reporter doesn't report the virtual section the test failure - if it did it would be easier to identify the failure
  2. I cannot specify on the command line the specific test I would like to run. This can be a big disadvantage if running the test across all parameters take much longer than running just the single case that is failing