Feature : automatic unit test generation based from contracts

[lcetinsoy]

Hello !

It is me again haha. Imagine the following : you put some CDD assertions with phpdeal (or other tool) and you have some automatic unit tests generated automatically. Wouldn't that be so cool ?.

I got this idea from the Hoa\Praspel project which is a specification language usable for CDD. It is similar to phpdeal, albeit a bit more potent maybe. They developed a small plugin which generates automatic unit test from Praspel contracts into atoum cf link. Unfortunately this idea was not pushed more and the main author of Hoa stopped to work on it.

I find this idea fascinating with a lot of potential and I may try to implement it for phpunit. I wanted to have your thoughts about it to get feedbacks. I am considering two options :

Idea 1 : Simply port hoaproject/Contributions-atoum-praspelExtension to phpunit since atoum ... well was not successful

Pros :

• No too difficult I would say for a first shot.
• Praspel comptability with Realistic domain can let specify very precise contracts

Cons :

• They do not have a CDD implementation like phpdeal which are useful to catch errors at system level.
• No integration with phpdeal which "has more user" than Praspel.

Idea 2 : Build the same mechanism directly from phpdeal. Maybe we could integration RealDom into it

Pros :

• You have CDD and PhpUnit automated tests
• Integration with Praspel\RealisticDom or other things (like Faker ) could be added to enrich assertions

Cons :

• The mechanism to sample inputs from Verify contracts would be complicated to build ? I do not know yet

So here are my questions :

1. What are your thoughts about this idea, overall ?
2. Would some people be interested to help if I started to build something ?
3. Would this built as a separated repository or integrated in this one ? I guess Separated.

4. Regarding implementation of idea 2, do you agree with the following ?

   A generated test would do the following :

   1. Samples inputs according the Verify contracts
   2. Run phpunit assertion using Ensure contracts.
   3. Also load phpdeal "normally" to have invariants contract checking for free during tests.

To do that, the following could be done :

Load a php file to test and for each public method:

  1. Generate input data
       1.1 Read Verify contracts with MethodConditionFetch class ?
       1.2 Sample input from it with some Sampler classes to be written.
       1.3 Generate code for input data
  2. Generate unit test assertion
       1.1 Read Ensure contracts with ??
       1.2 For each Ensure contract generate a test assertion.
  3. Generate the whole test code in phpunit (or other framework) with a class to be written.

As I write and specify the to-do list, I get it could be a bit difficult (how to construct the object to be tested automatically if it is complex). I will keep updating, the post as I progress in my thinking.

What do you think ?

edit : eris and property based testing are also relevant to the thinking

Best

Laurent

[Hywan]

you put some CDD assertions with phpdeal (or other tool) and you have some automatic unit tests generated automatically

It's called Contract-based Testing. Learn more by reading some research papers like https://hoa-project.net/En/Literature.html#Research.

I got this idea from the Hoa\Praspel project which is a specification language usable for CDD. It is similar to phpdeal, albeit a bit more potent maybe

Don't want to brag or anything, but Praspel is a specification language based on contract-based testing, which provides 2 main features: It can validate data, and it can generate data. That's the whole point of “realistic domains”, the underlying layer of Praspel. This latter feature is a lot more powerful than phpdeal (sorry, I just want to be factual). It is possible to generate strings based on regex, or grammar (it's called Grammar-based Testing). See https://github.com/hoaproject/Regex or https://mnt.io/2014/09/30/generate-strings-based-on-regular-expressions/ for a more in-depth explanation. Or even binary. Or to generate complex object layout. Or to generate complex arrays with a constraint-based solver. Or to combine all of them!

Unfortunately this idea was not pushed more and the main author of Hoa stopped to work on it.

We tried to promote it in several events, conferences, on Internet, Reddit, Twitter etc etc. Apart from a small groups of brave developers, the PHP community wasn't interested by it. We somehow give up after many years while the technology (Praspel) is awesome, and it has received excellent feedbacks in the research community (3 articles, 1 PhD thesis, 1 journal (!)).

Simply port hoaproject/Contributions-atoum-praspelExtension to phpunit since atoum ... well was not successful

That's a good idea. Side note: atoum is far superior to PHPUnit in many regards, but the PHP community is really polarized, and it's hard to move it forward. Implementing Praspel in PHPUnit is an excellent idea though!

Praspel comes with @invariant, @requires (precondition), @ensures (postconditions), @throws (postconditions on exceptions), \old(i) to refer in the post-state to values in the pre-state etc. I think it offers everything you need?

I hope my message doesn't sound too negative. I'm excited to see a new interest in Praspel! And I'm willing to help if you need it :-).

[icanhazstring]

Hi. Thanks for the input and the great ideas.

But I think phpdeal is not "source" to put this kind of integration into. I think of phpdeal as a tool to achieve this kind of implementation.

That said. Building an extension for PHPUnit to support Hoa\Praspel sounds nice. This extension then could use phpdeal to add contract based testing.