search

STAMP-project / dspot

Automatically detect and generate missing assertions for Junit test cases (also known as test amplification)
https://dspot-demo.stamp-project.eu/
GNU Lesser General Public License v3.0
114 stars 28 forks source link

add support for ordering / ranking amplified tests by "interestingness" for the developer #196

Open monperrus opened 6 years ago

monperrus commented 6 years ago

Hi Benjamin, I read in the paper "I-amplification can result with a large set test method. For sake of computation time, we reduce this sets at 200 test cases maximum, selected randomly."

Is this what's implemented? how are generated tests ordered before being presented to the user?

danglotb commented 6 years ago

Hi @monperrus

"I-amplification can result with a large set test method. For sake of computation time, we reduce this sets at 200 test cases maximum, selected randomly."

Yes, it is what is implemented. The method call to reduce() in Amplification#amplification() is doing this job.

In fact, applying I-Amplification to hundred test cases can results with thousand of amplified test cases. Generate assertions for such amount of test is not practicable, because of the instrumentation, and the triple run to discard flaky values.

I ran several different configuration (several months ago) to find the best ratio between time / potential of tests, and I found 200, and even with this amount, it is still a little bit long.

Anyway, this is a very important problem in DSpot, and we should probably fix it ASAP.

I investigated a little bit on those I-Amplification, and I think there are a lot of useless amplified tests, and DSpot may throw away goods test when selecting them randomly.

how are generated tests ordered before being presented to the user?

There is no particular order on this. I suppose that there are ordered in function of the history: first amplified tests are at the begin of the class / json (report file).

monperrus commented 6 years ago

OK, this may be something to work on in the future, but we have many things of higher priority now :-)

monperrus commented 6 years ago

We could order tests by "localness / spreading".

From paper: "A perfect amplified test kills mutants which are all located in the same method. From the developer's viewpoint, it means that the intention of the test is clear: it specifies the behavior provided by a given method or block."