search

STAMP-project / dspot

Automatically detect and generate missing assertions for Junit test cases (also known as test amplification)
https://dspot-demo.stamp-project.eu/
GNU Lesser General Public License v3.0
114 stars 28 forks source link

study the optimal configuration of amplifiers #777

Open monperrus opened 5 years ago

monperrus commented 5 years ago

We should have a default optimal configuration of amplifiers, and maybe a set of 2/3 alternative ones.

Zeming Dong will work on this.

zemingd commented 5 years ago

A represents the I-Amplification; P represents the Project.

P1=javapoet; P2=mybatis-3; P3=traccar; P4=stream-lib; P5=mustache.java; P6=twilio-java; P7=jsoup; P8=protostuff; P9=logback; P10=retrofit.

Figure in brackets denotes the Incresae Killed.

zemingd commented 5 years ago

Classify amplifier methods into

  1. Method calls: MethodAdd, MethodRemove, MethodGeneratorAmplifier
  2. Literals: StringLiteralAmplifier, NumberLiteralAmplifier, BooleanLiteralAmplifier, CharLiteralAmplifier, AllLiteralAmplifiers, TestDataMutator
  3. Test objects: ReturnValueAmplifier, NullifierAmplifier

Jsoup.AttributeTest

  1. Method calls： MethodAdd+MethodRemove, (330-332-0.6%) MethodAdd+MethodGenerator, (330-341-3.3%) MethodRemove+MethodGenerator(330-342-3.6%), MethodAdd+MethodRemove+MethodGenerator（330-339-2.7%）

  2. Literals： StringLiteralAmplifier+NumberLiteralAmplifier（330-332-0.6%） StringLiteralAmplifier+BooleanLiteralAmplifier（330-332-0.6%） StringLiteralAmplifier+CharLiteralAmplifier（330-332-0.6%） NumberLiteralAmplifier+BooleanLiteralAmplifier（330-331-0.3%） NumberLiteralAmplifier+CharLiteralAmplifier（330-332-0.6%） BooleanLiteralAmplifier+CharLiteralAmplifier（330-331-0.3%） StringLiteralAmplifier+NumberLiteralAmplifier+BooleanLiteralAmplifier（330-332-0.6%） StringLiteralAmplifier+NumberLiteralAmplifier+CharLiteralAmplifier（330-332-0.6%） NumberLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier (330-331-0.3%) StringLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier (330-332-0.6%) TestDataMutator+StringLiteralAmplifier+NumberLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+BooleanLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+NumberLiteralAmplifier+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+NumberLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+AllLiteralAmplifiers（330-333-0.9%） TestDataMutator+StringLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier（330-333-0.9%） TestDataMutator+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+CharLiteralAmplifier（330-333-0.9%）

  3. Test objects： ReturnValueAmplifier+NullifierAmplifier（330-331-0.3%）

  1. Method calls+Test objects MethodRemove+MethodGenerator+ReturnValueAmplifier+NullifierAmplifier（330-331-0.3%） MethodRemove+MethodGenerator+ReturnValueAmplifier（330-331-0.3%） MethodRemove+MethodGenerator+NullifierAmplifier（330-331-0.3%） MethodGenerator+ReturnValueAmplifier（330-331-0.3%） MethodRemove+ReturnValueAmplifier（330-331-0.3%） MethodGenerator+NullifierAmplifier（330-331-0.3%） MethodRemove+NullifierAmplifier（330-331-0.3%）

  2. Literals+Test objects AllLiteralAmplifiers+ReturnValueAmplifier+NullifierAmplifier（330-332-0.6%） AllLiteralAmplifiers+ReturnValueAmplifier（330-332-0.6%） AllLiteralAmplifiers+NullifierAmplifier（330-332-0.6%） TestDataMutator+ReturnValueAmplifier+NullifierAmplifier（330-333-0.9%） TestDataMutator+ReturnValueAmplifier（330-333-0.9%） TestDataMutator+NullifierAmplifier（330-333-0.9%）

  3. Method calls+Literals+Test objects MethodRemove+MethodGenerator+TestDataMutator+ReturnValueAmplifier（330-333-0.9%） MethodRemove+MethodGenerator+TestDataMutator+NullifierAmplifier（330-333-0.9%） MethodRemove+TestDataMutator+ReturnValueAmplifier（330-333-0.9%） MethodRemove+TestDataMutator+NullifierAmplifier（330-333-0.9%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator+ReturnValueAmplifier（330-334-1.2%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator+NullifierAmplifier（330-334-1.2%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator+NullifierAmplifier+ReturnValueAmplifier (330-333-0.9%)

monperrus commented 5 years ago

Could you add the absolute numbers, eg before:100 killed, after 110 killed (100-110-10%)?

monperrus commented 5 years ago

We have 11 configurations, so we will have the results for:

danglotb commented 5 years ago

Hello,

TestDataMutator is the old implementation of AllLiteralAmplifiers, so it should be in the category Literals, right?

Thank you.

monperrus commented 5 years ago

Hi Benjamin, Do you mean that TestDataMutatoris legacy code that can be discarded? even removed from the code base?

monperrus commented 5 years ago

@Deeplus Auto-correlation: do you mean within-group combination, eg all amplifiers coming from the same group? Cross-correlation: do you mean inter-group combination, eg at least two amplifiers coming from the same group?

danglotb commented 5 years ago

The fact is that since TestDataMutator is less stronger, i.e. it generates less amplified test methods at each iteration, it allows DSpot to obtain better result because good amplified test methods are not discarded. (You know that when we have a lot of amplified test methods, we discard a part of them to have a reasonable number of amplified test methods to pursue the amplification)

So it happens, that you can obtain better result using TestDataMutator instead of AllLiteralAmplifiers.

This why I kept it in DSpot.

This is an open question: according to our resources, how much can we explore?

It would be awesome to fine-tune the input amplification, at the level of the amplifier, in order to not exceed this arbitrary number of test methods and not waste any resource.

monperrus commented 5 years ago

I see. What about renaming TestDataMutator? for instance into FastLiteralAmplifier?

monperrus commented 5 years ago

according to our resources, how much can we explore?

That's indeed one of the hardest questions that @Deeplus is answering.

zemingd commented 5 years ago

Hi all,

  1. @monperrus Auto-correlation means within-group combination; Cross-correlation means inter-group combination.

  2. @danglotb ReturnValueAmplifier and NullifierAmplifier belong to the category Test objects, is this right?

monperrus commented 5 years ago

@danglotb What about renaming TestDataMutator whose name is confusing? for instance into FastLiteralAmplifier? Another idea?

danglotb commented 5 years ago

Hi, @Deeplus Yeah this is the correct category.

@monperrus okay to modify the name of the amplifier TestDataMutator to FastLiteralAmplifier