Big Project: Test Amplification with EvoSuite

[lacinoire]

this is a big project so only pick it if you have a lot of time and want to dive deep into EvoSuite and the idea of test amplification

EvoSuite is geared at creating whole test suites from scratch. However, it also has a mode that takes existing test cases into account.

While TestCube is currently relying on DSpot, it is in essence test-generation-tool independent so could also rely on EvoSuite.

Task: Explore using EvoSuite initialized with the current test suite to generate new test cases and propose them to developers.

[abolfazl-sh]

I think first of all we should break this issue to two part: 1- Generalize test-cube api till easily change amplification engine in future (e.g EvoSuite in this issue). I have no idea how test-cube structure is right now, so I will check and will notify if was needed. 2- Go through EvoSuite and nicely import it to the test-cube (I hope no hack needed :|)

Cheer :)

[lacinoire]

Sounds like good two steps!

Currently, TestCube simply contains a .jar of DSpot and calls that on the command line. (StartTestCubeAction) Maybe EvoSuite could be packaged similarly? This would make dependencies simple, as we just use EvoSuite as a separate packaged tool 🙂

In InspectTestCubeResultsAction lots of the magic of looking at the output of DSpot happens. I expect this to need a completely separate implementation for EvoSuite. Happy to discuss any abstraction ideas you have for the results and their presentation! 😊

[abolfazl-sh]

Hey Caro, I have gone through Evosuite for several days, but I didn't find anything to do this :

However, it also has a mode that takes existing test cases into account.

Could you please help me to find it?

[lacinoire]

Hey Abolfazl 🙂

I think my claim that EvoSuite supports this comes from this paper: https://www.evosuite.org/experimental-data/seeding/ (paper with the same name as the title of the blogpost: https://core.ac.uk/download/pdf/42626499.pdf) Diving into the EvoSuite source code atm to see if this option is still there.

[lacinoire]

Hmm no obvious configuration option :/ So either this would mean using / modifying an older version of EvoSuite or find where they moved the seeding with an existing test suite or incorporating an option (again) to start with existing tests

What I found is that the strategies seem to be respoonsible for the initialization, e.g. here one initializes with a random test suite: https://github.com/EvoSuite/evosuite/blob/8730aedd6cbcbeac1d7f246b4f223bda395212c9/client/src/main/java/org/evosuite/strategy/IndividualTestStrategy.java#L323

[abolfazl-sh]

I was checking this link : https://www.evosuite.org/experimental-data/seeding/. I think the command in the follow do what we need (use old test cases as seeds). java -jar evosuite-1.0.6.jar -target a\bin -projectCP a -Dtest_factory=JUNIT -Dselected_junit=Triangle_ESTest1 But I am thinking: Is using old test cases as seeds, is what we can call it "test case amplification"? (I just want to sure we are going in the right path)

[lacinoire]

Creating new test cases based on existing test cases is at the heart of the traditional notion test amplification 🙂 So yes!

How I mean "Test Amplification" at the moment: Generating test cases that improve an existing test suite. If the solution we bulid fulfills this is in my eyes all dependent on what we do with the output of EvoSuite: can we find a few test cases that improve (x?) in the existing test suite? How do we communicate this improvement to the developer to let them choose whether to take over the test case or not?

[abolfazl-sh]

I am usually a little skeptically in the new fields. OK, I got the idea and try to insert it to test-cube.

[abolfazl-sh]

Hello, I am a little busy with a paper. But I am working on this feature too. So this is just a notification which show the task is proceeding. Good luck!

[abolfazl-sh]

Hey Carolin, I have developed test-cube which can use evosuite to amplify the test cases. there is a point here: When we are using DSpot to amplify, there is a json output (which explain about the new test cases coverage), but when we use EvoSuite, the output is just a java file which contains the test cases. Now what is your Idea to continue the task? (I think as EvoSuite uses test case as seeds, it's better to replace Evosuite's output with amplified test cases, but everything you say has high priority)

[lacinoire]

Hey Abolfazl,

super cool that you integrated EvoSuite 🤩 I'm excited to see how it works 🙂

The json output is simply DSpot's way of telling us the additional coverage of the amplified test cases in a convenient way. As it is so easy to parse, I also use the json to build the list of amplifed test cases (and then parse those from the .java file that is also output by DSpot). Does EvoSuite have any report besides the java file? e.g. talking about the coverage of each of the generated test cases or the complete coverage? I think it would be cool to show this to the developer when they inspect the test cases. However, if there is no information provided, it would also be okay to just show the code of the new methods one by one for now 🙂

[lacinoire]

(I think as EvoSuite uses test case as seeds, it's better to replace Evosuite's output with amplified test cases, but everything you say has high priority)

I don't quite understand what you mean here... 🤔 It would be cool if there is a setting to choose the test generator (DSpot vs. EvoSuite) and then the interaction is largely the same: There is a popup that new test cases were generated successfully & then the developer can choose to inspect. Then the tool window opens and lets them browse through the tests, if possible with some information above them, and the developer can add a test with the buttons. 🙂