The scope of `search_budget` property?

[jiseongg]

Context

Hi, I'm doing my experiment with EvoSuite on open-source maven projects. After compiling project, I'm running EvoSuite for target class(CUT) with command like below, which is similar to usage in https://www.evosuite.org/documentation/tutorial-part-1/

$EVOSUITE -class <qualified.name.of.CUT> \
    -projectCP target/classes:target/dependencies/dependencies.jar \
    -criterion branch

I'm tuning several parameters to fix some baseline I wanna investigate later. One parameter still confusing me is -Dsearch_budget, since I could find short explanation only from $EVOSUITE -listParameters. If I set it as 60s, does this time is the budget for class, or method? I've read a paper and the artifact of it that setup time_budget for methods, so I become confused.

[apanichella]

Hi, Dsearch_budget is the search time per each class under test (CUT)

[jiseongg]

Hi, @apanichella

It's very fast answer.. :) Thank you very much. Does EvoSuite have function to generate test for target methods? I mean, method-level unit testing.

[apanichella]

No problem! EvoSuite targets entire classes and because we cannot test individual methods. For example, we would need to invoke at least the constructor plus the method itself to test a method. Besides, methods often indirectly interact via accessing the same attributes or external classes.

Suppose you are interested in targeting (or covering) only branches in one single method. In that case, you can apply a filter to the list of search objectives (i.e., by considering only the branches for a given method). That would allow you to optimize coverage for one single method only. However, EvoSuite will still generate complete test cases, e.g., with constructor call and other method calls if they contribute to the ''focused'' coverage.

I hope this answers your question.

[jiseongg]

Thanks a lot! It helped me a lot for better understanding about the various usages of EvoSuite in literatures.

[WendkuuniArzouma]

Hello to all! Sorry to reopen this topic. Currently I'm doing some experiments but on the whole jar like for example : $EVOSUITE0 -target my.jar -generateSuite -generateRandom -Dsearch_budget=180 -Dstopping_condition=MaxTime. I have indeed noticed that there the Dsearch_budget applies to every class of the jar. I would like to know if it is possible to allocate this time for all the Jar, that is to say in our exmple 180s for the whole classes contained in the jar.

[jiseongg]

Hi @WendkuuniArzouma,

Though I am not a maintainer of EvoSuite, as an author of this issue thread I would like to share some details I know.

As far as I know, TestGeneration#generateTestsTarget will be executed in your case (due to -target option), which will iterate all classes in my.jar. Each class is passed to TestGeneration#generateTests, and this method seems to call all the core parts of EvoSuite (all codes lying in client directory). Thus, I thought the last part was the point where the search_budget is allocated, which contradicts your observation. So I would like to observe your results together, if you could give the concrete configurations and commands. I am ready to run EvoSuite in my environment immediately.

[WendkuuniArzouma]

Hello @jiseongg! Thanks for your quick response, I really appreciate it! I want to run tests on the jar of a project, so consequently on all classes. For example, I would like to allocate a global time for test generation for all classes. But I could see that with -Dsearch_budget, the time allocated is for each class in the jar. And I was wondering if it's possible that the allocated time is taken into account for all the classes, but not individually. For example for the project commons-cli-1.5.0.jar there are 22 classes. Using the following command: $EVOSUITE0 -target commons-cli-1.5.0.jar -generateSuite -generateRandom -DOUTPUT_DIR=results -Dsearch_budget=180 -Dstopping_condition=MaxTime

Evosuite will take 180s to generate tests for each class. I wonder if it is possible to allocate the 180s globally for all classes?

[jiseongg]

Hi @WendkuuniArzouma, Firstly, I am sorry that I misunderstood your question. I think I understood it oppositely.

I think you're looking for the "Continuous Test Generation". EvoSuite has this mode, with a bunch of corresponding options starting with the prefix ctg. I was able to run EvoSuite with commons-cli like the command below.

$EVOSUITE0 -target ./commons-cli-1.6-SNAPSHOT.jar --generateSuite -generateRandom --continuous=execute -Dctg_time=21

In my case, the jar file I have built from Commons CLI project has 21 classes (different from your case but I don't think this number matters).

The progress in the console looks following:

* EvoSuite 1.1.0
Going to execute 21 jobs
Estimated completion time: 21 minutes, by 2023-02-24T21:55:08.854
Going to start job for: org.apache.commons.cli.PatternOptionBuilder. Expected to end in 60 seconds, by 2023-02-24T21:35:08.867
Completed job. Left: 20
Going to start job for: org.apache.commons.cli.UnrecognizedOptionException. Expected to end in 60 seconds, by 2023-02-24T21:35:45.841
Completed job. Left: 19
Going to start job for: org.apache.commons.cli.Options. Expected to end in 60 seconds, by 2023-02-24T21:35:49.231
Completed job. Left: 18
Going to start job for: org.apache.commons.cli.Parser. Expected to end in 60 seconds, by 2023-02-24T21:36:25.246
Completed job. Left: 17
Going to start job for: org.apache.commons.cli.HelpFormatter. Expected to end in 60 seconds, by 2023-02-24T21:36:58.796
Completed job. Left: 16
Going to start job for: org.apache.commons.cli.MissingOptionException. Expected to end in 60 seconds, by 2023-02-24T21:38:00.029
Completed job. Left: 15
Going to start job for: org.apache.commons.cli.TypeHandler. Expected to end in 60 seconds, by 2023-02-24T21:38:34.770
Completed job. Left: 14
Going to start job for: org.apache.commons.cli.DefaultParser. Expected to end in 60 seconds, by 2023-02-24T21:39:10.320

However, when I passed 3 minutes time budget for all of them, EvoSuite worked only for three classes with 1 minute per-class time budget. There seems to be a minimum per-class search budget. I suppose you can control this budget with related properties in link

[WendkuuniArzouma]

Dear @jiseongg , Many thanks four your reply, I am very sorry for the late reply! There is really no problem, I understand! You are right, the "Continuous Test Generation" should be what I am looking for! From your experiments, it means that even if we use the "Continuous Test Generation" for all the jar Evosuite will generate for a couple of class depending on the time budget, but not for all the class in the jar. I will try it, and see, what it will result in. Thanks a lot!