Manual testing of Release 2022.10

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed on IDEA 2022.1.4

• [x] Plugin from unit-test-bot/rc2992022 build with latest changes
• [x] Download plugin
• [x] Check downloaded zip-file size < 100 MB
• [x] Open IntelliJ IDE 2022.1.4
• [x] Remove previously installed UTBot plugin
• [x] Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• [x] Open the project in the IDE
• [x] Install the downloaded plugin

Check that the IntelliJ Idea UTBot plugin can be successfully installed on IDEA 2022.2.2

• [x] Open IntelliJ IDE 2022.2.2
• [x] Remove previously installed UTBot plugin
• [x] Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• [x] Open the project in the IDE
• [x] Install the downloaded plugin

Go through manual scenarios on IDEA 2022.1.4

Manual scenario #1

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• [x] Generate tests for the class
• [x] Remove results
• [x] Generate and Run test for a method

Manual scenario #2

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• [x] Generate tests with different Mocking options combinations

Go through manual scenarios on IDEA 2022.2.2

Manual scenario #1

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• [x] Generate tests for the class
• [x] Remove results
• [x] Generate and Run test for a method

Manual scenario #2

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• [x] Generate tests with different Mocking options combinations

@alisevych

Check following in IDEA 2022.1.4

Manual scenario #3

• [x] Create a new Gradle project with JDK 8
• [x] Add a simple java file to test
• [x] Generate a test in the existing test root

Manual scenario #4

• [x] Create a new Maven project with JDK 8
• [x] Add a simple java file to test
• [x] Generate a test with existing test sources root

Manual scenario #5

• [x] Create a new Idea project with JDK 11
• [x] Add a simple java file to test
• [x] Generate tests for several classes in a new test sources root

@tyuldashev

Check following in IDEA 2022.2.2

Manual scenario #3

• [x] Create a new Gradle project with JDK 8
• [x] Add a simple java file to test
• [x] Generate a test in the existing test root

Manual scenario #4

• [x] Create a new Maven project with JDK 8
• [x] Add a simple java file to test
• [x] Generate a test with existing test sources root

Manual scenario #5

• [x] Create a new Idea project with JDK 11
• [x] Add a simple java file to test
• [x] Generate tests for several classes in a new test sources root

Regression testing

@alisevych

IDEA 2022.1.4

• [x] Maven with JDK 11 : make following checks
• [x] Maven with JDK 8 : make following checks
• [x] Test frameworks JUnit4/ JUnit5/TestNG
• [x] - with Parametrization on / off
• [x] - with mocks on /off
• [x] - UtUtils class (with mocks off / on) - check it's regenerated when mocks are on for the first time
• [x] String Java 8 - with JDK 8
• [x] String Java 9 - 11 - with JDK 11

Project with JDK 11:

• [x] timeout changes - 1/10/60 sec
• [x] cancellation
• [x] regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• [x] exceptions - check regions for exceptions
• [x] context menu / shortcut
• [x] call from method / class / package - check Members list
• [x] math
• [x] make.symbolic
• [ ] inner
• [x] collections
• [x] Generics - with Fuzzing
• [x] Sandbox - smoke

@tyuldashev

IDEA 2022.2.2

• [x] Gradle project with JDK 8 : make following checks
• [x] Gradle project with JDK 11 : make following checks
• [x] Test frameworks JUnit4/ JUnit5/TestNG
• [x] - with Parametrization on / off
• [x] - with mocks on /off
• [x] - UtUtils class (with mocks off / on) - check it's regenerated when mocks are on for the first time
• [ ] String Java 8 - with JDK 8
• [ ] String Java 9 - 11 - with JDK 11

Project with JDK 11:

• [x] timeout changes - 1/10/60 sec
• [ ] cancellation
• [x] regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• [x] exceptions - check regions for exceptions
• [x] context menu / shortcut
• [x] call from method / class / package - check Members list
• [x] Date - check
• [x] Arrays
• [x] annotations
• [x] Generics - with Fuzzing
• [x] Sandbox - smoke

[alisevych]

reproducing #1028 - for Recursion reproducing #1054 - in IDEA 2022.1.4 (JBR 11), 2022.2.2 (JBR 17) reproducing #980 reproducing #1064 reproducing #747 - for utbot-sample/src/main/java/org/utbot/examples/collections/CustomerExamples.java

[alisevych]

new bug with Recursion - empty test class is generated, StackOverflow in Concrete executor log (not always reproducing) reproducing with Cycles

1130

[alisevych]

new bug with MockRandomExample - Fuzzing is generating false-successful test

1135

[alisevych]

new bug? - StringExamples hanging, cancellation doesn't help Illegal reflective access in concrete executor logs, error test for IntPair: /* cannot get resident body for phantom method : <java.lang.Object: void

()> */ #1141

[alisevych]

question - conditions in parametrized tests : if (expectedResult == null) {} else {} better generate 2 separate tests without conditions.

created #1140

[alisevych]

question - EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS - String.repeat(-1) throws [java.lang.IllegalArgumentException: count is negative: -1] is it correct, that explicitly?

1133

[tyuldashev]

Reproducing:

• 1095

• 1099

[alisevych]

new bug: java.lang.reflect.InvocationTargetException для ClassWithComplicatedMethods.sqrt2() with Fuzzing

1117