Manual testing of Release 2022.10.2

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• [x] Build from the unit-test-bot/rc10192022 branch
• [x] Download plugin
• [x] Check downloaded zip-file size < 100 MB
• [x] Open IntelliJ IDEA 2022.2.3 (last released)
• [x] Remove previously installed UTBot plugin
• [x] Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• [x] Open the project in the IDE
• [x] Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• [x] Generate tests for the class
• [x] Remove results
• [x] Generate and Run test for a method

Manual scenario #2

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• [x] Generate tests with different Mocking options combinations

Verify fixes

• [x] #1028
• [ ] #747
• [x] #1098
• [x] #1129
• [ ] #273
• [x] #1149
• [ ] #983
• [x] #1173
• [ ] #448
• [ ] #711
• [ ] #1054

Regression checks

Collect compulsory reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• [x] Installed on 2021.2.4 - Gradle project
• [x] Installed on 2022.1.4 - Gradle project
• [x] Installed on 2022.2.3 - Gradle project
• [ ] Installed on 2021.2.4 - Maven project
• [x] Installed on 2022.1.4 - Maven project
• [x] Installed on 2022.2.3 - Maven project
• [ ] Installed on 2021.2.4 - IntelliJ project
• [x] Installed on 2022.1.4 - IntelliJ project
• [ ] Installed on 2022.2.3 - IntelliJ project

Add a class / package - generate tests for class / method / class

• [ ] Gradle - existing test sources root - No mocks
• [x] Maven - existing test sources root - Mocks outside package
• [ ] Maven - new test sources root - Mocks outside class
• [ ] IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• [ ] JDK 8 - JUnit 5 - Parametrized
• [x] JDK 11 - JUnit 5 - Parametrized off
• [ ] JDK 11 - JUnit 4 - Parametrized off
• [x] JDK 11 - TestNG - Parametrized
• [ ] JDK 17 - JUnit 5 - Parametrized
• [ ] JDK 17 - JUnit 4 - Parametrized off
• [x] JDK 17 - TestNG - Parametrized off

UI checks - in a project with JDK 17

• [x] timeout changes - 1/10/60 sec - check with stopwatch
• [x] cancellation - check on different steps
• [ ] regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• [ ] exceptions - check regions for exceptions
• [ ] exceptions - check Problems tab
• [ ] context menu / shortcut
• [ ] call from method / class / package - check Members list

Check Settings - in a project with JDK 17

• [ ] Symbolic 100%
• [x] Symbolic 95%
• [ ] Symbolic 5%
• [x] Symbolic 0% - Fuzzing 100%
• [ ] Summaries with custom tags - check all tags
• [ ] Summaries with plain text - check for Fuzzing
• [ ] Summaries turned off
• [ ] Tests with exceptions: Fail
• [ ] Tests with exceptions: Pass
• [ ] Overflow : Ignore overflows
• [ ] Overflow: Treat overflows as errors

Other features testing

• [ ] Sandboxing - check default behavior
• [ ] Sandboxing - add some permission
• [ ] Strings

[alisevych]

new bugs:

• 1195

• 1196

[alisevych]

reproducing

• 1174

[alisevych]

Bug is not reproducing, but there are concrete executor errors on the same code:

• https://github.com/UnitTestBot/UTBotJava/issues/1129#issuecomment-1285163470 As investigated by @Markoutte , it is a common case - constructor throws exception if input is not correct. Need to think of a common solution for such cases. Should such cases be shown as error test?

[alisevych]

new bug - has not reproduced yet:

• com.intellij.util.IncorrectOperationException: Sorry but parent: ClientId(value=Host) (class com.intellij.openapi.client.ClientProjectSessionImpl) has already been disposed (see the cause for stacktrace) so the child: Action Update (class com.intellij.openapi.util.Disposer$2) will never be disposed

[alisevych]

RC branch reconstructed: unit-test-bot/rc10202022 branch

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• [x] Build from the unit-test-bot/rc10202022 branch
• [x] Download plugin
• [x] Check downloaded zip-file size < 100 MB
• [x] Open IntelliJ IDEA 2021.2.4 (first supported)
• [x] Remove previously installed UTBot plugin
• [x] Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• [x] Open the project in the IDE
• [x] Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• [x] Generate tests for the class
• [x] Remove results
• [x] Generate and Run test for a method

Manual scenario #2

• [ ] Use default plugin settings
• [ ] Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• [ ] Generate tests with different Mocking options combinations

Recheck fixes

• [x] #1028
• [x] #1098
• [ ] #273
• [x] #1149
• [x] #1173
• [ ] #448
• [ ] #711
• [ ] #1054

Regression checks

Collect compulsory reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• [x] Installed on 2021.2.4 - Gradle project
• [ ] Installed on 2022.1.4 - Gradle project
• [x] Installed on 2022.2.3 - Gradle project
• [ ] Installed on 2021.2.4 - IntelliJ project
• [ ] Installed on 2022.2.3 - IntelliJ project

Add a class / package - generate tests for class / method / class

• [ ] Gradle - existing test sources root - No mocks
• [ ] Maven - new test sources root - Mocks outside class
• [ ] IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• [ ] JDK 8 - JUnit 5 - Parametrized
• [ ] JDK 11 - JUnit 4 - Parametrized off
• [ ] JDK 17 - JUnit 5 - Parametrized
• [ ] JDK 17 - TestNG - Parametrized

UI checks - in a project with JDK 17

• [ ] regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• [ ] exceptions - check regions for exceptions
• [ ] exceptions - check Problems tab
• [x] context menu / shortcut
• [x] call from method / class / package - check Members list

Check Settings - in a project with JDK 17

• [ ] Symbolic 100%
• [ ] Symbolic 5%
• [x] Symbolic 0% - Fuzzing 100%
• [ ] Summaries with custom tags - check all tags
• [ ] Summaries with plain text - check for Fuzzing
• [ ] Summaries turned off
• [ ] Tests with exceptions: Fail
• [ ] Tests with exceptions: Pass
• [ ] Overflow : Ignore overflows
• [ ] Overflow: Treat overflows as errors

Other features testing

• [ ] Sandboxing - check default behavior
• [ ] Sandboxing - add some permission
• [ ] Strings

[alisevych]

new bug:

• 1204

[alisevych]

new bug:

• Test generation is started for several classes. Progress bar shows that generation for 2nd / 3rd class has been already started. Cancel is clicked. No test classes are produced. - expected: Tests for already finished classes should be saved.

[alisevych]

new blocker bug - IDEA is not responding during/after test generation:

• 1207

[alisevych]

Testing was blocked