Manual testing of release 2022.12

[tyuldashev]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• [x] Build from the unit-test-bot/rc16122022 branch
• [x] Download plugin
• [x] Check downloaded zip-file size < 100 MB
• [x] Open IntelliJ IDEA 2022.2.4 (last minor release of 2022.2, 2022.3 not supported yet - #1466)
• [x] Remove previously installed UTBot plugin
• [x] Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• [x] Open the project in the IDE
• [x] Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• [x] Generate tests for the class
• [x] Remove results
• [x] Generate and Run test for a method

Manual scenario #2

• [x] Use default plugin settings
• [x] Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• [x] Generate tests with different Mocking options combinations

Regression checks

Collect too complex reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• [x] Installed on 2022.1.4 - Gradle project
• [x] Installed on 2022.2.4 - Gradle project
• [x] Installed on 2022.1.4 - Maven project
• [x] Installed on 2022.2.4 - Maven project
• [ ] Installed on 2022.1.4 - IntelliJ project
• [x] Installed on 2022.2.4 - IntelliJ project

Add a class / package - generate tests for class / method / class

• [x] Gradle - existing test sources root - No mocks
• [ ] Maven - existing test sources root - Mocks outside package
• [ ] Maven - new test sources root - Mocks outside class
• [ ] IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• [x] JDK 8 - JUnit 5 - Parametrized
• [ ] JDK 8 - JUnit 4 - Parametrized off
• [x] JDK 8 - TestNG - Parametrized off
• [x] JDK 11 - JUnit 5 - Parametrized
• [ ] JDK 11 - JUnit 4 - Parametrized off
• [x] JDK 11 - TestNG - Parametrized
• [x] JDK 17 - JUnit 5 - Parametrized (@alisevych)
• [x] JDK 17 - JUnit 4 - Parametrized off (@alisevych)
• [x] JDK 17 - TestNG - Parametrized off (@alisevych)

UI checks - in a project with JDK 11

• [x] timeout changes - 1/10/60 sec - check with stopwatch
• [x] cancellation - check on different steps
• [x] regions: Symbolic first, Fuzzing,
• [x] regions: error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• [x] exceptions - check Problems tab
• [x] context menu / shortcut
• [x] call from method / class / package - check Members list
• [x] call for several classes - Generate and Run - check all test classes are run

Check Settings - in a project with JDK 17

• [x] Symbolic 100%
• [x] Symbolic 95%
• [ ] Symbolic 5%
• [x] Fuzzing 100%
• [x] Summaries with custom tags - check all tags
• [x] Summaries with plain text - check for Fuzzing
• [x] Summaries turned off
• [x] Tests with exceptions: Fail
• [x] Tests with exceptions: Pass
• [ ] Overflow : Ignore overflows
• [ ] Overflow: Treat overflows as errors

Other features testing

• [ ] Strings
• [x] Mocks
• [ ] Collections
• [x] IDEA Community 2022.2.4 smoke test

[tyuldashev]

Bugs found:

• 1588

• 1589

• 1586

• 1560