Fuzzing is generating a false test for Random

[alisevych]

Description

Fuzzing is generating a test for Random without mocks - when Mock everything outside class is checked. The test asserts the value returned by Random and fails.

To Reproduce

1. Run a project in IntelliJ Idea 2022.1.3 - 2022.2.2
2. Install RC plugin build
3. Set Fuzzing to 100% (in File -> Settings -> Tools -> UnitTestBot)
4. Generate tests with UnitTestBot for utbot-sample/src/main/java/org/utbot/examples/mock/MockRandomExamples.java for useSecureRandom() - check Mock everything outside class.

Expected behavior

There should be only valid tests generated.

Possible solutions

Fuzzer can use mock Random(?) Generated tests should be double-checks that they return same results(?) other solution(?)

Actual behavior

Fuzzing is generating a test for Random without mocks. It asserts the value returned by Random and fails.

Visual proofs

The following test is generated by Fuzzer:

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method useSecureRandom()

    @Test
    @DisplayName("useSecureRandom: arg_0 = MockRandomExamples() -> return 1")
    public void testUseSecureRandomReturnsOne() {
        MockRandomExamples mockRandomExamples = new MockRandomExamples();

        int actual = mockRandomExamples.useSecureRandom();

        assertEquals(1, actual);
    }
    ///endregion

Environment

Windows 10 Pro IntelliJ IDEA 2022.2.2

Additional context

Originally posted by @alisevych in https://github.com/UnitTestBot/UTBotJava/issues/1102#issuecomment-1269804477

[alisevych]

@korifey @denis-fokin @Markoutte May be the team can consider a common solution for handling changing data? That can be Random, Date/DateTime, data from files/other resources, etc. How to determine them? and how to mock?

[korifey]

We should definitely reproduce every test, traversed by concrete executor. My proposal is to enhance concrete executor. It must automatically fixate behavior of certain methods and return back information to engine information about it. E.g. if concolic executor find some new Random() it should return value to engine that will generate static mocking for this case. @SBOne-Kenobi could you try to elaborate this idea.

[alisevych]

concolic executor find some new Random()

Also need to find Random usages. And other Random implementations can be used.

@korifey Would it be too long to rerun every test generated by Fuzzing? - to check if received data is changing from run to run

[sofurihafe]

Also, there is the same problem with Random when generating tests for vertexSum method from recursion package. It is worth to mention that it reproduces not every time. Steps to reproduce:

1. Run runIde task
2. Copy recursion package to the test project inside runIde
3. Generate a test for vertexSum method with the following settings: JUnit5, not parameterized, any mocking strategy. You have to have two generated tests, and regenerate if there is only one.

One (the test below) of the two generated tests will fail when running because of Random usage inside vertexSum method.

///region FUZZER: SUCCESSFUL EXECUTIONS for method vertexSum(int)

/**
 * @utbot.classUnderTest {@link Recursion}
 * @utbot.methodUnderTest {@link Recursion#vertexSum(int)}
 */
@Test
@DisplayName("vertexSum: n = 134217732 (mutated from 4) -> return 293591018")
public void testVertexSum() {
    Recursion recursion = new Recursion();

    int actual = recursion.vertexSum(134217732);

    assertEquals(293591018, actual);
}
///endregion