Test on Recursion generated by Fuzzer with int overflow should fail when `Treat overflows as errors`

[alisevych]

Description

Tests generated by Symbolic Execution for OverflowExamples with Treat overflows as errors option on - are supposed to fail with ArithmeticException, but they are passing.

To Reproduce

1. Install one of the latest plugin build from main in IDEA
2. Run the 'UTBotJava' project in IntelliJ Idea
3. Set Overflow detection to Treat overflows as errors and Fuzzing on:

4. Use plugin to generate tests for the sample org.utbot.examples.recursion.Recursion.factorial:

Expected behavior

Test produces int overflow inside factorial method (2^32 => 0) and is supposed to fail. The test also should go to OVERFLOWS region and should have a comment about int overflow.

Actual behavior

A regular test is generated. It is added to ///region FUZZER: SUCCESSFUL EXECUTIONS for method factorial(int) The test is passing.

Visual proofs (screenshots, logs, images)

    /**
     * @utbot.classUnderTest {@link Recursion}
     * @utbot.methodUnderTest {@link Recursion#factorial(int)}
     */
    @Test
    @DisplayName("factorial: n = 1024 (mutated from 0) -> return 0")
    public void testFactorialReturnsZero() {
        Recursion recursion = new Recursion();

        int actual = recursion.factorial(1024);

        assertEquals(0, actual);
    }

Environment

Windows 10 Pro IntelliJ IDEA 2022.2.3

[CaelmBleidd]

This one is generated by fuzzer, so I guess it cannot know anything about overflow/underflow. @Markoutte what do you think?

[Markoutte]

Yes, fuzzer cannot track the problem until it explicitly thrown from the concrete execution. And fuzzed doesn't do any analysis like a symbolic engine to recognize such problems, like integer overflows. I see that we have some options that are not applicable to fuzzer's tests and there's no way to bring them some sense. I can make 2 suggestions:

1. Explicitly write a comment about that this option doesn't influent to fuzzer
2. or implement such analysis inside concrete execution somehow and throw an exception from it.