UnitTestBot / UTBotJava

Automated unit test generation and precise code analysis for Java
Apache License 2.0
133 stars 40 forks source link

HashCode of a mocked object is used in path constraints #286

Open alisevych opened 2 years ago

alisevych commented 2 years ago

Description

DoNotMockHashCode is failing on ExampleClass. hashCode of the mocked object is returning.

To Reproduce

Steps to reproduce the behavior:

  1. Run IntelliJ IDEA
  2. Open UTBotJava project
  3. Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
  4. Generate tests for doNotMockHashCodewith "Other packages: Mockito" and "No static mocking"
  5. Run the generated test

Expected behavior

testDoNotMockHashCode_ExampleClassHashCode is passed

Actual behavior

testDoNotMockHashCode_ExampleClassHashCode fails on Assertion hashcode of the mocked object is returned

Visual proofs (screenshots, logs, images)

hashCode is different in every run. Error message in log is like that: expected: <1119242936> but was: <227290491> Expected :1119242936 Actual :227290491

Environment

Java 8

Additional context

Even if we make ExampleClass::hashCode return a constant - there will be a random number in the test:

 @Override
    public int hashCode() {
        return 100 * 31;
    }
CaelmBleidd commented 2 years ago

The problem occurs when we have a method that returns a hashcode of some mocked value or uses it in path conditions. In such cases, we will have path divergence since the engine takes information from the real methods of the value (we cannot mock hashcode and equals due to Mockito limitations), but in the tests, we will use the ones from the Mocked value => have failing tests.

I think the best way is to detect such situations inside the engine and generate resulting tests without assertion for expected values (with a remark about the reason).

But seems like it is something that is not easy to do, so I suggest removing this bug from the release milestone if we haven't seen such situations in real (users') code and fixing it later. What do you think, @alisevych?