PowerMockTestCase with TestNG should use alwaysRun=true on its annotations

[GoogleCodeExporter]

What steps will reproduce the problem?

0. Use TestNG. Use test groups when running the tests.

1. Let some test class call mockStatic on some class, e.g Utils.class:

@PrepareForTest({ Utils.class })
public class MyTest1 extends PowerMockTestCase {

  @Test(groups="SANDBOX")
  public void someTest() {
    ...
    mockStatic(Utils.class);
    expect(Utils.someMethodThatMustBeMocked()).andReturn(true).anyTimes();
    ...
  }

}

2. Let some other test class invoke methods on the class mocked by the first 
test class, but NOT mocked at all in this test class:

@PrepareForTest({ SomethingElse.class })
public class MyTest2 extends PowerMockTestCase {

  @Test(groups="SANDBOX")
  public void anotherTest() {
    ...
    // Class Utils should not be mocked in this test
    // However, for some reason PowerMock expects it to be mocked and crashes
    Utils.someMethodThatWasNotMockedInMyTest1();
    ...
  }

}

3. Run a Maven build with some test group enabled, e.g.

        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.4.2</version>
          <configuration>
            <groups>SANDBOX</groups>
          </configuration>
        </plugin>

Very often, but not always, the second test will be crashing with things like:

java.lang.AssertionError: Unexpected method call 
someMethodThatWasNotMockedInMyTest1():
 at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:45)
 at org.powermock.api.easymock.internal.invocationcontrol.EasyMockMethodInvocationControl.invoke(EasyMockMethodInvocationControl.java:95)
 at org.powermock.core.MockGateway.doMethodCall(MockGateway.java:105)
 at org.powermock.core.MockGateway.methodCall(MockGateway.java:60)
 at xxx.yyy.zzz.Utils.someMethodThatWasNotMockedInMyTest1(Utils.java)
 at xxx.yyy.zzz.UtilsTest.anotherTest(MyTest2.java:...)

What is the expected output? What do you see instead?

I expect test MyTest2 to pass. However, it crashes occasionally. It never 
crashes when I run the test alone. But when I run it together with many other 
tests, e.g. during a Maven build which runs all tests in the project, the 
MyTest2 crashes sometimes.

What version of the product are you using? On what operating system?

I have used it with PowerMock 1.4.10 and 1.5.1. Both behave exactly the same.
OS: Windows 7 64-bit.

Please provide any additional information below.

I suspect that the reason for the failures is that sometimes MyTest1 runs 
before MyTest2 and creates a mock for Utils.class, but then fails to remove the 
mock when it's done. And when MyTest2 runs, it uses the mock created by 
MyTest2, while in fact it doesn't want to use a mock, so it crashes.

I think the reason is that PowerMockTestCase uses annotations like @BeforeClass 
instead of @BeforeClass(alwaysRun=true). Therefore, if we run a particular test 
group, e.g. "SANDBOX", the method annotated with @BeforeClass doesn't get 
executed, so MockRepository.clear() is not called and all the old mocks get 
used on the test even though they should've been cleared.

I think that sometimes the build does not crash because tests are run in a 
non-deterministic order and sometimes MyTest1 runs before MyTest2, while at 
other times it runs after MyTest2. So, sometimes MyTest2 uses a mock inherited 
from MyTest1, and sometimes it doesn't use a mock because MyTest1 hasn't run 
yet.

Original issue reported on code.google.com by vesuvius...@gmail.com on 16 Sep 2013 at 11:35