apex-enterprise-patterns / fflib-apex-mocks

An Apex mocking framework for true unit testing in Salesforce, with Stub API support
BSD 3-Clause "New" or "Revised" License
417 stars 214 forks source link

Intermittent unit test failures in Test_ApexMocksTest in Spring '16 orgs #15

Closed tfuda closed 8 years ago

tfuda commented 8 years ago

I recently deployed the latest fflib-apex-mocks code into a couple of different Spring '16 Developer Edition org. When I tried running tests in the fflib_ApexMocksTest class, I got a number of failures. I've included details of the failures I received in the table at the bottom of this issue description. I found that these failures appeared to be intermittent, and in Spring '16 orgs only. After some debugging of one particular failure (PatronTicket.fflib_ApexMocksTest.whenStubExceptionTheExceptionShouldBeThrown: line 222, column 1), we discovered what appears to be a platform bug with that causes intermittent failures with statements that use the instanceof operator, like System.assert(ex instanceof MyException) statements. This particular assertion expects the caught Exception to be an "instanceof" the MyException class. Sometimes it is, and sometimes it isn't ;) I have opened a case with Salesforce on this, but I wanted to log it here as an issue, so that others using the fflib-apex-mocks framework are aware of this.

Method name Message Stack trace
whenStubSingleCallWithSingleArgumentShouldReturnStubbedValue System.TypeException: Invalid conversion from runtime type String to java.lang.Exception Class.PatronTicket.fflib_ApexMocks.mockNonVoidMethod: line 248, column 1 Class.PatronTicket.fflib_Mocks.Mockfflib_MyList.get: line 21, column 1 Class.PatronTicket.fflib_ApexMocksTest.whenStubSingleCallWithSingleArgumentShouldReturnStubbedValue: line 41, column 1
whenStubSameCallWithDifferentArgumentValueShouldReturnLastStubbedValue System.TypeException: Invalid conversion from runtime type String to java.lang.Exception Class.PatronTicket.fflib_ApexMocks.mockNonVoidMethod: line 248, column 1 Class.PatronTicket.fflib_Mocks.Mockfflib_MyList.get: line 21, column 1 Class.PatronTicket.fflib_ApexMocksTest.whenStubSameCallWithDifferentArgumentValueShouldReturnLastStubbedValue: line 83, column 1
whenStubVoidMethodWithExceptionAndCallMethodTwiceThenExceptionShouldBeThrownTwice System.AssertException: Assertion Failed: Stubbed exception should have been thrown. Class.PatronTicket.fflib_ApexMocksTest.whenStubVoidMethodWithExceptionAndCallMethodTwiceThenExceptionShouldBeThrownTwice: line 306, column 1
whenStubVoidMethodWithExceptionThenExceptionShouldBeThrown System.AssertException: Assertion Failed: Stubbed exception should have been thrown. Class.PatronTicket.fflib_ApexMocksTest.whenStubVoidMethodWithExceptionThenExceptionShouldBeThrown: line 242, column 1
whenStubMultipleCallsWithMultipleArgumentShouldReturnStubbedValues System.TypeException: Invalid conversion from runtime type String to java.lang.Exception Class.PatronTicket.fflib_ApexMocks.mockNonVoidMethod: line 248, column 1 Class.PatronTicket.fflib_Mocks.Mockfflib_MyList.get2: line 26, column 1 Class.PatronTicket.fflib_ApexMocksTest.whenStubMultipleCallsWithMultipleArgumentShouldReturnStubbedValues: line 370, column 1
whenStubMultipleCallsWithSingleArgumentShouldReturnStubbedValues System.TypeException: Invalid conversion from runtime type String to java.lang.Exception Class.PatronTicket.fflib_ApexMocks.mockNonVoidMethod: line 248, column 1 Class.PatronTicket.fflib_Mocks.Mockfflib_MyList.get: line 21, column 1 Class.PatronTicket.fflib_ApexMocksTest.whenStubMultipleCallsWithSingleArgumentShouldReturnStubbedValues: line 60, column 1
whenStubCallWithNoArgumentsShouldReturnStubbedValue System.TypeException: Invalid conversion from runtime type Boolean to java.lang.Exception Class.PatronTicket.fflib_ApexMocks.mockNonVoidMethod: line 248, column 1 Class.PatronTicket.fflib_Mocks.Mockfflib_MyList.isEmpty: line 36, column 1 Class.PatronTicket.fflib_ApexMocksTest.whenStubCallWithNoArgumentsShouldReturnStubbedValue: line 101, column 1
whenStubExceptionTheExceptionShouldBeThrown System.AssertException: Assertion Failed Class.PatronTicket.fflib_ApexMocksTest.whenStubExceptionTheExceptionShouldBeThrown: line 222, column 1
whenStubMultipleVoidMethodsWithExceptionsThenExceptionsShouldBeThrown System.AssertException: Assertion Failed: Stubbed exception should have been thrown. Class.PatronTicket.fflib_ApexMocksTest.whenStubMultipleVoidMethodsWithExceptionsThenExceptionsShouldBeThrown: line 268, column 1
stubAndVerifyMethodCallsWithNoArguments System.TypeException: Invalid conversion from runtime type Boolean to java.lang.Exception Class.PatronTicket.fflib_ApexMocks.mockNonVoidMethod: line 248, column 1 Class.PatronTicket.fflib_Mocks.Mockfflib_MyList.isEmpty: line 36, column 1 Class.PatronTicket.fflib_ApexMocksTest.stubAndVerifyMethodCallsWithNoArguments: line 195, column 1
daveespo commented 8 years ago

The concrete example that we provided to support reproduces the problem reliably on Spring '16 orgs (about 50% of the time):

@isTest
public class Test_InstanceofBug {
    public static testMethod void testTypes(){
        Object o = new MyException('payload');
        System.assert(o instanceof Exception, 'Expected an instance of Exception; it\'s not!');
    }

    public class MyException extends Exception {   }
}

And the problem with the tests that Tom outlined above occurs here: https://github.com/financialforcedev/fflib-apex-mocks/blob/master/src/classes/fflib_ApexMocks.cls#L209

Since the instanceof test sometimes evaluates to true and sometimes it doesn't, the result is that an Exception is returned as a return value vs thrown as an Exception for the mocked function and fails a to be casted to a String

afawcett commented 8 years ago

Just tested this on NA11, seems to be ok, also twitter conv of others reporting similar seem to confirm this.

tfuda commented 8 years ago

Hi @afawcett . I feel like this may actually be more of a pod dependency, than a Spring '16 dependency. I first observed it in an org on NA6, and also reproduced it in an org on NA7. Both of those orgs had already been upgraded to Spring '16, so perhaps I mis-characterized this as a Spring '16 issue. I also have a Spring '16 org on NA14, but was unable to reproduce the problem there. This morning, I tried running fflib_ApexMocksTest 10 times in my NA6 org. It failed 5 out of 10 times. I also ran the Test_InstanceofBug test that @daveespo provided and saw similar results. This is definitely still an issue on NA6, and probably NA7 as well. We have a case open with Salesforce to look into this.

afawcett commented 8 years ago

Wow thanks @tfuda thanks for the update, keep us posted!

hubeiyichangzy commented 8 years ago

Thanks for open the case here. I also encounter the same issue on NA32.

tfuda commented 8 years ago

@hubeiyichangzy I see that NA32 was upgraded to Spring '16 over the weekend. Are you able to confirm whether you saw this issue prior to the Spring '16 upgrade, or did you first notice it after the upgrade?

hubeiyichangzy commented 8 years ago

@tfuda I run the case on Feb 6th, it still pass. The upgrade was happened at Feb 13th on NA32. I rerun the test at Feb 15th and failed. It failed each time I run around 10 times. And the interesting part is it always pass on CS13 which is also Spring 16 at least 10 times run.

tfuda commented 8 years ago

Just a quick update; Tier 3 is investigating this. We've provided them with a repro case and granted them access to a couple of orgs where this problem is occurring. As usual, they aren't very forthcoming with progress reports, other than "we're working on it".

afawcett commented 8 years ago

We are also going to raise a case on this, @tfuda would be happy to share your case number? We will do the same here as well. cc @agarciaodeian

agarciaodeian commented 8 years ago

@tfuda I was checking your super simple piece of code in a Spring'16 and I found something .... If I run several times the test from the org, and the behavior was 1 worked, next one failed, then worked, then failed. Then decided to click on View Test History and clicked on Clear Test Data and since that moment, all executions worked fine. Does your test in the same way in your org?

In any case, please, share with us your case number and we will keep you update as well.

tfuda commented 8 years ago

@afawcett - The case number we have open on this issue is 13125686.

@agarciaodeian - The org where I see this happening most frequently is on NA6. Basically I get about a 50% failure rate using the Test_InstanceofBug test class attached to this thread. Clearing the test history did NOT impact the frequency of the failures for me, one way or the other. FYI, this is NOT an an issue that occurs in unit test context only. This will also fail if you declare a top level class called MyException that extends Exception, and just execute these two lines of code as anonymous Apex:

Object ex = new MyException('This is MyException');
System.assert(ex instanceof Exception, 'Object is NOT instanceof Exception!');

Again, I see about a 50% failure rate in my NA6 org, but it's not always every other invocation. Sometimes it will fail three times in a row, and then pass three times in a row.

agarciaodeian commented 8 years ago

Thanks for the clarification @tfuda . We have raised a case also. This is the number: 13112538

We will keep you update with any useful response.

agarciaodeian commented 8 years ago

@tfuda we got a response from Salesforce support

R&D has logged a bug for the issue and create a known issue

They also let us know that the fix would be released next week (safe harbor) but in any case they recommended to click on the "This issue affects me", so please, go for it.

tfuda commented 8 years ago

Thanks! You Financial Force folks get much faster response from Salesforce support than we do. We've been waiting on them to file a known issue for this for the past week.

dfruddffdc commented 8 years ago

Fixed now :) https://success.salesforce.com/issues_view?id=a1p30000000eVDMAA2