Closed tfuda closed 8 years ago
The concrete example that we provided to support reproduces the problem reliably on Spring '16 orgs (about 50% of the time):
@isTest
public class Test_InstanceofBug {
public static testMethod void testTypes(){
Object o = new MyException('payload');
System.assert(o instanceof Exception, 'Expected an instance of Exception; it\'s not!');
}
public class MyException extends Exception { }
}
And the problem with the tests that Tom outlined above occurs here: https://github.com/financialforcedev/fflib-apex-mocks/blob/master/src/classes/fflib_ApexMocks.cls#L209
Since the instanceof test sometimes evaluates to true and sometimes it doesn't, the result is that an Exception is returned as a return value vs thrown as an Exception for the mocked function and fails a to be casted to a String
Just tested this on NA11, seems to be ok, also twitter conv of others reporting similar seem to confirm this.
Hi @afawcett . I feel like this may actually be more of a pod dependency, than a Spring '16 dependency. I first observed it in an org on NA6, and also reproduced it in an org on NA7. Both of those orgs had already been upgraded to Spring '16, so perhaps I mis-characterized this as a Spring '16 issue. I also have a Spring '16 org on NA14, but was unable to reproduce the problem there. This morning, I tried running fflib_ApexMocksTest 10 times in my NA6 org. It failed 5 out of 10 times. I also ran the Test_InstanceofBug test that @daveespo provided and saw similar results. This is definitely still an issue on NA6, and probably NA7 as well. We have a case open with Salesforce to look into this.
Wow thanks @tfuda thanks for the update, keep us posted!
Thanks for open the case here. I also encounter the same issue on NA32.
@hubeiyichangzy I see that NA32 was upgraded to Spring '16 over the weekend. Are you able to confirm whether you saw this issue prior to the Spring '16 upgrade, or did you first notice it after the upgrade?
@tfuda I run the case on Feb 6th, it still pass. The upgrade was happened at Feb 13th on NA32. I rerun the test at Feb 15th and failed. It failed each time I run around 10 times. And the interesting part is it always pass on CS13 which is also Spring 16 at least 10 times run.
Just a quick update; Tier 3 is investigating this. We've provided them with a repro case and granted them access to a couple of orgs where this problem is occurring. As usual, they aren't very forthcoming with progress reports, other than "we're working on it".
We are also going to raise a case on this, @tfuda would be happy to share your case number? We will do the same here as well. cc @agarciaodeian
@tfuda I was checking your super simple piece of code in a Spring'16 and I found something .... If I run several times the test from the org, and the behavior was 1 worked, next one failed, then worked, then failed. Then decided to click on View Test History and clicked on Clear Test Data and since that moment, all executions worked fine. Does your test in the same way in your org?
In any case, please, share with us your case number and we will keep you update as well.
@afawcett - The case number we have open on this issue is 13125686.
@agarciaodeian - The org where I see this happening most frequently is on NA6. Basically I get about a 50% failure rate using the Test_InstanceofBug test class attached to this thread. Clearing the test history did NOT impact the frequency of the failures for me, one way or the other. FYI, this is NOT an an issue that occurs in unit test context only. This will also fail if you declare a top level class called MyException that extends Exception, and just execute these two lines of code as anonymous Apex:
Object ex = new MyException('This is MyException');
System.assert(ex instanceof Exception, 'Object is NOT instanceof Exception!');
Again, I see about a 50% failure rate in my NA6 org, but it's not always every other invocation. Sometimes it will fail three times in a row, and then pass three times in a row.
Thanks for the clarification @tfuda . We have raised a case also. This is the number: 13112538
We will keep you update with any useful response.
@tfuda we got a response from Salesforce support
R&D has logged a bug for the issue and create a known issue
They also let us know that the fix would be released next week (safe harbor) but in any case they recommended to click on the "This issue affects me", so please, go for it.
Thanks! You Financial Force folks get much faster response from Salesforce support than we do. We've been waiting on them to file a known issue for this for the past week.
I recently deployed the latest fflib-apex-mocks code into a couple of different Spring '16 Developer Edition org. When I tried running tests in the fflib_ApexMocksTest class, I got a number of failures. I've included details of the failures I received in the table at the bottom of this issue description. I found that these failures appeared to be intermittent, and in Spring '16 orgs only. After some debugging of one particular failure (PatronTicket.fflib_ApexMocksTest.whenStubExceptionTheExceptionShouldBeThrown: line 222, column 1), we discovered what appears to be a platform bug with that causes intermittent failures with statements that use the instanceof operator, like System.assert(ex instanceof MyException) statements. This particular assertion expects the caught Exception to be an "instanceof" the MyException class. Sometimes it is, and sometimes it isn't ;) I have opened a case with Salesforce on this, but I wanted to log it here as an issue, so that others using the fflib-apex-mocks framework are aware of this.