Cedar spec bundles don't execute in Xcode 7 Beta

alexbasson commented 9 years ago

The ios-school project uses a spec bundle to run tests. This works fine in Xcode 6.3. In Xcode 7 Beta, the project builds, but no tests are run. The console output is simply Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds.

tjarratt commented 9 years ago

Interesting, I wouldn't have thought that test bundles would have regressed in the beta. If anything, I would have anticipated that spec suites would have stopped working.

Were you able to find out anything else? Were there any exceptions raised and recovered from when you ran the tests?

akitchen commented 9 years ago

It is probably the result of the test case discovery mechanism changing. I don't assume it is an Xcode regression.

Suites probably still work fine, though I haven't had a chance to test it yet. This may reignite the conversation about inheriting directly from XCTestCase and friends.

On Tue, Jun 9, 2015 at 7:34 AM, Tim Jarratt notifications@github.com wrote:

Interesting, I wouldn't have thought that test bundles would have regressed in the beta. If anything, I would have anticipated that spec suites would have stopped working.

Were you able to find out anything else? Were there any exceptions raised and recovered from when you ran the tests?

Reply to this email directly or view it on GitHub: https://github.com/pivotal/cedar/issues/333#issuecomment-110381574

tooluser commented 9 years ago

There are some other lovely test output functionalities in Xcode 7.

On Jun 9, 2015, at 07:56, Andrew Kitchen notifications@github.com wrote:

It is probably the result of the test case discovery mechanism changing. I don't assume it is an Xcode regression.

Suites probably still work fine, though I haven't had a chance to test it yet. This may reignite the conversation about inheriting directly from XCTestCase and friends.

On Tue, Jun 9, 2015 at 7:34 AM, Tim Jarratt notifications@github.com wrote:

Interesting, I wouldn't have thought that test bundles would have regressed in the beta. If anything, I would have anticipated that spec suites would have stopped working.

Were you able to find out anything else? Were there any exceptions raised and recovered from when you ran the tests?

Reply to this email directly or view it on GitHub: https://github.com/pivotal/cedar/issues/333#issuecomment-110381574 — Reply to this email directly or view it on GitHub https://github.com/pivotal/cedar/issues/333#issuecomment-110392450.

codeman9 commented 9 years ago

I can confirm that test bundles don't run out of the box and that the xctest discovery process is different. Right now Cedar checks for an argument that is either -XCTest or -XCTestScopeFile. These arguments are no longer passed when running the test bundle. I have changed the discovery method to a different approach that does detect xctest runs, but the bundle still doesn't run any of the tests. It seems like the swizzle of defaultTestSuite is not actually getting called by the xctest system.

alexbasson commented 9 years ago

I just had an opportunity to speak with one of the Apple engineers responsible for testing framework development. After looking at my code, he suspects that whatever Cedar is doing to bootstrap tests doesn't depend on publicly available APIs and so is vulnerable to this kind of problem with each new Xcode release.

He pointed me to some new APIs available in Xcode 7, specifically the XCTestObservation protocol which declares methods such as -testBundleWillStart and -testSuiteWillStart, providing hooks that Cedar could potentially use in its bootstrapping stage.

akitchen commented 9 years ago

Outstanding. Thanks for following up!

On Wed, Jun 10, 2015 at 4:03 PM, Alex Basson notifications@github.com wrote:

I just had an opportunity to speak with one of the Apple engineers responsible for testing framework development. After looking at my code, he suspects that whatever Cedar is doing to bootstrap tests doesn't depend on publicly available APIs and so is vulnerable to this kind of problem with each new Xcode release.

He pointed me to some new APIs available in Xcode 7, specifically the XCTestObservation protocol which declares methods such as -testBundleWillStart and -testSuiteWillStart, providing hooks that Cedar could potentially use in its bootstrapping stage.

Reply to this email directly or view it on GitHub: https://github.com/pivotal/cedar/issues/333#issuecomment-110941840

mjstallard commented 9 years ago

Hey all,

We discovered a few more issues when trying out Xcode 7 on our current project which uses test bundles.

we're no longer seeing a summary of test failures at the end of a run - perhaps there's an issue with symbolication?
focussing specs does not appear to work (although marking specs as pending does work)
rivets do not appear next to specs in the editor window
executing an individual spec by tapping on its rivet in the Test Navigator runs the entire test suite
calling addObserver: on a Cedar fake raises an exception for us. It looks we end up calling through to NSObject's actual implementation. This worked fine for us in XCode 6.3. Note that the production implementation of the fake objects overrode -addObserver: to something entirely related to KVO. Renaming them to addBeholder: fixed those failures.

Hope this helps!

tjarratt commented 9 years ago

I was investigating the symbolication failures with @elanakoren yesterday and we discovered that it DOES work. Cedar reaches into the app bundle for Xcode to run atos, when we changed this to refer to the Xcode 7 beta version of atos, the specs for symbolication passed.

akitchen commented 9 years ago

I'm not sure this is the same issue. Is it?

On Tue, Jul 21, 2015 at 10:03 AM, Tim Jarratt notifications@github.com wrote:

I was investigating the symbolication failures with @elanakoren https://github.com/elanakoren yesterday and we discovered that it DOES work. Cedar reaches into the app bundle for Xcode https://github.com/pivotal/cedar/blob/master/Source/CDRSymbolicator.m#L211 to run atos, when we changed this to refer to the Xcode 7 beta version of atos, the specs for symbolication passed.

— Reply to this email directly or view it on GitHub https://github.com/pivotal/cedar/issues/333#issuecomment-123403323.

tjarratt commented 9 years ago

Well the symbolication tests start passing when we change @"/Applications/Xcode.app/Contents/Developer/usr/bin/atos"

to

@"/Applications/Xcode-beta.app/Contents/Developer/usr/bin/atos"

...so it very much feels to be the same issue to me.

Granted, I know very little about symbolication and why atos might be segfaulting when the tests are run from Xcode 7.

akitchen commented 9 years ago

Yes Tim, you have found the issue with Cedar’s own symbolication tests, however I don’t think this is related to the headline of this email thread. Please correct me if I’m wrong.

On Tue, Jul 21, 2015 at 2:26 PM, Tim Jarratt notifications@github.com wrote:

Well the symbolication tests start passing when we change @"/Applications/Xcode.app/Contents/Developer/usr/bin/atos" to @"/Applications/Xcode-beta.app/Contents/Developer/usr/bin/atos" ...so it very much feels to be the same issue to me.

Granted, I know very little about symbolication and why atos might be segfaulting when the tests are run from Xcode 7.

Reply to this email directly or view it on GitHub: https://github.com/pivotal/cedar/issues/333#issuecomment-123482505

tjarratt commented 9 years ago

Oh, I see. I was responding to some of the additional issues that @mjstallard found related to Xcode 7. Perhaps those should be logged as a separate issue. Sorry for the noise!

joemasilotti commented 9 years ago

Are there any updates on this? I haven't seen activity on the Xcode7 branch but wanted to double check.

tjarratt commented 9 years ago

There's been some slow, steady progress on Xcode 7 support for test bundles @joemasilotti. In the past few weeks we've fixed symbolication and discovered the root cause of why test bundles no longer have reporter output in the console after specs finish running.

To the best of my knowledge, running your specs in Xcode 7 beta 5 with Cedar in a test bundle should work today, using the Xcode7 branch. The only remaining issue to fix that I'm aware of is the missing reporter output.

tjarratt commented 9 years ago

Have you had any other, unrelated issues with Xcode 7 and test bundles lately, @joemasilotti?

joemasilotti commented 9 years ago

Not a single run of my test suite has completed on Xcode 7. Of the fifteen or so times I've attempted to do so, I receive EXC_BAD_ACCESS or EXC_BREAKPOINT exceptions.

These seem to be occurring on or around view lifecycle methods and when instantiating WebKit objects. Not sure if that's a red herring, though.

If it helps, I'm running ~1600 tests but the suite fails before getting to even 100 of them.

tjarratt commented 9 years ago

Could you provide a sanitized backtrace? It's not clear to me if the exception is being caused by Cedar or maybe something else from your description. If the backtrace is pretty bare, you might try enabling NSZombies and see if there's anything there.

I also saw something similar helping a pivot upgrade from Xcode 6 to 7, but the error seemingly went away when we cleaned the build folder. While I hate cargo-culting that "solution", that's also probably a relevant thing to try.

joemasilotti commented 9 years ago

Well this might all be moot now. Running my suite against Beta 6 raises EXC_BAD_ACCESS in main.m. Not much of a stack trace to share, but I'm interested in hearing if anyone else is experiencing this.

I know that a bunch of XCTest stuff changed with this release, specifically XCTestObservationCenter and XCTestProbe.

tjarratt commented 9 years ago

Absolutely right, Joe. The Xcode 7 branch uses XCTestObservationCenter to hook into XCTest's runtime in a somewhat saner way than we had before in the past.

So far, I know this runs Cedar's specs, some specs for my own projects, the iOS school specs, and a few other SF pivots' projects as well, but I wouldn't be too surprised if there weren't more lingering issues to resolve.

joemasilotti commented 9 years ago

That's reassuring. Has anyone had a chance to try this with Beta 6? I understand that that's a huge ask as it was released only yesterday.

akitchen commented 9 years ago

I'm pretty sure the WebKit stuff has internal async going on. I would consider it a red herring...

On Tue, Aug 25, 2015 at 5:04 PM, Joe Masilotti notifications@github.com wrote:

That's reassuring. Has anyone had a chance to try this with Beta 6? I understand that that's a huge ask as it was released only yesterday.

Reply to this email directly or view it on GitHub: https://github.com/pivotal/cedar/issues/333#issuecomment-134614958

joemasilotti commented 9 years ago

@akitchen meaning I have other issues to fix or Cedar is responsible for the crash? The crash only occurs when testing the app with Xcode 7, not when running the app in Xcode 6 or 7, nor when testing the app with Xcode 6.

akitchen commented 9 years ago

I would be very surprised if cedar is causing your EXC_BAD_ACCESS crash.

Cedar runs your specs synchronously on the main queue. A crash like this is typically due to side effects of something your code under test is doing. Do any tests run before this crash happens? Can you binary search to isolate the test or group of tests which are involved?

It would also be interesting to know if you are faking or spying instances of certain framework classes. This is really the only case where the crash could be related to cedar. Or do you think it is happening right at startup?

On Tue, Sep 1, 2015 at 12:26 PM, Joe Masilotti notifications@github.com wrote:

@akitchen meaning I have other issues to fix or Cedar is responsible for the crash? The crash only occurs when testing the app with Xcode 7, not when running the app in Xcode 6 or 7, nor when testing the app with Xcode 6.

Reply to this email directly or view it on GitHub: https://github.com/pivotal/cedar/issues/333#issuecomment-136833882

joemasilotti commented 9 years ago

Me too, which is why this issue has me so confused.

Any test that tries to instantiate a WKWebView crashes on -init. Simple as that. I do spy on the web view, but not until later in the test (code that is never reached). I removed all other tests except this simple example yet it still fails.

If it helps, I've attached a screenshot of the stack and exception.

akitchen commented 9 years ago

You might remember that we tend to fully stub out the UIWebView in testing scenarios, largely to avoid stuff like this but also to keep the tests fast and synchronous. WKWebView probably wants similar treatment.

Could you share a simple test as a gist which demonstrates this?

On Tue, Sep 1, 2015 at 1:30 PM, Joe Masilotti notifications@github.com wrote:

Me too, which is why this issue has me so confused.

Any test that tries to instantiate a WKWebView crashes on -init. Simple as that. I do spy on the web view, but not until later in the test (code that is never reached). I removed all other tests except this simple example yet it still fails.

If it helps, I've attached a screenshot of the stack and exception.

[image: exception] https://cloud.githubusercontent.com/assets/2092156/9615528/b6b34c8c-50c6-11e5-80a0-191110cde7e8.png

— Reply to this email directly or view it on GitHub https://github.com/pivotal/cedar/issues/333#issuecomment-136852749.

joemasilotti commented 9 years ago

Yea, that's a good point. I should probably be doing that anyway. Thanks for the insight.

I'm having trouble reproducing this in a fresh project, which makes me think there is something else I'm not thinking about. I'll post something when I can get it to reliably fail. Thanks for the help.

joemasilotti commented 9 years ago

Finally nailed down the issue. WKUserContentController has a "write-only" property for adding script handlers. ~~Very similar to how UIAlertController has "write-only" actions.~~ I added a spec helper to swizzle -addScriptMessageHandler:, add the name and message it to a dictionary, and then expose the blocks. Turns out this royally screws up WKWebView and related WebKit classes when testing under Xcode 7 on iOS 9.

That being said it doesn't sound like it's an issue related to Cedar. Thanks for walking me through the debugging process though, it was a huge help!

I'm going to open an issue in PCK to see if there are any better ways to accomplish what I am trying to do.

akitchen commented 9 years ago

Thanks for following up with more info @joemasilotti

Where does this leave us on this particular issue then?

joemasilotti commented 9 years ago

It's not related to Cedar at all. I am able to reproduce the issue with a plain old XCTest example. :-)

tjarratt commented 9 years ago

Thanks for all the feedback and testing everyone. Now that this has been merged into master, I'm more comfortable closing this.

Please open new issues for any new regressions that you may find while writing happy little Cedar specs.

cedarbdd / cedar