search

cedarbdd / cedar

BDD-style testing using Objective-C
http://groups.google.com/group/cedar-discuss
1.19k stars 140 forks source link

Xcode 9 compatibility #403

Open DogeWowzie opened 6 years ago

DogeWowzie commented 6 years ago

Are there any known compatibility issues with Xcode 9 and Cedar? So far I am experiencing strange side effects like looping unit tests and unfoldable spec scopes. Codebase is Objective C.

tjarratt commented 6 years ago

Hey @DogeWowzie - this is the first I've heard of it. Unfortunately we don't have a lot of teams within Pivotal that are using Cedar much these days - possibly none at all. In short, we don't find out that there are problems and that Xcode and Cedar are no longer best friends until someone files a helpful issue, like yourself :)

Have you investigated at all yet ? Are you sure that a small test case illustrates the same problem as your codebase does ? If you create a bare-bones minimal test suite with a single test case, do you see the same behavior ?

During the next few weeks I'll try to dedicate some of my own time to look into this, but I'm a parent, have several days I'll be out of the office on vacation here in France, running interviews and managing a team in my office, so I can't promise a fix on any given timeline. That said, I love helping first-time contributors to open source projects and I'd love to help you out if you're interested in digging in. Let me know how I can best help you, if you'd like :)

DogeWowzie commented 6 years ago

Hey @tjarratt, thank you for your reply.

Cedar tests in our project just work fine with Xcode 8, we tried to isolate the problem by deactivating "suspicious" spec classes, but the "looping problem" always occurs when the test target contains a certain number of specs, that's why we think it might be a general problem due to the recent XCTest changes of Xcode 9.

Furthermore, we feel like Cedar tends to be behave inconstantly in Xcode 9, some specs classes are only succeeding while being focussed. I appreciate your effort in having a look into this.

Thank you very much in advance :)

tjarratt commented 6 years ago

Regarding "cold folding" - I've opened up a separate issue to discuss that. Apparently apple broke it when they introduced a new editor in Xcode 9.

tjarratt commented 6 years ago

@DogeWowzie have you tried to create a new, standalone project and see if you can reproduce the same behavior ? I've been running Cedar's specs today in Xcode 9.2 and I haven't yet been able to reproduce this yet.

The thing that would be the most beneficial to me is having a standalone project (ideally something I could put on my machine and debug) so that we have a realistic scenario to explore. I'd say the chances are pretty slim of me being able to fix the bug alone, unless I happen to come across some tests in Cedar's test suite that happen to reproduce the issue.

tjarratt commented 6 years ago

@DogeWowzie I've tried a few times to replicate this with Xcode 9.3, comparing my results with Xcode 8.3.3 and haven't seen any cases where specs have been running repeatedly - even for some of our larger specs where we have something like 200 examples inside a single spec file.

Have you found out anything new ? Ideally, in order to resolve this, I would want to be able to dig into a codebase that consistently reproduces this problem that I can run on my machine. If you can upload a stand-alone example of this as a repo on github that would allow me to make a lot more progress.

Barring that I'd love to know some other details that might help identify the problem:

DogeWowzie commented 6 years ago

Hey @tjarratt, sorry for the late reply, we took a closer look into our "looping" test target. We have three targets with specs, only one loops. This problem appeared since we updated to Xcode 9, we are currently running Xcode 9.31 with iOS 11.3 simulators.

Turns out specs are looping due to our excessive usage of spy_on in that test target. Now our tests don't loop anymore when we perform a "clean build folder" with Xcode in advance. Everything seems fine now, but we were kind of confused by the fact that everything works just fine with Xcode 8.

tjarratt commented 6 years ago

Oh wow, that's really interesting. spy_on and friends are some dark magic, for sure - I've seen a number of interesting bugs related to Cedar spies throughout the years, especially when you throw Swift into the mix.

I wish I had thought of cleaning the build folder. When strange things start to happen, that's usually my initial instinct. My best guess to explain this behavior is that perhaps there were some changes in XCTest between Xcode 8 and 9 and Xcode somehow managed to load multiple versions of the same spec that were compiled separately, but somehow loadable into the same process after you upgraded, if it was using the same build directory. Normally this changes between releases, but who knows in this case.

Are there any more issues to investigate, or do you feel we could close this issue for now ?

DogeWowzie commented 6 years ago

Hey @tjarratt, I feel like closing this issue for now. Please note that we refactored all the specs (removed the spy_ons) and then managed to prevent the specs from looping by performing "clean build folder" afterwards. If we don't do that, looping does still occur, but the spec target itself feels more stable. Thank you very much for your patience :) Cheers