CocoaPods event: Testing panel/discussion

[orta]

Hosted at Spotify NYC, on Feb 25th. Organizing with Json Palmer @palmerj3

Panel:

• [ ] @dblock
• [ ] @benjaminjackson
• [ ] @lazerwalker
• [ ] @klaaspieter
• [ ] @paulyoung
• [ ] @btgerst

Host: @orta

Major Topics:

• QA vs Automated Unit testing
• TDD vs BDD
• iOS implementation strategies vs other dev ecosystems
• Ways to improve it for everyone

[orta]

Open to discussion around topics, ideas etc on here.

[lazerwalker]

It might be worth touching on the difference between unit, acceptance, and integration-level testing, and iOS integration testing tools like Frank and Kif (I'm admittedly not familiar with either of 'em personally).

[klaaspieter]

I have some experience with Kif and I'm doing integration testing in an interesting way (in my opinion). Would love to talk about it and also explain why I'm not using things like Kif and Frank.

@orta Are we doing a Q/A panel, talks or both? Also when you say 25th I assume that means February 25th?

[dblock]

I want to know why it's so hard and why nobody is doing anything about it? :) OK, more seriously, just the first part.

[paulyoung]

@lazerwalker and I were discussing how we might test the following or refactor it to be more testable: https://gist.github.com/paulyoung/8709561f37560f59aee4

[paulyoung]

For a seemingly simple setup there's a lot that's not easy to test IMO. At least, not without getting caught up in implementation details.

e.g. how to test that the params are passed along correctly, or that the block gets invoked correctly on success and failure

[paulyoung]

Also, probably worth talking about testing MVVM.

[orta]

Yes the 25th of Feb, I'll figure a way to announce over the weekend.

The format I have in my head is similar to the oxford style but without such enforced us vs them-isms. As in general everyone is on board with the concepts but ideas around implementations and methodologies are different. I'd have you guys up front on stage split into two tables.

We'd have 4-6 major topics, I'd give a brief discussion about each one giving an overview of the problem and ideas around it then pass it on to one of you guys to make a starting statement. Someone'll probably have something to start with based on my middle-of-the-road opinions anyway. Whilst the discussion is ongoing my job will be to poke people who've not said too much to offer something up. Or to throw an opinion into the mix.

We do 2/3 - take a break - grab fresh beers - do another 2/3. Thank the audience, thank the host, thank the speakers, then tell everyone to fix their cocoapods install.

[klaaspieter]

And then we'll party!

[bgerstle]

:+1: to discussing differences between various kinds of tests (i.e. user, acceptance, functional, integration, unit, etc.).

Other areas I think are worth discussing:

State vs. Behavior Verification

• Pros and cons
• Frameworks

I'm only familiar with OCMock, which we're using for BDD in some cases and trad'l TDD plus some mocking/stubbing utilities in others. It's also an interesting case study in the ObjC runtime, but that's probably beyond the scope of this meetup.

Mocking

• Design patterns for separation of concerns in the client and how that relates to mocking
• Approaches for mocking a backend service (e.g. response schema validation, caching behavior, # of round trips, etc.)

Test Coverage

This is something I've only hacked around a bit with using the CoverStory as well as lcov and gcov, but I'm curious if anyone else is doing anything with it. It seems well established in other languages environments (e.g. Java/IDEA/maven), but a bit lacking in iOS.

Automated(?) Performance Regression Testing

This is a topic I've been quite interested in lately, and would love to hear about anyones experiences with this. I've tried to think of ways to do it, but haven't had time to prototype any of them (yet!). The idea being you could pair performance metrics with automated tests to measure differences in application performance across version, and catch regressions before they're merged.

[orta]

Re: performance testing, there was an interesting article about performance testing UIViews where they would generate them and shake them around a bit and use numbers from that to generate the tests.

Re: Test Coverage this got mentioned in https://github.com/orta/life/issues/15 but @jasperblues mentioned his work on xcode-gradle which might also be of interest: https://github.com/jasperblues/xcode-gradle which aims to provide some of these tools wrapped together.

The other two are great ideas also.

[orta]

http://m.youtube.com/watch?v=QvHf94hxzRc&desktop_uri=%2Fwatch%3Fv%3DQvHf94hxzRc

[orta]

Also, mobile app performance startup: http://pulse.io

[orta]

Re: tooling.

I've seen a lot of recent activity around libClang, mix that with automation tools like CocoaPods / xctool. Could there other ways to test or ways to improve.

[landonf]

Orta sent me in this direction, so I figured I'd jump right in.

Hello!

Automated(?) Performance Regression Testing

This is a topic I've been quite interested in lately, and would love to hear about anyones experiences with this. I've tried to think of ways to do it, but haven't had time to prototype any of them (yet!). The idea being you could pair performance metrics with automated tests to measure differences in application performance across version, and catch regressions before they're merged.

Back in 2008, I wrote a simple xUnit-style library for automated performance analysis, but never got around to the automatic performance regression detection problem: https://code.google.com/p/plinstrument/

My thinking was that I'd implement a new PLInstrumentResultHandler that output a machine-readable format for automated comparison, but in practice, I've only used it to perform directed iterative optimization as part of the development process (eg, write a plain C implementation, measure it to see how fast it is, try optimizing in C, drop to SIMD, see if that has improved anything, and so on).

Even in that capacity, it's really handy to be able to close the code+test loop and quickly run reproducible micro-benchmarks of small changes on multiple targets.

Of course, there's more than just CPU utilization to consider; being able to automatically compare memory utilization, power efficiency, etc, across builds and releases would be really handy ...

[orta]

Closing this issue, thanks guys!

[jasperblues]

Great discussion @orta . . . although this is closed now, I wanted to raise one point with @lazerwalker

I've always called tools like kif, calabash, iCuke, etc automated functional testing. . . here's the definitions that I'm familiar with from other languages:

* Unit Test:*

Testing a class in isolation from significant collaborators using stubs, mocks or other test doubles

Advantages: Pinpoints failure. Usually very fast. Proves a component is fit for purpose, before integration.

Disadvantages: Leads to 'glass box' style testing - to use a mock or a stub you have to look inside a class, beyond the external interface.

Integration test:

tests a class along with its significant collaborators or dependencies.

• Advantages - proves a use-case works end to end. Sometimes offer "biggest testing bang for your buck". . . executing across several components.
• Disadvantages - Can't pinpoint exactly where something goes wrong. . . Can be difficult to set up or put the system into the required state. . Can have side effects that can't be easily rolled back.

Functional Test

Tests the system through its top-level external interface. In the case of iOS this would be the UI.

[lazerwalker]

Yeah, you're right — I used to work with a bunch of folks who (erroneously) tended to use those terms interchangeably, so I'm bad about misspeaking :). I agree 100% with your nomenclature.