Acceptance Testing: Cucumber, Spinach, or RSpec/MiniTest?

[joeyjoejoejr]

There has been a decent amount of discussion about this topic in #2 about acceptance testing and I think it deserves it's own topic.

A couple of questions to ponder

• Do we believe acceptance testing is valuable? (Does it reduce costs? What function does it serve? What effects does it have on developer happiness)
• Does this need rise to the level of including a default setup in box-cutter?
• Which one do we like best? Why? What are the tradeoffs?

[joeyjoejoejr]

Let me start with my thoughts on these questions.

I find value in having acceptance(end-to-end) level tests in the TDD process and as regression protection, I don't know how many times, in applications that don't have a basic acceptance test suite, I've ended up getting calls from the client saying they are getting 500 errors. I think a lot of time and money can be saved by having some basic end-to-end tests.

That being said, I have seen far too many acceptance test suites that take 20 minutes to run and/or break for no good reason. This eliminates the value in my opinion. We should be careful only run happy path and business critical functions.

I think that we should have a standard of how we write acceptance tests, and include it in box cutter.

I actually don't really care which one we use, however It looks like Spinach may be a better paradigm than Cucumber. I've also written a number of acceptance tests in RSpec, and I find them to be easy enough to read and work on. But I've also done some work with Cucumber and am fine with that as well.

[reedlaw]

I think it really depends on the client. If the client thinks best in Excel, then maybe FitNesse. If the client understands user stories, then Spinach may be best. If the client can't involve themselves in acceptance testing for whatever reason, then simple RSpec tests would be the easiest to write and maintain. Maybe this is something best left out of Box Cutter and left as a decision for each project.