[RFC] Unit testing with Kitchen Guide

[rex]

[RFC] Integration Testing Guide

[This is a subset of the work for #13, please try to keep it synchronized]

Whenever a new cookbook is "adopted" by our team, we may implement integration tests. Here's a document with some guidance to getting started, resources, tools, processes and situations.

This is just a guide, it's not a requirement. If it doesn't make sense in your case, that's fine. Feel free to ignore it, or (even better!) update this doc to mention when it doesn't.

Please update this with your thoughts, comments, questions, concerns, and constructive criticism

Considerations

Advantages, Features & Benefits

• Testing would allow us to weed out possible bugs in the cookbook
• Testing would allow us to learn how the cookbook was originally implemented by its author (oddities, caveats, internal magic, etc)
• Testing would allow us to implement a strong foundation in each cookbook that would streamline collaborative development and maintenance
• Testing lends more credibility to our efforts (Read: Makes us look damn good to the community)
• Testing would allow us to properly automate deployment of new releases to the Supermarket

  Disadvantages, Downsides & Drawbacks

• Testing would take some time to implement initially in a way that satisfies most/all team members due to (possibly) differing opinions on implementation
• Adding testing to each cookbook individually would take some time
• Very likely that there will be urgent outstanding issues in these cookbooks that we will want to resolve prior to any other development
• test-kitchen and Serverspec are for integration tests and ChefSpec and RSpec for unit tests.

  Processes

In the documentation, describe:

• [ ] adding integration testing to an existing project
• [ ] installing the dependencies
• [ ] writing an integration test
• [ ] running an integration test

  Tips

In the documentation, provide guidance to:

• [ ] recommended tests and how to implement them
• [ ] when to use unit tests vs integration tests

  File Structure

Create examples of:

• test/integration/default/bats/default.bats
• .gitignore
• .kitchen.yml
• Berksfile
• Gemfile
• README.md
• Rakefile

  Tools

In the documentation, point to resources for:

• bundler
• rake
• test-kitchen
• ServerSpec

[dblessing]

:-1: I'm not in favor of making any sort of mandate like this. A recommendation would be fine but the maintainers of each cookbook should be decided what testing to implement and the style, etc.

See my comment at https://github.com/chef-brigade/meta/issues/6#issuecomment-120942931 for a little more background on my reasoning.

[rex]

@dblessing I agree with this. Re-reading this RFC I realize that the wording is a bit strong in that regard. For the sake of transparency I will leave the original RFC untouched but offer this ideological refactor: (Edits in bold text)

Whenever a new cookbook is "adopted" by our team, the assigned maintainers must consider implementing Kitchen tests before further development resumes. If a gradual approach is assumed, then development can and should continue with the understanding that the cookbook should eventually be thoroughly tested.

I am a fan of the "eventual consistency" approach in which all cookbooks we contribute to should ultimately follow the same (or similar) standards of unit testing (that is, to have unit testing at all). I absolutely think it should be the maintainers' responsibility to decide what type of testing is best for each cookbook but I feel strongly that testing of some sort should be in place across the board. That these are problematic cookbooks to begin with underscores the need for testing of some kind.

[dblessing]

Thanks for clarifying, @rex My biggest concern is adding too much overhead. If the barrier to entry is too high then owners will not want to move their cookbook. I hope others will come here and share their thoughts as well.

[zarry]

I personally like the idea of having Kitchen tests although I can understand the fear of them being a requirement. I have been using test kitchen pretty extensively for the past year and I think the little bit of overhead is well worth it in how fast it lets you move. In addition it would allow us to better take contributions from the community which I think is the end goal here.

[josephholsten]

Could you turn this into a PR? At the very least, I'd like to have a living document with tips and tools for maintainers.

[josephholsten]

oh, one idea: designated experts who volunteer to work with maintainers to start using a particular tech. I'd much rather offer-help-building-awesome than demand-awesome-by-fiat.

[dblessing]

@josephholsten i'd much rather take the approach you mention in your last comment. Recommendations, not mandates and have experience people available to help get going when a maintainer is interested. As the group gains momentum there will be natural pressure for maintainers to have good test coverage, style, etc.

[zuazo]

First of all, I think test-kitchen and Serverspec are for integration tests and ChefSpec and RSpec for unit tests.

I agree with the idea of the need to recommend a (not required) testing approach.

There are a lot of tools for testing Chef Cookbooks and each person uses them in its own way (Rakefiles, Gemfiles, directory structure, CI service and all these weird stuff). Perhaps a document about recommended tests and how to implement them would be fine. Chef cookbook ecosystem is somewhat overloaded with tools and methodologies: test-kitchen, Serverspec, bats, ChefSpec, RuboCop, Travis CI, CircleCI, how to do the release bumps, what kind of tests to do and so on.

I'm not sure, maybe this is too much right now and having a list of recommended tools or tests is enough for now.

Once the cookbook is covered by tests, I think big and complex PRs will require tests. But not necessarily all the contributions. I would not establish it as a rule. Each case must be measured on its own.

[zarry]

I can try to work on one such document. I tend to prefer rubocop, foodcritic, test-kitchen, serverspec. I will get a PR in at some point in the coming days with the start of one such document. We can hash it over and go from there.

[josephholsten]

I modified the ticket entirely to describe requirements. Hope the subtasks are clear now.

[josephholsten]

The example stuff should probably go into https://github.com/chef-brigade/cookbook-skeleton

Documentation, guides, &c should probably stay in meta for the moment

[josephholsten]

I'm going to throw together a strawman integration-testing.md and we can work out its contents in code review

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.