Closed rex closed 7 years ago
:-1: I'm not in favor of making any sort of mandate like this. A recommendation would be fine but the maintainers of each cookbook should be decided what testing to implement and the style, etc.
See my comment at https://github.com/chef-brigade/meta/issues/6#issuecomment-120942931 for a little more background on my reasoning.
@dblessing I agree with this. Re-reading this RFC I realize that the wording is a bit strong in that regard. For the sake of transparency I will leave the original RFC untouched but offer this ideological refactor: (Edits in bold text)
Whenever a new cookbook is "adopted" by our team, the assigned maintainers must consider implementing Kitchen tests before further development resumes. If a gradual approach is assumed, then development can and should continue with the understanding that the cookbook should eventually be thoroughly tested.
I am a fan of the "eventual consistency" approach in which all cookbooks we contribute to should ultimately follow the same (or similar) standards of unit testing (that is, to have unit testing at all). I absolutely think it should be the maintainers' responsibility to decide what type of testing is best for each cookbook but I feel strongly that testing of some sort should be in place across the board. That these are problematic cookbooks to begin with underscores the need for testing of some kind.
Thanks for clarifying, @rex My biggest concern is adding too much overhead. If the barrier to entry is too high then owners will not want to move their cookbook. I hope others will come here and share their thoughts as well.
I personally like the idea of having Kitchen tests although I can understand the fear of them being a requirement. I have been using test kitchen pretty extensively for the past year and I think the little bit of overhead is well worth it in how fast it lets you move. In addition it would allow us to better take contributions from the community which I think is the end goal here.
Could you turn this into a PR? At the very least, I'd like to have a living document with tips and tools for maintainers.
oh, one idea: designated experts who volunteer to work with maintainers to start using a particular tech. I'd much rather offer-help-building-awesome than demand-awesome-by-fiat.
@josephholsten i'd much rather take the approach you mention in your last comment. Recommendations, not mandates and have experience people available to help get going when a maintainer is interested. As the group gains momentum there will be natural pressure for maintainers to have good test coverage, style, etc.
First of all, I think test-kitchen and Serverspec are for integration tests and ChefSpec and RSpec for unit tests.
I agree with the idea of the need to recommend a (not required) testing approach.
There are a lot of tools for testing Chef Cookbooks and each person uses them in its own way (Rakefiles, Gemfiles, directory structure, CI service and all these weird stuff). Perhaps a document about recommended tests and how to implement them would be fine. Chef cookbook ecosystem is somewhat overloaded with tools and methodologies: test-kitchen, Serverspec, bats, ChefSpec, RuboCop, Travis CI, CircleCI, how to do the release bumps, what kind of tests to do and so on.
I'm not sure, maybe this is too much right now and having a list of recommended tools or tests is enough for now.
Once the cookbook is covered by tests, I think big and complex PRs will require tests. But not necessarily all the contributions. I would not establish it as a rule. Each case must be measured on its own.
I can try to work on one such document. I tend to prefer rubocop, foodcritic, test-kitchen, serverspec. I will get a PR in at some point in the coming days with the start of one such document. We can hash it over and go from there.
I modified the ticket entirely to describe requirements. Hope the subtasks are clear now.
The example stuff should probably go into https://github.com/chef-brigade/cookbook-skeleton
Documentation, guides, &c should probably stay in meta for the moment
I'm going to throw together a strawman integration-testing.md
and we can work out its contents in code review
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
[RFC] Integration Testing Guide
[This is a subset of the work for #13, please try to keep it synchronized]
Whenever a new cookbook is "adopted" by our team, we may implement integration tests. Here's a document with some guidance to getting started, resources, tools, processes and situations.
This is just a guide, it's not a requirement. If it doesn't make sense in your case, that's fine. Feel free to ignore it, or (even better!) update this doc to mention when it doesn't.
Please update this with your thoughts, comments, questions, concerns, and constructive criticism
Considerations
Advantages, Features & Benefits
Disadvantages, Downsides & Drawbacks
Processes
In the documentation, describe:
Tips
In the documentation, provide guidance to:
File Structure
Create examples of:
test/integration/default/bats/default.bats
.gitignore
.kitchen.yml
Berksfile
Gemfile
README.md
Rakefile
Tools
In the documentation, point to resources for:
bundler
rake
test-kitchen
ServerSpec