Testing - Githubissues

derekgroh commented 10 years ago

Include testing support using chef-spec? test-kitchen?

miguelaferreira commented 9 years ago

chef-spec matchers have been added in 29f24a31d3051eec43d2fa0ca619d7416b452c39. I do have some stuff for testing with Vagrant (kitchen-winrm plugins were not mature enough, but I know there have been developments).

I wound't mind committing all my testing code and configuration. The only thing I can't make available is the windows vagrant boxes I use (due to licensing).

derekgroh commented 9 years ago

@miguelaferreira As long as your vagrant boxes are built without a registered Microsoft license OEM, you should be able to use reference those vagrant boxes for others to use.

Alternatively, are there any significant changes or modifications you've made to the vagrant base box that wouldn't be covered by any of the existing boxes on https://vagrantcloud.com/ ?

miguelaferreira commented 9 years ago

@derekgroh I will investigate further this issue. We have a automated build with packer that spits out boxes (cause they need to be renewed every 180 days or so). It would be great to post them to vagrant cloud.

Regarding modifications, we do make sure:

a vagrant user with password vagrant is created
that a new SID be generated when the box is spun-up the first time
that winrm is enabled
that RDP is enabled
to set a lenient password policy

jhsubscribe commented 8 years ago

I'm obviously very new to this project but I was thinking it might be a good idea to try a .kitchen.yml based on the one in the "windows" cookbook (ie using the same vagrant boxes) https://github.com/chef-cookbooks/windows/blob/master/.kitchen.yml .

The main barrier I see to using test-kitchen is moving the test recipes out into the normal recipes folder (maybe prefix them with test_ and add comments to make clear they're only for test usage?).

derekgroh commented 8 years ago

From my initial dive into this, it seems we will need syspreped guests or ensure that testing is controlled in that a domain controller and domain member are never the same box. I've started work on building syspreped systems for use with a .kitchen.yml but have only been successful with server 2012 family. These boxes can be found here: https://atlas.hashicorp.com/derekgroh

Speaking with the Chef community folks there is a further need to get unit testing up and running as well as update the providers to use Chef 12. I need to do a deep dive into unit testing in order to be able to help grow it, but @miguelaferreira maybe able to fill in the gaps currently.

miguelaferreira commented 8 years ago

I've been using vanilla images to test with kitchen. Then I have a test cookbook that does the prepping. Having pre-preped images does make the test run a bit faster, which is great, but also increases the maintenance effort in keeping the images up to date.

On the actual testing:

It is a really bad idea to mix the test and production code, so I would say :-1: to @jhsubscribe's idea of moving test recipes to the normal recipes folder. The test cookbook can and should be used for testing only. If it has functionality that we want to include in the main cookbook, then we should just implement that in a recipe and remove it from the test cookbook.
unit-testing (with RSpec for pure ruby code, and ChefSpec for chef code) is a great idea, but does require more programming/testing knowledge than just system-testing (with kitchen). One big advantage is that we don't actually need a Windows box to run the unit tests. A Travis build would be enough to run all unit-tets
system-testing with kitchen requires dedicated infrastructure. On our laptops that can very well be Virtualbox, but if we want to have automated builds for PRs (for example) we will need to have dedicated infrastructure for it (AFAIK there is no free CI service out there that provides windows VMs).

I currently don't have the time to spend on all thse things, but I can point you to a very small cookbook I made that contains ChefSpec tests (under spec), uses kitchen (two kitchen files, one meant for testing on the laptop, and another for testing in a CI server) with ServerSpec tests (under test/ntegration/default/serverspec), and has a script that puts it all together (run-ci-build.sh). The script that puts it all together would best be implemented as a Rake task instead of a bash script.

Feel free to ping me if you need help with something.

sous-chefs / windows_ad

Testing #11