Announcing TestLab

[zpatten]

I have started development on my TestLab framework. This also includes work on the LXC gem. Cucumber-Chef will be migrated to this framework for version 4.0.

TestLab Gem: https://github.com/zpatten/testlab

LXC Gem: https://github.com/zpatten/lxc

Example: https://github.com/zpatten/testlab-repo

[zpatten]

Update: TestLab is progressing nicely. I will be starting work on a 4.0.0.rc cucumber-chef branch in the coming weeks.

[englishm]

Can I ask why you've decided to go this route rather than integrating with test-kitchen 1.0 and kitchen-lxc? It seems to me like there might be an awful lot of duplicated effort going into solving what is essentially the same set of problems.

[zpatten]

TestLab and Test Kitchen are apples and oranges. Test-Kitchen is singular in focus; TestLab is not.

[englishm]

I'm not sure I quite understand. Test-Kitchen 1.0 will soon have support for multiple provisioners (cf. opscode/test-kitchen#128), and already has support for multiple providers (e.g. vagrant lxc bluebox cloudstack digitalocean ec2 openstack rackspace) and test frameworks via busser plugins (e.g. bash, shunit2, bats, minitest, aruba, serverspec, ...). It looks like TestLab is targeted only at bringing up virtual infrastructure on LXC. Can you help me better understand how TestLab is more generic or where the project goals diverge?

I ask because I'm looking for ways we can adopt a TDD/BDD workflow for our infrastructure development and trying to figure out how the existing tools fit together (or don't).

[zpatten]

@englishm So hopefully I can clarify. Test-Kitchen is a great framework and does many of the same functions that TestLab does. But.

TestLab is meant to help you build scaled-down, fully functional representations of (your) complex infrastructure. You can try to do this with vagrant and traditional VM technologies, but you are going to run out of horsepower and memory fast if you are trying to replicate any sort of infrastructure at scale.

TestLab is also meant to allow you to programmatically interact with these representations, enabling programs such as cucumber-chef to provide integration testing using the TestLab framework for example.

Additionally TestLab allows for you to seamlessly interact with the network segments you have defined and have "servers" running on allowing for very intimate interaction with the infrastructure (i.e. servers) you have mocked up.

TestLab does all of the above via linux containers (i.e. LXC). Because containers are lightweight you can run many more "servers" than if you had each one virtualized under Vagrant for example.

When it comes to your development workflows, one of the most important features in TestLab is the use of LXC's ephemeral cloning. This allows you to bring up a container, make changes to it if needed (i.e. get it to a "pristine" state) and then you clone said container. Once the container is cloned you can make all the changes you want to it; brick it if you will even. When you want to revert the container back to a pristine state you re-clone and the container is almost instantly reverted back (which takes 2 seconds max from what I've seen). When cloning, the ephemeral container usurps the pristine container on the network and vice versa.

Hopefully this highlights some of the larger differences. Tools like Test-Kitchen are awesome and Fletcher is a damn good programmer (I've meet him in person as well). But Test-Kitchen is very static in that it is meant to only test in essence.

TestLab is meant to be used to test with and develop with and build and experiment with infrastructure as you would in a real computer lab. And this is only the beginning, I have many more exciting features in the pipeline for TestLab.

Right now we run all of the containers on a raring base box under a single vagrant VM. This is mostly to aid in development iteration (for me while working on TestLab). A local provider will be released soon which will allow running containers directly on the localhost.

[englishm]

Thank you for taking the time to write this detailed response. I have a better idea of where TestLab is headed now.

Incidentally, @LordCope has been working on a new project that sounds like it might fit my particular needs best: https://github.com/Atalanta/leibniz / http://leibniz.cc/

Thanks again for the detailed explanation, @zpatten. Hopefully it will also serve as a helpful guide to anyone else searching for the right tools in the future.

[zpatten]

No problem @englishm. I should note @LordCope's project is a extension/plugin for TestKitchen.

To be honest I hope that blurb above helps others too; the scope and complexity I am dealing with is often lost on most people and I get lumped into the "oh that looks like he's duplicating insert framework here".

In all honesty unless people can waste dollars on high end machines with massive amounts of RAM, vagrant is only going to allow someone to spin up 2 fully functional "servers" at most (people with MacBooks only have 8GB and only want to devote 2-3GB, 4GB max towards VM's; people with desktops are a little better off, but not by much).

If someone only has a handful of servers, roles and cookbooks, and your idea of integration testing is standing up two servers that "talk" to each other only in the vaguest sense, then the vagrant solution works well (FYI, standing up two servers at a time, to me, is not a true integration test). On the other hand if you have hundreds to thousands of servers and you have a massive amount of roles and cookbooks to support all those servers you surpass vagrant's capabilities with a quickness.

With all that in mind I mainly aim at supporting massive infrastructure; but the tools I build work just as well; and much, much faster, than using a vagrant VM per "server". On the flip side with TestLab, you have to have fairly broad and deep domain knowledge to leverage it properly; whereas TestKitchen and other solutions likely provide a more "plug-n-play" experience.

In short, I keep the needs of people who deal with hundreds or more servers and support the operations teams that manage those servers at the forefront of my mind because that is what I do every day at work and that is how I build my tools.