Automatic testing for 3rd party providers should be easier

[smathieu]

Most of the 3rd party providers listed here (https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins) are not tested. Those that are usually only test the Vagrant config since that's the only part that is easily tested.

Vagrant itself seems to have multiple facilities for acceptance testing in a isolated environment. It would be useful if those test helpers were to be extracted into a gem so that they can be reused across all plugins.

This should make contributing to plugins easier and also make the whole vagrant ecosystem more uniform.

[ukd1]

:+1:

@mitchellh thoughts?

[fgrehm]

I've had a quick chat with @mitchellh on https://github.com/mitchellh/vagrant/issues/1635#issuecomment-16837394 and his recommendations by then were to shell out to vagrant commands to test things out when doing acceptance testing. The downside of that is that you probably won't get code coverage numbers (but I'm personally ok with that :)

I've done acceptance testing with both rspec and Bats and I can say that using Bats for that seems like a better fit and it will make our specs more "future proof" :P

[fgrehm]

Oh, and on a side note, IIRC the current acceptance tests are deprecated

[mitchellh]

@fgrehm correct, historically.

I do want to reinstate some sort of acceptance testing system. There are too many parts of Vagrant that are too hard to test, and despite being ABYSMALLY slow [to run a full suite], it would really help a lot as a sanity test before release. As Vagrant grows and is integrated into more things, it is becoming more and more important to validate its correctness in some way prior to release.

@fgrehm Bats is interesting, but if I were to go for an official acceptance testing route, I would want a system that ran on Windows, as well. It may make sense to just use RSpec again.

What I'd like to do, ideally is create some sort of generic vagrant plugin test framework (library may be a better word) that plugins can plug into. For providers, there should be a generic list of tests that every provider needs to pass that will test the various features of Vagrant that are provider-specific (such as synced folder options and such).

Let's pull in @phinze: what are your thoughts on this?

[phinze]

Hi folks - I am Plus One Billion for improving the testing ecosystem both in and around Vagrant. So hooray for this issue! :rainbow:

There are too many parts of Vagrant that are too hard to test

I myself subscribe to the idea that difficulty testing is a code smell, and that the process of building up and maintaining tests around code can serve to improve the overall structure and quality of that code (in addition to the obvious benefit of automated verification of behavior).

Here are some opinions for you:

1. A generic vagrant plugin test library is a great idea, and I think one we should pursue.
2. A solid integration / acceptance / end-to-end suite is worth building, but it will be difficult to design well and slow to run.
3. I believe there is plenty left to be gained from bolstering the unit testing in the codebase. While some of the interfaces are hairy to wire up, I believe that with some effort we can build up patterns that will increase reliability and code quality.

Now, (2) and (3) are probably matters for other threads. I think for this issue what we'd want to figure out would be a modest initial feature set for this hypothetical vagrant test tool.

At this point I see two paths going in different directions:

(A) Provide something like "official stubs" which simulate the execution of vagrant in a unit-test style environment (B) Provide facilities for spinning up guests and verifying them

I believe both have the potential to be incredibly useful, and the creation of either involves substantial work however I think (A) is a better path to pursue for now. Ideally I'd like to see a suite of unit tests that verify a high-level Vagrant interface and can be used to verify the testing library as well - essentially something that binds the test helpers to the real vagrant code.

So there's a braindump for you. :thought_balloon: :wink: Where should we go from here?

[smathieu]

I'm really happy that I could help start this discussion. I also whole heartily support the creation of a testing library for Vagrant. I haven't been hacking on Vagrant for a long time, so I apologize in advance if these ideas aren't feasible or relevant, but here are a few thoughts.

1. Currently, it's fairly hard to test providers. It would be worth having the mentioned suite of common test for them, as well as a bunch of rspec matchers/helpers. Doing thing like isolating the environment is not extremely hard, but is painful to setup in all 3rd party plugins.

I feel that providing helpful guideline on testing would drastically improve the quality and stability of all plugins.

1. One of the approach I took (and that has failed) when I was writing acceptance tests for vagrant-kvm was than rather to boot a linux image in the acceptance test, I was trying to boot an empty image (without an OS). The idea is that I cared very little that the OS was booted properly, I simply wanted to test most of that the provider mostly interacted correctly with kvm. The issue I ran into is that a lot of Vagrant makes assumption on the OS being booted. For instance, it assumes that it can mount NFS drives onto it.

A fast way of acceptance testing most of the code base would be for Vagrant to support OSes that do not have SSH and NFS (or any kind of network really) support. That way, when we shell out to Virtualbox or similar, the time spent booting the OS would be greatly reduce.

This is obviously not a complete enough test for everything is Vagrant, but it could serve to reduce the run time of these acceptance tests drastically.

1. Testing actions is very hard because the environment passed to the call method is very large and thus difficult to mock. The only thing I can think would help here (short of redesiging the whole interface of actions) would be to have helpers to create mock environments. It would be nice to have something a la FactoryGirl to generate custom mock environments with sane defaults.

I hope these ideas help.

[mitchellh]

I agree with both of you.

I think the solution is to have a separate library (vagrant-testing ?) that has a bunch of helpers that make testing very easy. This can also include rspec matchers/helpers/etc. but we should probably make the helpers agnostic so that people can use whatever test framework they'd like.

These helpers can provide both helpers for unit testing (easy mocking of complex objects) as well as acceptance testing (creating Vagrantfiles).

I think if we make the version of the vagrant-testing library the same as Vagrant, then we can say "if you want to test your plugin against Vagrant 1.4, use version 1.4 of vagrant-testing" and so on.

Thoughts?

[rogeriopradoj]

[totally off-topic]... but how hard I'd like to program in Ruby to help with that! On Dec 4, 2013 2:00 PM, "Mitchell Hashimoto" notifications@github.com wrote:

I agree with both of you.

I think the solution is to have a separate library (vagrant-testing ?) that has a bunch of helpers that make testing very easy. This can also include rspec matchers/helpers/etc. but we should probably make the helpers agnostic so that people can use whatever test framework they'd like.

These helpers can provide both helpers for unit testing (easy mocking of complex objects) as well as acceptance testing (creating Vagrantfiles).

I think if we make the version of the vagrant-testing library the _same_as Vagrant, then we can say "if you want to test your plugin against Vagrant 1.4, use version 1.4 of vagrant-testing" and so on.

Thoughts?

— Reply to this email directly or view it on GitHubhttps://github.com/mitchellh/vagrant/issues/2550#issuecomment-29817610 .

[mitchellh]

I've created the repo: https://github.com/mitchellh/vagrant-testlib

I'm going to work on the acceptance testing side of things because I want to get some basic acceptance tests in before releasing 1.4. But eventually I want it to support a lot more.

[mitchellh]

I've got a lot of boilerplate in! Progress pic:

I renamed the repo to "vagrant-spec" similar to "rubyspec" since the spirit is that it is an executable specification of what makes Vagrant Vagrant.

No docs yet, but pretty good for day one.

[mitchellh]

Another update. Things are looking good. I merged a branch into Vagrant that adds the new acceptance test suite. See: https://github.com/mitchellh/vagrant#acceptance-tests

I'm working on generic provider tests now.

[fgrehm]

Joining the party a bit late....

@fgrehm Bats is interesting, but if I were to go for an official acceptance testing route, I would want a system that ran on Windows, as well. It may make sense to just use RSpec again.

Yes, I keep forgetting about Windows users :(

So... I think you guys covered pretty much I'd have to say... But just to make sure no one thinks I'm against acceptance specs, I'm actually pretty happy that this is being discussed and acted upon :) (@smathieu thanks a lot for kicking this off!) As @phinze said I'm also Plus One Billion for improving the testing ecossystem around Vagrant, I have personally written quite a few plugins and most of them lack testing because I'm lazy testing vagrant plugins is not a trivial task.

I'd also want to highlight the importance of having acceptance specs around Vagrant itself by sharing with you guys a thing I noticed a while ago but ended up forgetting to share with @mitchellh. If you pay enough attention at the changelog, you might notice that since 1.1.0 came out (and at the same time the acceptance specs were deprecated I think) there is a "trend" of "hotfix releases" coming out on the same day or on the following 1 / 2 days after a release:

• 1.1.1 (March 18, 2013) -> 1.1.2 (March 18, 2013)
• 1.1.3 (March 25, 2013) -> 1.1.4 (March 25, 2013)
• 1.1.5 (April 2, 2013) -> 1.1.6 (April 3, 2013)
• 1.2.0 (April 16, 2013) -> 1.2.1 (April 17, 2013)
• 1.2.5 (July 26, 2013) -> 1.2.6 (July 26, 2013) -> 1.2.7 (July 28, 2013)
• 1.3.0 (September 5, 2013) -> 1.3.1 (September 6, 2013)
• 1.3.2 (September 17, 2013) -> 1.3.3 (September 18, 2013)

Please don't get me wrong, I'm not questioning anyone's work here, I just want to show you guys that since most of that was related to bug fixing, not only I believe part of that could have been picked up on some level of testing (both unit and acceptance) but also we lost the opportunity to implement regression tests for some things that were fixed along the way.

That said, I'm pretty excited about vagrant-spec and improving testing in general on vagrant itself and I'll be watching the project closely to see how it evolves. I also have plans to do some pretty big refactoring on the lxc provider codebase and having generic provider tests that I can just drop in to make sure I don't break things along the way will be GREAT.

@mitchellh, please give us a heads up once things are somewhat stable over vagrant-spec so that we can go crazy experimenting with it :P

[mitchellh]

OKAY. I think things are stable enough over in vagrant-spec to take a look. I wrote a ton of docs in the README: https://github.com/mitchellh/vagrant-spec/blob/master/README.md

Please check it out. As I mentioned earlier, I only focused on acceptance tests for now in the interest of getting Vagrant 1.4 out the door. But I hope that we will eventually augment this library to also provider unit testing helpers that we can dogfood in Vagrant core.

@fgrehm My secret has been found! Haha, no, I'm kidding. But yes, I noticed this trend as well and hate it. One of my big motivations behind vagrant-spec is to stop this trend, definitely.

[mitchellh]

Closing this. Work on this is over at vagrant-spec, which we need to improve further.