State of Test Kitchen Suite

[zarry]

This is an ongoing issue to track the test kitchen state across the mongodb cookbook. As I get some time I will run these tests initially to get a view of where we stand. From there we can choose what tests to fix, enhance, remove etc...

Instance	Driver	State	Comments
default-ubuntu-1204	Vagrant	Passing	one test, checks mongo started
default-ubuntu-1004	Vagrant	Passing	one test, checks mongo started
default-debian-720	Vagrant	Passing	one test, checks mongo started
default-centos-70	Vagrant	Passing	one test, checks mongo started
default-centos-65	Vagrant	Passing	one test, checks mongo started
default-centos-510	Vagrant	Passing	one test, checks mongo started
default-mongodb-org-repo-ubuntu-1204	Vagrant	Failing	#49
default-mongodb-org-repo-ubuntu-1004	Vagrant	Failing	#50
default-mongodb-org-repo-debian-720	Vagrant	Failing	#51
default-mongodb-org-repo-centos-70	Vagrant	Converging	No Tests
default-mongodb-org-repo-centos-65	Vagrant	Converging	No Tests
default-mongodb-org-repo-centos-510	Vagrant	Converging	No Tests
replicaset-ubuntu-1204	Vagrant	Failing	#52
replicaset-debian-720	Vagrant	Failing	#53 Two tests: start mongo, replica set up(failing)
replicaset-centos-70	Vagrant	Failing	#54
replicaset-centos-65	Vagrant	Not Executed
replicaset-centos-510	Vagrant	Not Executed
configserver-ubuntu-1204	Vagrant	Not Executed
configserver-ubuntu-1004	Vagrant	Not Executed
configserver-debian-720	Vagrant	Not Executed
configserver-centos-70	Vagrant	Not Executed
configserver-centos-65	Vagrant	Not Executed
configserver-centos-510	Vagrant	Not Executed
mongos-ubuntu-1204	Vagrant	Not Executed
mongos-ubuntu-1004	Vagrant	Not Executed
mongos-debian-720	Vagrant	Not Executed
mongos-centos-70	Vagrant	Not Executed
mongos-centos-65	Vagrant	Not Executed
mongos-centos-510	Vagrant	Not Executed
mms-monitoring-agent-ubuntu-1204	Vagrant	Not Executed
mms-monitoring-agent-centos-70	Vagrant	Not Executed
mms-monitoring-agent-centos-65	Vagrant	Not Executed
mms-backup-agent-ubuntu-1204	Vagrant	Not Executed
mms-backup-agent-centos-70	Vagrant	Not Executed
mms-backup-agent-centos-65	Vagrant	Not Executed
user-management-ubuntu-1204	Vagrant	Failing	#48
user-management-debian-720	Vagrant	Failing	#47
user-management-centos-70	Vagrant	Passing	Three tests: Start mongo, require authentication, create user
user-management-centos-65	Vagrant	Not Executed
user-management-centos-510	Vagrant	Not Executed

[rex]

Thanks for opening this issue @zarry. I'll run tests throughout the day and post results of each individually so we can get a better idea of where we stand.

[zarry]

Likewise @rex I will be running them on and off throughout the day/evening. I will start from the top maybe you start from bottom so we don't overlap?

[rex]

@zarry Sounds like a plan!

[rex]

As an aside, just to make sure we're on the same page: I am going to run tests from the current state of the master branch. Additionally, I'm not going to modify any code if a test fails. This is purely to assess the current state of affairs in the master branch so that we can plan our next moves.

[zarry]

@rex :+1: agreed.

[rex]

Hmm. I'm getting a weird failure in my first test. hostnamectl: command not found. It doesn't seem particularly related to this cookbook, which suggests it's a configuration issue. I'm looking into it, but have you seen anything like this on your end @zarry?

Full stack trace:

    default: Guest Additions Version: 4.3.8
    default: VirtualBox Version: 5.0
==> default: Setting hostname...
STDERR: The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

hostnamectl set-hostname 'user-management-ubuntu-1204'

Stdout from the command:

Stderr from the command:

stdin: is not a tty
bash: line 2: hostnamectl: command not found
---- End output of vagrant up --no-provision --provider virtualbox ----
Ran vagrant up --no-provision --provider virtualbox returned 1/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/mixlib-shellout-2.1.0/lib/mixlib/shellout.rb:289:in `invalid!'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/mixlib-shellout-2.1.0/lib/mixlib/shellout.rb:276:in `error!'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/shell_out.rb:70:in `run_command'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/kitchen-vagrant-0.18.0/lib/kitchen/driver/vagrant.rb:318:in `run_command'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/kitchen-vagrant-0.18.0/lib/kitchen/driver/vagrant.rb:302:in `run'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/kitchen-vagrant-0.18.0/lib/kitchen/driver/vagrant.rb:348:in `run_vagrant_up'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/kitchen-vagrant-0.18.0/lib/kitchen/driver/vagrant.rb:88:in `create'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:424:in `public_send'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:424:in `block in perform_action'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:485:in `call'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:485:in `block in synchronize_or_call'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:483:in `synchronize'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:483:in `synchronize_or_call'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:453:in `block in action'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:452:in `action'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:424:in `perform_action'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:352:in `create_action'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:341:in `block in transition_to'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:340:in `each'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:340:in `transition_to'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:160:in `verify'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:189:in `block in test'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/instance.rb:185:in `test'
/Users/piercemoore/.rvm/gems/ruby-2.1.5/gems/test-kitchen-1.4.1/lib/kitchen/rake_tasks.rb:54:in `block (3 levels) in define'
Tasks: TOP => kitchen:user-management-ubuntu-1204
(See full trace by running task with --trace)

[zarry]

@rex I did not hit this issue on any of the boxes I tested thus far, however I ran the ubuntu 12.04 user-management test and it did error out in the same spot for me locally. Unfortunately, not too much more to add at this point.

[rex]

@zarry Alright I'm running into another issue with a separate test. I think it's going to be a good idea to just create new issues for each failing test with the name of the test and the full stack trace output.

[rex]

Failing test: user-management-debian-720 https://github.com/chef-brigade/mongodb-cookbook/issues/47

[zarry]

@rex That sounds like a good plan to me. I started with most of the defaults so we are passing now ok, but tests are really vanilla, should be good enough for now though.

[rex]

@zarry Cool. Something to build on :)

Failing test: user-management-ubuntu-1204 #48

[zarry]

Moved to the wiki. https://github.com/chef-brigade/mongodb-cookbook/wiki/State-of-Test-Kitchen-Suite

[josephholsten]

hmm, once you get all these running, I may start building a terrible home-brewed nightly builder. Don't really want to use travis for that since it's a little easy to abuse.

[zarry]

@josephholsten Funny you should say that. I wrote and open sourced this for the company I work at. I would also be able to give it a few cycles nightly.

https://github.com/commercehub-oss/sous_chef

[rex]

@josephholsten I'm honestly not worried about using up the resources of Travis or other CI cloud providers that offer unlimited free builds to open source projects. That's what they're there for and the integrations they give us are invaluable. I am all about writing scripts or an app that keeps builds/tests running all the time, but I would prefer to leave the implementation to cloud services.

[josephholsten]

@rex that's fine for specs, but not for kitchen runs. We'd need to actually provide a place for them to provision to.

If you can figure out a way to do that which isn't attached to a credit card, that would be awesome!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.