search

test-kitchen / kitchen-vagrant

A Test Kitchen Driver for Vagrant
Other
349 stars 189 forks source link

kitchen verify fails but same command in the VM shell succeeds #214

Closed miguelaferreira closed 8 years ago

miguelaferreira commented 8 years ago

I'm testing a cookbook that installs a jenkins master.

My Gemfile looks like this:

group :development do
  gem 'berkshelf',         git: 'https://github.com/berkshelf/berkshelf'
  gem 'test-kitchen',      git: 'https://github.com/test-kitchen/test-kitchen.git',    tag: 'v1.6.0'
  gem 'kitchen-vagrant',   git: 'https://github.com/test-kitchen/kitchen-vagrant.git', tag: 'v0.19.0'
  gem 'chef-vault'
end

I'm running vagrant version 1.8.1. And using the opscode centos-7.1 template.

When I ask kitchen to verify the convergence using my ServerSpec tests, it fails on some checks (I assume the ones that need sudo):

bundle exec kitchen verify --log-level debug jenkins-master
(...)
D      [SSH] vagrant@127.0.0.1<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>"2222", :compression=>true, :compression_level=>6, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/mferreira/development/git/github.schubergphilis.com/MCC/mcc_development_island/chef-repo/cookbooks/mcc_cloudstack_ci/.kitchen/kitchen-vagrant/kitchen-mcc_cloudstack_ci-jenkins-master-centos-71/.vagrant/machines/default/virtualbox/private_key"], :auth_methods=>["publickey"], :user=>"vagrant"}> (sh -c '

BUSSER_ROOT="/tmp/verifier"; export BUSSER_ROOT
GEM_HOME="/tmp/verifier/gems"; export GEM_HOME
GEM_PATH="/tmp/verifier/gems"; export GEM_PATH
GEM_CACHE="/tmp/verifier/gems/cache"; export GEM_CACHE

sudo -E /tmp/verifier/bin/busser test
')
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.4.1/lib:/tmp/verifier/gems/gems/rspec-core-3.4.3/lib /opt/chef/embedded/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec

       mcc_cloudstack_ci::jenkins_master
         should be running
         Port "8080"
           should be listening (FAILED - 1)
         Port "443"
           should be listening (FAILED - 2)
         Port "22"
           should be listening (FAILED - 3)
         Group "jenkins"
           should exist
         File "/home/jenkins"
           should be directory
         File "/home/jenkins/tmp/jobs"
           should be directory
         File "/home/jenkins/plugins"
           should be directory
         iptables
           should have rule "-p tcp -m tcp --dport 8080" with chain IN_public_allow (FAILED - 4)
         File "/home/jenkins/credentials.xml"
           content
             should match /<username>mccd-jenkins<\/username>/
           content
             should match /<description>mct_user betacloud api key<\/description>/
           content
             should match /<description>mct_user betacloud secret key<\/description>/

       Jenkins home page
         Command "curl http://localhost:8080 -L"
           stdout
             should match /<title>Dashboard \[Jenkins\]<\/title>/

       Failures:

         1) mcc_cloudstack_ci::jenkins_master Port "8080" should be listening
            Failure/Error: it { should be_listening }
              expected Port "8080" to be listening
              /bin/sh -c ss\ -tunl\ \|\ grep\ --\ :8080\\\ 

            # /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:5:in `block (3 levels) in <top (required)>'

         2) mcc_cloudstack_ci::jenkins_master Port "443" should be listening
            Failure/Error: it { should be_listening }
              expected Port "443" to be listening
              /bin/sh -c ss\ -tunl\ \|\ grep\ --\ :443\\\ 

            # /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:9:in `block (3 levels) in <top (required)>'

         3) mcc_cloudstack_ci::jenkins_master Port "22" should be listening
            Failure/Error: it { should be_listening }
              expected Port "22" to be listening
              /bin/sh -c ss\ -tunl\ \|\ grep\ --\ :22\\\ 

            # /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:13:in `block (3 levels) in <top (required)>'

         4) mcc_cloudstack_ci::jenkins_master iptables should have rule "-p tcp -m tcp --dport 8080" with chain IN_public_allow
            Failure/Error: it { should have_rule('-p tcp -m tcp --dport 8080').with_chain('IN_public_allow') }
              expected iptables to have rule "-p tcp -m tcp --dport 8080" with chain IN_public_allow
              /bin/sh -c iptables\ -S\ IN_public_allow\ \|\ grep\ --\ -p\\\ tcp\\\ -m\\\ tcp\\\ --dport\\\ 8080

            # /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:35:in `block (3 levels) in <top (required)>'

       Finished in 0.31895 seconds (files took 0.30821 seconds to load)
       13 examples, 4 failures

       Failed examples:

       rspec /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:5 # mcc_cloudstack_ci::jenkins_master Port "8080" should be listening
       rspec /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:9 # mcc_cloudstack_ci::jenkins_master Port "443" should be listening
       rspec /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:13 # mcc_cloudstack_ci::jenkins_master Port "22" should be listening
       rspec /tmp/verifier/suites/serverspec/jenkins_master_spec.rb:35 # mcc_cloudstack_ci::jenkins_master iptables should have rule "-p tcp -m tcp --dport 8080" with chain IN_public_allow

       /opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.4.1/lib:/tmp/verifier/gems/gems/rspec-core-3.4.3/lib /opt/chef/embedded/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec failed
       !!!!!! Ruby Script [/tmp/verifier/gems/gems/busser-serverspec-0.5.9/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/verifier/suites/serverspec] exit code was 1
(...)

If I login to the VM and run the same command, all tests pass:

bundle exec kitchen login jenkins-master
[vagrant@jenkins-master-centos-71 ~]$ sh -c '
> 
> BUSSER_ROOT="/tmp/verifier"; export BUSSER_ROOT
> GEM_HOME="/tmp/verifier/gems"; export GEM_HOME
> GEM_PATH="/tmp/verifier/gems"; export GEM_PATH
> GEM_CACHE="/tmp/verifier/gems/cache"; export GEM_CACHE
> 
> sudo -E /tmp/verifier/bin/busser test
> '
-----> Running serverspec test suite
/opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.4.1/lib:/tmp/verifier/gems/gems/rspec-core-3.4.3/lib /opt/chef/embedded/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec

mcc_cloudstack_ci::jenkins_master
  should be running
  Port "8080"
    should be listening
  Port "443"
    should be listening
  Port "22"
    should be listening
  Group "jenkins"
    should exist
  File "/home/jenkins"
    should be directory
  File "/home/jenkins/tmp/jobs"
    should be directory
  File "/home/jenkins/plugins"
    should be directory
  iptables
    should have rule "-p tcp -m tcp --dport 8080" with chain IN_public_allow
  File "/home/jenkins/credentials.xml"
    content
      should match /<username>mccd-jenkins<\/username>/
    content
      should match /<description>mct_user betacloud api key<\/description>/
    content
      should match /<description>mct_user betacloud secret key<\/description>/

Jenkins home page
  Command "curl http://localhost:8080 -L"
    stdout
      should match /<title>Dashboard \[Jenkins\]<\/title>/

Finished in 0.40484 seconds (files took 0.32557 seconds to load)
13 examples, 0 failures
gawbul commented 8 years ago

I'm having this issue too! kitchen verify gives the following failures:

         Port "443"
           should be listening with tcp (FAILED - 1)
         Port "8000"
           should be listening with tcp (FAILED - 2)
         Port "8089"
           should be listening with tcp (FAILED - 3)

       Failures:
         1) splunk::web Port "443" should be listening with tcp
            Failure/Error: it { should be_listening.with('tcp') }
              expected Port "443" to be listening with tcp
              /bin/sh -c ss\ -tnl4\ \|\ grep\ --\ :443\\\

            # /tmp/verifier/suites/serverspec/web_spec.rb:27:in `block (3 levels) in <top (required)>'

         2) splunk::web Port "8000" should be listening with tcp
            Failure/Error: it { should be_listening.with('tcp') }
              expected Port "8000" to be listening with tcp
              /bin/sh -c ss\ -tnl4\ \|\ grep\ --\ :8000\\\

            # /tmp/verifier/suites/serverspec/web_spec.rb:31:in `block (3 levels) in <top (required)>'

         3) splunk::web Port "8089" should be listening with tcp
            Failure/Error: it { should be_listening.with('tcp') }
              expected Port "8089" to be listening with tcp
              /bin/sh -c ss\ -tnl4\ \|\ grep\ --\ :8089\\\

            # /tmp/verifier/suites/serverspec/web_spec.rb:35:in `block (3 levels) in <top (required)>'

But when I do a kitchen login and run ss -tnl4 | grep -- :443 it matches fine:

LISTEN     0      511          *:443                      *:*

But running /bin/sh -c ss -tnl4 | grep -- :443 returns nothing and echo $? returns 1?

gawbul commented 8 years ago

However, this works (note the trailing space at the end of the command):

[kitchen@560fa9e639f3 ~]$ /bin/sh -c ss\ -tnl4\ \|\ grep\ --\ :443\\\

LISTEN     0      128          *:443                     *:*

I wonder if a change somewhere has caused trailing spaces removed?

gawbul commented 8 years ago

I think this is related to https://github.com/test-kitchen/busser-serverspec/issues/32?

cheeseplus commented 8 years ago

I'll try to repro tomorrow but this I have doubts that it's an issue with kitchen-vagrant itself. What serverspec versions are you seeing this on?

miguelaferreira commented 8 years ago

@cheeseplus I wasn't pinning the version of serverspec (nor busser), but I did that tot he following versions

(...)
  gem 'serverspec', '2.31.1'
  gem 'busser',     '0.7.1'
(...)

The result was the same.

bjenkins001 commented 8 years ago

@miguelaferreira I have tried those versions and received the same error.

bjenkins001 commented 8 years ago

I added:

set :path, '/sbin:/usr/sbin:$PATH'

after 

set :backend, :exec

now it works for me

cheeseplus commented 8 years ago

That makes a ton of sense and that is a good catch

miguelaferreira commented 8 years ago

Sorry for the delay, I can also confirm that the suggestion of @bjenkins101 does indeed fix the problem.