chef-boneyard / minitest-chef-handler

Run minitest suites after your Chef recipes to check the status of your system.
Other
163 stars 44 forks source link

Assertions never seem to fail #51

Open johntdyer opened 11 years ago

johntdyer commented 11 years ago

Hey Guys,

So I just started playing with minitest, and so far the idea sounds great but I am having a little trouble...

I was able to get the tests to run, as evident in the logs below, however it doesn't seem like any of my assertions ever fail

Here is my default_test.rb

require 'minitest/spec'
require File.expand_path('../support/helpers', __FILE__)

describe_recipe "prism::default" do
    include Helpers::Prism

    describe "test" do
        ## THIS SHOULD FAIL
        it {     file("/etc/god/conf.d/apache2.god").must_exist }
    end

    describe "voxeo-smanager" do

        it "should have voxeo-smanger running" do
            service("voxeo-smanager").must_be_running
        end

        it "should have voxeo-smanger running" do
            service("voxeo-smanager").must_be_enabled
        end

        it "should respond on control port when running" do
            assert shell_out("curl -sI http://localhost:10086").exitstatus == 0
        end

    end

    describe "voxeo-as" do

        it "should respond via http when running" do
            assert shell_out("curl -sI http://localhost:8080").exitstatus == 1
        end

        it "should have voxeo-as running" do
            service("voxeo-as").must_be_enabled
        end

        it "should have voxeo-ms running" do
            service("voxeo-ms").must_be_enabled
        end
    end

    describe "voxeo-ms" do

        it "should have voxeo-as running" do
            service("voxeo-as").must_not_be_running
        end

        it "should have voxeo-ms running" do
            service("voxeo-ms").must_be_running
        end

        it "should respond on control port when running" do
            assert shell_out("curl -sI http://localhost:10099").exitstatus == 0
        end

    end

    describe "service_report" do
        it "should not have any failures" do
            service_report = shell_out "service_report"
            service_report.stdout.must_not_match(/Failures/) #this is a regex match
        end
    end

end

my helpers.rb

module Helpers
    module Prism
        require 'chef/mixin/shell_out'
        include Chef::Mixin::ShellOut
        include MiniTest::Chef::Assertions
        include MiniTest::Chef::Context
        include MiniTest::Chef::Resources

    end
end

You can see that all my tests always pass

[2013-03-23T02:49:45+00:00] DEBUG: service[nrpe] is running
nrpe            0:off   1:off   2:on    3:on    4:on    5:on    6:off
Shutting down nrpe: [  OK  ]
Starting nrpe: [  OK  ]
[2013-03-23T02:49:46+00:00] INFO: service[nrpe] restarted
[2013-03-23T02:49:46+00:00] INFO: Chef Run complete in 29.398107 seconds
[2013-03-23T02:49:46+00:00] DEBUG: Cleaning the checksum cache
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1u4g80w
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1m6uzar
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1b3rpia
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-30h6lw
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1kyuag4
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1sj4cj2
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1f3ljly
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1lm9jo2
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-13zzk83
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-9t9m09
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-5lmzex
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-avbwll
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-jk0gp6
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-7ud2xb
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1c07crs
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1f8nde1
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-r7k0ck
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-h04wcx
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-16fhew1
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-e8odfw
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1syakew
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-p6im31
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-y7wr1n
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-ylf6nj
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-r1f17p
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-1u4caxi
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-7so71u
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-wix8zj
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-m93nxi
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-4ofkgh
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-s4skzm
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-4tppwo
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-tbx863
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-8m621g
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-mayhyw
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-ecm4g4
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-15ejnul
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-s727qn
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-17f7i6d
[2013-03-23T02:49:46+00:00] DEBUG: Removing unused checksum cache file /var/chef/cache/checksums/chef-file--tmp-chef-rendered-template20130323-4041-milp0k
[2013-03-23T02:49:46+00:00] INFO: Running report handlers
Run options: -v --seed 47050

# Running tests:

recipe::prism::default::service_report#test_0001_should_not_have_any_failures = 0.00 s = true
recipe::prism::default::test#test_0001_anonymous = 0.00 s = true
recipe::prism::default::voxeo-as#test_0001_should_respond_via_http_when_running = 0.00 s = true
recipe::prism::default::voxeo-as#test_0002_should_have_voxeo_as_running = 0.00 s = true
recipe::prism::default::voxeo-as#test_0003_should_have_voxeo_ms_running = 0.00 s = true
recipe::prism::default::voxeo-ms#test_0001_should_have_voxeo_as_running = 0.00 s = true
recipe::prism::default::voxeo-ms#test_0003_should_respond_on_control_port_when_running = 0.00 s = true
recipe::prism::default::voxeo-ms#test_0002_should_have_voxeo_ms_running = 0.00 s = true
recipe::prism::default::voxeo-smanager#test_0001_should_have_voxeo_smanger_running = 0.00 s = true
recipe::prism::default::voxeo-smanager#test_0003_should_respond_on_control_port_when_running = 0.00 s = true
recipe::prism::default::voxeo-smanager#test_0002_should_have_voxeo_smanger_running = 0.01 s = true

Finished tests in 0.066084s, 166.4548 tests/s, 0.0000 assertions/s.

11 tests, 0 assertions, 0 failures, 0 errors, 0 skips
[2013-03-23T02:49:47+00:00] INFO: Report handlers complete
[2013-03-23T02:49:47+00:00] DEBUG: Exiting
[root@prism-berkshelf vagrant-chef-1]#

However some of these tests should absolutely be failing.... . Any idea what the heck I am doing wrong?

calavera commented 11 years ago

At first glance everything looks good. My only guess is that the resource helpers are not returning what they should.

Can you tell us which version of Chef and minitest-chef-handler you're using? That would help to narrow the possibilities.

calavera commented 11 years ago

The version of minitest would also help.

johntdyer commented 11 years ago

Chef 10.16.2 Minitest-hander cookbook 0.1.2

When I checked the system gems on the VM I see the following

[vagrant@prism-berkshelf cookbooks]$ gem list | grep mini minitest (3.0.1, 1.6.0) minitest-chef-handler (0.6.8)

I hope this helps!

-John

-John Dyer m. +1.407.474.c0214

e. johntdyer@gmail.com

Sent from mobile On Mar 23, 2013, at 5:40 AM, David Calavera notifications@github.com wrote:

The version of minitest would also help.

— Reply to this email directly or view it on GitHub.< https://github.com/notifications/beacon/cA0wFlwoaNexlU9ZzfIX4WTgn-MiSlCI-J4unnfxelPmrY8Kifnza7CogDDJB89e.gif

calavera commented 11 years ago

thanks @johntdyer.

For three or four versions, the file resources in Chef were completely broken due to their effort to support windows.

I know this is not what you expect but I'd recommend you to upgrade to the latest version in the series 10.18 if you want to use the resource matchers, everything should work fine then.

If you can't/don't want to upgrade your version of Chef, you can go with straight ruby and do something like this:

it { File.exist?(file_path).should be_true }

That should work regardless your version of Chef.

Please, let me know if any this works for you. Thanks again for taking your time reporting this.

johntdyer commented 11 years ago

Hey,

So I am not sure if this is in fact the issue... So I set up my test so pretty much ALL of them should fail, especially the file one

require 'minitest/spec'
require File.expand_path('../support/helpers', __FILE__)

describe_recipe "prism::default" do
    include Helpers::Prism

    describe "test" do
        ## THIS SHOULD FAIL
        it { File.exist?("/file/that/doesnt/exist").should be_true }

    end

    describe "voxeo-smanager" do

        it "should have voxeo-smanger running" do
            service("voxeo-smanager").must_be_running
        end

        it "should have voxeo-smanger running" do
            service("voxeo-smanager").must_be_enabled
        end

        it "should respond on control port when running" do
         ## THIS SHOULD FAIL
            assert shell_out("curl -sI http://localhost:10086").exitstatus.should eql 1
        end

    end

    describe "voxeo-as" do

        it "should respond via http when running" do
        ## THIS SHOULD FAIL
            assert shell_out("curl -sI http://localhost:8080").exitstatus.should eql 1
        end

        it "should have voxeo-as running" do
            service("voxeo-as").must_be_enabled
        end

        it "should have voxeo-ms running" do
            service("voxeo-ms").must_be_enabled
        end
    end

    describe "voxeo-ms" do

        it "should have voxeo-as running" do
            service("voxeo-as").must_not_be_running
        end

        it "should have voxeo-ms running" do
            service("voxeo-ms").must_be_running
        end

        it "should respond on control port when running" do
        ## THIS SHOULD FAIL
            assert shell_out("curl -sI http://localhost:10099").exitstatus.should eql 1
        end

    end

    describe "service_report" do
        it "should not have any failures" do
            service_report = shell_out "service_report"
            service_report.stdout.must_not_match(/Failures/) #this is a regex match
        end
    end

end

If you notice its saying I am making 0 assertions

2013-03-24T16:00:52+00:00] INFO: Running report handlers
Run options: -v --seed 3199

# Running tests:

recipe::prism::default::service_report#test_0001_should_not_have_any_failures =
0.00 s =
true

recipe::prism::default::test#test_0001_anonymous =
0.00 s =
true

recipe::prism::default::voxeo-as#test_0001_should_respond_via_http_when_running =
0.00 s =
true

recipe::prism::default::voxeo-as#test_0002_should_have_voxeo_as_running =
0.00 s =
true

recipe::prism::default::voxeo-as#test_0003_should_have_voxeo_ms_running =
0.00 s =
true

recipe::prism::default::voxeo-ms#test_0001_should_have_voxeo_as_running =
0.00 s =
true

recipe::prism::default::voxeo-ms#test_0002_should_have_voxeo_ms_running =
0.00 s =
true

recipe::prism::default::voxeo-ms#test_0003_should_respond_on_control_port_when_running =
0.00 s =
true

recipe::prism::default::voxeo-smanager#test_0003_should_respond_on_control_port_when_running =
0.00 s =
true

recipe::prism::default::voxeo-smanager#test_0002_should_have_voxeo_smanger_running =
0.00 s =
true

recipe::prism::default::voxeo-smanager#test_0001_should_have_voxeo_smanger_running =
0.00 s =
true

Finished tests in 0.051378s, 214.0994 tests/s, 0.0000 assertions/s.

11 tests, 0 assertions, 0 failures, 0 errors, 0 skips

[2013-03-24T16:00:52+00:00] INFO: Report handlers complete
[2013-03-24T16:00:52+00:00] DEBUG: Exiting

Any ideas?

johntdyer commented 11 years ago

@calavera Any ideas?

johntdyer commented 11 years ago

Anyone else have any ideas here ?