Tests don't run from chef-client

[johnsmyth]

I cant seem to get minitest to run tests when i run it from chef-client. I have placed all the tests in files/default/test in my cookbook. When I run through vagrant and chef-solo, the tests run as I would expect. When running from chef-client though, the tests don't run (0 tests, 0 assertions, 0 failures, 0 errors, 0 skips) .

I receive warnings like the following:

WARN: A test file was detected at map_test.rb in the files directory of the windows_base cookbook. As of version 1.2.0 of the minitest-handler-cookbook storing test files in this location is not supported. If you would like these tests executed you should move them into files/default/test/ WARN: A test file was detected at assetmanager_test.rb in the files directory of the windows_base cookbook. As of version 1.2.0 of the minitest-handler-cookbook storing test files in this location is not supported. If you would like these tests executed you should move them into files/default/test/

FYI, these are windows 2008r2 nodes, using minitest-handler cookbook 1.3.0, chef-client 11.12.8.

js

[dpetzel]

Sounds a lot like #59. FWIW, going back to an older version should work. That said I'd really like to fix this bug, but so far have been unable to reproduce it myself.

[johnsmyth]

Unfortunately the older version stopped working last friday when ci-reporter changed, which is why I tried upgradomg today...

[dpetzel]

hmm, indeed :(. What version of Chef are you running? I still can't explain why some folks are having issues, we've got the cookbook running on our Windows 2008r2 nodes and all seems well, but I have no doubt there is a bug somewhere.

Any chance you could you could edit https://github.com/btm/minitest-handler-cookbook/blob/master/libraries/test_loader.rb#L151 (on your local install) and add puts cookbook_file_names(cookbook_name)?

[johnsmyth]

returns the file names:

!!!!!!!!!!!!!!!! COOKBOOK_FILE_NAMES !!!!!!!!!!!!!!!!!!!!!!!! helpers.rb bginfo_test.rb dotf_scheduled_task_test.rb explorer_settings_test.rb installdrive_map_test.rb

!!!!!!!!!!!!!! END COOKBOOK_FILE_NAMES !!!!!!!!!!!!!!!!!!!!!!

[dpetzel]

Interesting, that is helpful and narrows down the issue to this section: https://github.com/btm/minitest-handler-cookbook/blob/master/libraries/test_loader.rb#L210

On line 231 could you insert

puts cb_files
puts Chef::Config[:cookbook_path]
puts relative_file_names

This will be super helpful in narrowing things down.

[johnsmyth]

repeats this several times:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! line 231 !!!!!!!!!!!!!!!!!!!!!! helpers.rb bginfo_test.rb dotf_scheduled_task_test.rb explorer_settings_test.rb installdrive_map_test.rb c:/chef/cache/cookbooks helpers.rb bginfo_test.rb dotf_scheduled_task_test.rb explorer_settings_test.rb installdrive_map_test.rb

[dpetzel]

Interesting. Not what I would have expected for cb_files. Can you let me know what version of Chef you are running? Also one more output. Can you add puts cb_files on line 214 so we can see the value before its manipulated by the code?

Thanks for grabbing all this info.

[johnsmyth]

chef-client is 11.12.8. I'm having some network issues, but I'll get you that output tomorrow morning.

[johnsmyth]

looks the same - filenames only, not paths:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! cbfiles_Before !!!!!!!!!!!!!!!!!!!!!! helpers.rb bginfo_test.rb dotf_scheduled_task_test.rb explorer_settings_test.rb installdrive_map_test.rb !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! end cbfiles_before !!!!!!!!!!!!!!!!!!!!!!

[dpetzel]

Interesting so it would seem that this:

      ckbk = run_context.cookbook_collection[cookbook_name]
      cb_files = ckbk.file_filenames || []

Yields different results on some combination(s). I have so far not heard of anyone having issues on anything but Windows machines, however I know at least internally here its working on Windows with 11.8.0. I try and see if I upgrade to 11.12.8 if it stops working (might take me a little bit to do that).

[dpetzel]

Just ran this on an 11.12.8-1 and 11.12.8-2 install on a Windows 2008R2 setup and didn't observe the issue. In case anyone that is following along has any ideas, here's the info on my system:

22 tests, 23 assertions, 0 failures, 0 errors, 3 skips

C:\>chef-client --version
Chef: 11.12.8

C:\>ohai | find /i "os_version"
  "os_version": "6.1.7601", 

When I print out the cb_files I see output like this:

tests/afile.rb
tests/minitest/one_test.rb
tests/minitest/default_test.rb
tests/minitest/two_test.rb
tests/minitest/support/test_helper.rb
random.inf  

Its great we've narrowed down where in the code the issue is, but I'm pretty stumped on what is different between our environments.

[johnsmyth]

I will try moving them back to the old location in tests\mintest. currently they are in files/default/test/

[dpetzel]

Its worth a shot, but I would expect that you would have gotten paths outputted with a leading test/.. We'll see what happens. Thanks for taking so much time to help debug this.

[johnsmyth]

as you would expect, the behavior is the same with the tests in files\default\tests\minitest. No tests run.

As for the time debugging, we rely on this cookbook as part of our testing workflow so I'll do whatever I can to get it working again. I appreciate YOUR responsiveness an assistance on this!

Just to get things working again for myself, I hardcoded a hack into all_test_files so that it loads my files from files/default/test:

        #files << file_name if file_name =~ /^tests?\//
        files << "test/#{file_name}" if file_name =~ /_test\.rb$/ 
        files << "test/#{file_name}" if file_name =~ /helpers?\.rb$/ 

This gets the tests to load, but they seem to run twice. The first time they run successfully:

Finished tests in 9.704056s, 2.2671 tests/s, 5.3586 assertions/s.

22 tests, 52 assertions, 0 failures, 0 errors, 0 skips
[2014-07-29T19:34:30+00:00] INFO: Report handlers complete
[2014-07-29T19:34:30+00:00] INFO: Sending resource update report (run-id: a90d4cdd-4b35-48de-a86c-fc0866efbb60)

Then they run again and i get errors relating to run_context, and nil for nilclass errors when using the node object:

# Running tests:

EEE.EEEEEEEEEEEEEEEEEE

Finished tests in 0.859457s, 25.5976 tests/s, 1.1635 assertions/s.

  1) Error:
recipe::bginfo_test.rb#test_0001_Installed BGInfo properly:
ArgumentError: Cannot find the provider for a resource with no run context set
    C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.8-x86-mingw32/lib/chef/platform/provider_mapping.rb:443:in `provider_for_resource'

  2) Error:
recipe::install_assetmanager_test.rb#test_0001_AssetManager should be installed:
ArgumentError: Cannot find the provider for a resource with no run context set
    C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.8-x86-mingw32/lib/chef/platform/provider_mapping.rb:443:in `provider_for_resource'

  3) Error:
recipe::install_assetmanager_test.rb#test_0002_BMC FootPrints Asset Core Agent processes should be runing:
ArgumentError: Cannot find the provider for a resource with no run context set
    C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.8-x86-mingw32/lib/chef/platform/provider_mapping.rb:443:in `provider_for_resource'

This may just be related to the hack that I put in, but have you seen this behavior before?

[dpetzel]

Hmm, I've not seen that before :(. The files are not duplicated on disk correct?

Grasping for straws here, what is your Chef server version. I'm wondering if maybe the Chef Server API is returning the call for cb_files in a different way

[dpetzel]

We maybe able to test that theory pretty easily. Using knife can do you

knife cookbook show windows_base <insert_version_number> files  | grep -P "name|path"

[johnsmyth]

yes - definitely only one copy, and the knife command confirms.

[dpetzel]

Sorry, I'm unclear, what did the knife command confirm (that the files don't have the path or they do). Any chance you could paste the output of that command?

[johnsmyth]

paths are there:

$ knife cookbook show windows_base 0.1.44 files | grep path | sort path: files/default/tests/minitest/bginfo_test.rb path: files/default/tests/minitest/explorer_settings_test.rb path: files/default/tests/minitest/install_assetmanager_test.rb path: files/default/tests/minitest/install_debug_diag_test.rb path: files/default/tests/minitest/install_virus_scan_test.rb path: files/default/tests/minitest/installdrive_map_test.rb

[johnsmyth]

but not in the names:

knife cookbook show zs_windows_base 0.1.44 files | grep name | sort name: bginfo_test.rb name: explorer_settings_test.rb name: install_assetmanager_test.rb name: install_debug_diag_test.rb name: install_virus_scan_test.rb name: installdrive_map_test.rb

[dpetzel]

OK, now we are getting somewhere. What version of Chef server are you running?

I suspect based on this output, I can probably crib the logic in knife to get the paths, rather than names. I'll look a little closer in a bit.

[johnsmyth]

Not sure how to determine, but it is hosted chef if that helps.

Sent from my iPhone

On Jul 29, 2014, at 5:41 PM, dpetzel notifications@github.com wrote:

OK, now we are getting somewhere. What version of Chef server are you running?

I suspect based on this output, I can probably crib the logic in knife to get the paths, rather than names. I'll look a little closer in a bit.

— Reply to this email directly or view it on GitHub.

[dpetzel]

I'm not sure either, but it is helpful. In our case were enterprise chef. So I think we've found our differences. I'll fire up my free hosted account and see if I can prove that is indeed it.

To Recap in single sport (so when I ask for help I can point someone at a concise explanation). What we are seeing is on Windows 2008R2 Nodes (maybe others) that the output of knife cookbook show <cookbook_name> <version> files is different when communicating with enterprise chef vs Hosted chef. In the enterprise Chef example the name includes part of the path, in Hosted Chef it does not.

Enterprise Chef Example:

knife cookbook show erlang latest files | grep name | sort
name:        tests/minitest/default_test.rb
name:        tests/minitest/support/helpers.rb

Hosted Chef Example:

knife cookbook show zs_windows_base 0.1.44 files | grep name | sort
name: bginfo_test.rb
name: explorer_settings_test.rb
name: install_assetmanager_test.rb
name: install_debug_diag_test.rb
name: install_virus_scan_test.rb
name: installdrive_map_test.rb

Thanks again for all your help. I think we've got enough info to make some traction on this super annoying bug.

[btm]

@dpetzel what version of chef are you using in the Hosted Chef example? What version of EC?

PS C:\Users\btm_000> knife cookbook show windows 1.30.2 files
name:        handlers/windows_reboot_handler.rb
path:        files/default/handlers/windows_reboot_handler.rb
[ snipped out the other lines]

[dpetzel]

Hey @btm I actually don't have access to the EC instance so I'll need to ask the team that manages it (unless there is a way to check it from the client).

As to the chef version, the machine I'm on right now is using Chef-DK 0.0.2:

which knife; knife --version; knife cookbook show windows 1.30.2 files | grep -P "name|path"
/opt/chefdk/bin/knife
Chef: 11.14.0.rc.2
name:        windows_reboot_handler.rb
path:        files/default/handlers/windows_reboot_handler.rb

I'm not sure what version @johnsmyth is was using though.

side note: I have a bead on a fix (independent of the output discrepancy), which I should have pushed shortly.

[dpetzel]

@johnsmyth Can you give the version on this branch a run. I'm hopeful this will take care of it.

[johnsmyth]

Good new and bad news - it did find and run the tests! But, it ran them twice - once successfully, and once unsuccessfully (without proper run context i think) as I saw before when I hacked it (see previous comment). Any ideas?

[dpetzel]

My first reaction is its something with the tests themselves, but I believe you said they worked on an earlier version so I'm not sure how much water that holds. Any chance you could pass along the actual contents of the tests?

If you manually install the proper version of CI Reporter, before hand and then downgrade to 1.1 do the tests run correctly?

Are you able to tell if its 100% of tests are running twice, or a subset?

[johnsmyth]

its definitely all the tests - output says:

Finished tests in 1.125000s, 19.5556 tests/s, 46.2222 assertions/s.

22 tests, 52 assertions, 0 failures, 0 errors, 0 skips

then runs again with:

Finished tests in 0.875000s, 25.1429 tests/s, 1.1429 assertions/s.
.... a bunch of errors ....
22 tests, 1 assertions, 0 failures, 21 errors, 0 skips

I can't say with 100% certainty that this works on a previous version from chef-client. Locally, we run all the tests with chef-solo and vagrant and didnt have any issues. I am in the process of moving our CI process over to use chef-client instead of chef-solo right now, so we never fully tested it with chef-client in this config before. Here's this simplest example test file:

require File.expand_path('../helpers.rb', __FILE__)

describe_recipe 'bginfo_test.rb' do
  include Helpers::Windows_base
  # Complete the test
  it 'Installed BGInfo properly' do
    # Add attribute for location
    assert file('c:\\bginfo\\bginfo.exe').must_exist, 'BG Info File not there'
  end
end

and the helpers.rb:

module Helpers
  module Windows_base
    include MiniTest::Chef::Assertions
    include MiniTest::Chef::Context
    include MiniTest::Chef::Resources
    require 'minitest/spec'
    require 'minitest/pride'
    require 'chef' 
    require 'minitest/autorun'
    require 'win32/registry' 
    require 'ruby-wmi'
  end
end

[dpetzel]

Its almost as if the handler is somewhere registered twice... You don't have some other fork or anything that would end up in the nodes run_list causing two copies of the handler do you?. This cookbook is listed only once in your nodes run_list correct?

Is there any change if you remove these:

    require 'chef' 
    require 'minitest/autorun'

[johnsmyth]

That seems to have fixed it !!! Thanks so much for all your help on this!!!!!

[dpetzel]

That is awesome!, I'll merge that branch and cut an official release tonight. Thanks for your data collection it was very helpful. Very glad to have finally squashed this bug.