search

rodjek / rspec-puppet

RSpec tests for your Puppet manifests
http://rspec-puppet.com
MIT License
363 stars 203 forks source link

Untouched resources that are "touched" #615

Closed cdenneen closed 6 years ago

cdenneen commented 6 years ago

Here is the output of my spec run which shows the resources are touched but something is making it report as not in the coverage output:

puppet/profile_base - [2.x●] » be rake spec_standalone
puppet/profile_base - [2.x●] » be rake spec_standalone
Could not find semantic_puppet gem, falling back to internal functionality. Version checks may be less robust.
/Users/cdenneen/.rbenv/versions/2.1.9/bin/ruby -I/Users/cdenneen/src/gitlab/puppet/profile_base/.bundle/gems/rspec-core-3.7.0/lib:/Users/cdenneen/src/gitlab/puppet/profile_base/.bundle/gems/rspec-support-3.7.0/lib /Users/cdenneen/src/gitlab/puppet/profile_base/.bundle/gems/rspec-core-3.7.0/exe/rspec --pattern spec/\{aliases,classes,defines,unit,functions,hosts,integration,type_aliases,types\}/\*\*/\*_spec.rb --color

profile_base
  supported operating systems
    on centos-7-x86_64
      profile_base with default parameters
        should contain Class[profile_base]
        should contain Class[profile_base::sensu]
        should contain Class[profile_base::filebeat]
        should compile into a catalogue without dependency cycles
        should contain Class[profile_base::el]
      profile_base with sensu disabled
        should not contain Class[profile_base::sensu]
      profile_base with filebeat disabled
        should not contain Class[profile_base::filebeat]
      profile_base on vmware
        should contain Class[profile_base::linux::vmware]
      profile_base not on vmware
        should not contain Class[profile_base::linux::vmware]

profile_base::el
  supported operating systems
    on centos-7-x86_64
      profile_base::el class without any parameters
        should contain Class[profile_base::el]
        should contain Class[profile_base::el::yum]
        should contain Class[profile_base::linux]
        should contain Class[profile_base::unix]
        should compile into a catalogue without dependency cycles
        should contain Stage[last]
        should contain Class[epel]
        should contain Package[yum] with ensure => "latest"
        should contain Exec[yum clean]
        should contain Package[policycoreutils-python]

profile_base::el::yum
  supported operating systems
    on centos-7-x86_64
      profile_base::el::yum class without any parameters
        should contain Class[profile_base::el::yum]
        should compile into a catalogue without dependency cycles
        should contain Package[yum] with ensure => "latest"
        should contain Exec[yum clean]

profile_base::filebeat
  supported operating systems
    on centos-7-x86_64
      profile_base::filebeat class
        should contain Class[profile_base::filebeat]

profile_base::linux::ca
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::ca class
        should contain Class[profile_base::linux::ca]
        should contain Class[ca_cert]
        should contain Package[ca-certificates]
        should contain Ca_cert::Ca[AP Root Certificate Chain] with ensure => "absent"
        should contain Ca_cert::Ca[ctcgssadcs01_sha1]
        should contain Ca_cert::Ca[ctcgssadcs02_sha1]
        should contain Ca_cert::Ca[ctcgssadcs03_sha2]

profile_base::linux::facts
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::facts class
        should contain Class[profile_base::linux::facts]

profile_base::linux::firewall
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::firewall class
        should contain Class[profile_base::linux::firewall]

profile_base::linux::groups
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::groups class
        should contain Class[profile_base::linux::groups]

profile_base::linux::postfix
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::postfix class
        should contain Class[profile_base::linux::postfix]

profile_base::linux::snmp
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::snmp class
        should contain Class[profile_base::linux::snmp]

profile_base::linux
  supported operating systems
    on centos-7-x86_64
      profile_base::linux class
        should contain Class[profile_base::linux]
        should contain Class[profile_base::linux::timezone]
        should contain Class[profile_base::linux::firewall]
        should contain Class[profile_base::linux::ca]
        should contain Class[profile_base::linux::sudo]
        should contain Class[profile_base::linux::snmp]
        should contain Class[profile_base::linux::facts]
        should contain Class[profile_base::linux::postfix]
        should contain Class[profile_base::linux::users]
        should contain Class[profile_base::linux::groups]
        should contain Class[sudo]
        should contain Class[snmp]
        should contain Accounts::User[orion] with ensure => "present", comment => "Orion User" and uid => "7001"
        should contain Accounts::User[chris] with ensure => "absent"
        should contain Accounts::User[rdeangel] with ensure => "absent"
        should contain Accounts::User[rshetty] with ensure => "absent"
        should contain Accounts::User[jhayes] with ensure => "absent"
        should contain Accounts::User[dkipperm] with ensure => "absent"
        should contain Accounts::User[jventimi] with ensure => "absent"
        should contain Group[admin]
        should contain Group[developer]
        should contain Group[sudo]
        should contain Group[sudonopw]
        should contain Package[sudo]
        should contain Sudo::Conf[admins]
        should contain Sudo::Conf[shells]
        should contain Sudo::Conf[su]
        should contain Sudo::Conf[sudo]
        should contain Sudo::Conf[sudonopw]
        should contain Sudo::Conf[sensu]
        should contain Firewall[000 accept all icmp]
        should contain Firewall[001 accept all to lo interface]
        should contain Firewall[002 accept related established rules]
        should contain Firewall[003 allow ssh access]
        should contain Firewall[300 SNMP]
        should contain Firewall[998 log packet drops]
        should contain Firewall[999 drop all]
        should contain Firewallchain[LOGGING:filter:IPv4]
        should contain Firewallchain[DOCKER:filter:IPv4]
        should contain Firewallchain[DOCKER-ISOLATION:filter:IPv4]
        should contain Firewallchain[DOCKER:nat:IPv4]
        should contain Firewallchain[FORWARD:filter:IPv4]
        should contain Firewallchain[INPUT:filter:IPv4]
        should contain Firewallchain[POSTROUTING:nat:IPv4]
        should contain Firewallchain[PREROUTING:nat:IPv4]
        should contain File[/etc/facter] with ensure => "directory", owner => "root", group => "root" and mode => "0644"
        should contain File[/etc/facter/facts.d] with ensure => "directory", owner => "root", group => "root" and mode => "0644"
        should contain File[/etc/facter/facts.d/custom_facts.txt] with owner => "root", group => "root", mode => "0644" and content =~ /myfact=testrole/
        should compile into a catalogue without dependency cycles
        profile_base::linux::timezone
          should contain Class[timezone]
        profile_base::linux::firewall
          should contain Class[firewall]
          should contain Class[profile_base::linux::firewall::pre]
          should contain Class[profile_base::linux::firewall::post]

profile_base::linux::sudo
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::sudo class
        should contain Class[sudo]
        should contain Sudo::Conf[centos]

profile_base::linux::timezone
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::timezone class
        should contain Class[profile_base::linux::timezone]

profile_base::linux::users
  supported operating systems
    on centos-7-x86_64
      profile_base::linux::users class
        should contain Class[profile_base::linux::users]

profile_base::linux::vmware
  supported operating system
    on centos-7-x86_64
      profile_base::linux::vmware class without any parameters
        should contain Class[profile_base::linux::vmware]
        should contain Class[openvmtools]
        should not contain Class[vmwaretools]
        should compile into a catalogue without dependency cycles

profile_base::sensu
  supported operating systems
    on centos-7-x86_64
      profile_base::sensu class
        should contain Class[profile_base::sensu]
        should contain Class[profile_base::sensu]
        should contain Package[sensu-plugins-aws] with provider => "sensu_gem"
        should contain Package[sensu-plugins-process-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-graphite] with provider => "sensu_gem"
        should contain Package[sensu-plugins-memory-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-cpu-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-filesystem-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-hardware] with provider => "sensu_gem"
        should contain Package[sensu-plugins-uptime-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-load-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-logs] with provider => "sensu_gem"
        should contain Package[sensu-plugins-mailer] with provider => "sensu_gem"
        should contain Package[sensu-plugins-disk-checks] with provider => "sensu_gem"
        should contain Package[sensu-plugins-http] with provider => "sensu_gem"
        should contain Package[sensu-plugins-vmstats] with provider => "sensu_gem"
        should contain Package[sensu-plugins-puppet] with provider => "sensu_gem"
        should contain Package[vmstat]
        should contain File[/etc/sensu/plugins/echo.sh] with ensure => "present", owner => "sensu", group => "sensu" and mode => "0555"
        should contain Package[nagios-plugins]
        should contain Package[nagios-common]
        should contain Yum::Group[development group install for sensu] with ensure => "installed"
        should contain Class[epel]
        checking multi-level hierarchy merge
          should contain Sensu_client_config[foo.example.com] with custom =~ /\"Team\"=>\"Level2\"/ and custom =~ /\"Service\"=>\"common\"/

profile_base::unix
  supported operating systems
    on centos-7-x86_64
      profile_base::unix with default parameters
        should contain Class[profile_base::unix]
        should compile into a catalogue without dependency cycles
        should contain Class[ntp]
        should contain Class[profile_base::unix::ssh]
        should contain File[/etc/motd] with content =~ /AP Network Location/
        should contain Ssh_authorized_key[root@fortress] with user => "root" and type => "ssh-rsa"
        should contain Resources[sshkey] with purge => "true"

profile_base::unix::ssh
  supported operating systems
    on centos-7-x86_64
      profile_base::unix::ssh with default parameters
        should contain Class[profile_base::unix::ssh]
        should compile into a catalogue without dependency cycles
        should contain Ssh_authorized_key[root@fortress] with user => "root" and type => "ssh-rsa"
        should contain Resources[sshkey] with purge => "true"

Total resources:   85
Touched resources: 79
Resource coverage: 92.94%
Untouched resources:

  Class[Profile_base::Linux::Ca]
  Class[Profile_base::Linux::Facts]
  Class[Profile_base::Linux::Firewall]
  Class[Profile_base::Linux::Groups]
  Class[Profile_base::Linux::Postfix]
  Class[Profile_base::Linux::Snmp]
Must be at least 95% of code coverage (FAILED - 1)

Failures:

  1) Code coverage. Must be at least 95% of code coverage
     Failure/Error: RSpec::Puppet::Coverage.report!(95)

       expected: >= 95.0
            got:    92.94
     # ./.bundle/gems/rspec-puppet-2.6.9/lib/rspec-puppet/coverage.rb:104:in `block in coverage_test'
     # ./.bundle/gems/rspec-puppet-2.6.9/lib/rspec-puppet/coverage.rb:106:in `coverage_test'
     # ./.bundle/gems/rspec-puppet-2.6.9/lib/rspec-puppet/coverage.rb:95:in `report!'
     # ./spec/spec_helper.rb:16:in `block (2 levels) in <top (required)>'

Finished in 14.42 seconds (files took 8.94 seconds to load)
132 examples, 1 failure

Failed examples:

rspec  # Code coverage. Must be at least 95% of code coverage

puppet/profile_base - [2.x●] »
cdenneen commented 6 years ago

@rodjek any ideas how I can work around this or where the problem is and I’ll attempt a PR if I can

rodjek commented 6 years ago

@cdenneen Could you create a simple test repo that reproduces the problem that I could look at?

lopaka commented 6 years ago

Hello @rodjek and @cdenneen, I'm hoping my issue is in line with this one. We get this same error of Untouched resources, but our spec tests does indeed touch it. After playing around a bit, we realized it has something to do with the ordering of the spec test files. Here's a simple test repo that reproduces the issue: https://github.com/lopaka/puppet_tests/tree/master/system_setup

And the passing branch with the reordering of the single spec file brought to the top alphanumerically: https://github.com/lopaka/puppet_tests/tree/passes/system_setup

Quick diff of the branches: https://github.com/lopaka/puppet_tests/compare/passes

The spec command and output on failing branch (master):

puppet_tests/system_setup$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
puppet_tests/system_setup$ bundle exec rake spec
.......
Total resources:   5
Touched resources: 4
Resource coverage: 80.00%
Untouched resources:

  Class[System_setup]
F

Failures:

  1) Code coverage. Must be at least 100% of code coverage
     Failure/Error: RSpec::Puppet::Coverage.report!(100)

       expected: >= 100.0
            got:    80.0
     # /Users/ldelp/.rvm/gems/ruby-2.3.4/gems/rspec-puppet-2.6.9/lib/rspec-puppet/coverage.rb:104:in `block in coverage_test'
     # /Users/ldelp/.rvm/gems/ruby-2.3.4/gems/rspec-puppet-2.6.9/lib/rspec-puppet/coverage.rb:106:in `coverage_test'
     # /Users/ldelp/.rvm/gems/ruby-2.3.4/gems/rspec-puppet-2.6.9/lib/rspec-puppet/coverage.rb:95:in `report!'
     # ./spec/spec_helper.rb:23:in `block (2 levels) in <top (required)>'

Finished in 2.77 seconds (files took 1.79 seconds to load)
7 examples, 1 failure

Failed examples:

rspec  # Code coverage. Must be at least 100% of code coverage

And, spec command and output on passing branch (passes):

puppet_tests/system_setup$ git checkout passes
Switched to branch 'passes'
puppet_tests/system_setup$ bundle exec rake spec
.......
Total resources:   4
Touched resources: 4
Resource coverage: 100.00%

Finished in 2.55 seconds (files took 1.5 seconds to load)
7 examples, 0 failures

This same behavior, including the fix/work around also happens when using include and contain (contain was just a quick test, made no branches with it): using_include branch using_include_passes branch

Perhaps there may be a way to read all spec tests for parent classes first regardless of order of the spec files?

Any questions, please let me know.

Also, please let me know if my issue does not pertain to this, and I will open a new one.

Thanks!

cdenneen commented 6 years ago

Thanks for creating sample repo. Yes I think it’s a ordering issue too. Basically I think some spec tests touch certain resources and others don’t but based on how that order works it says it’s Untouched even though there is spec test for it (probably comes later/after it reports error)

rodjek commented 6 years ago

@cdenneen @lopaka I've pushed up a PR with a fix for this (#652), can you give it a try on your problem repo(s)?

lopaka commented 6 years ago

@rodjek The fix works! Here are the diffs showing the changes using your branch: using inherits using include

Thanks!

Felixoid commented 5 years ago

Hello @rodjek It looks like I've met the same issue in https://github.com/innogames/puppet-clickhouse/tree/parallel_rspec somehow. The coverage is good when it runs serial, but fails on parallel.

┌[17:23:03] felixoid@ig-fel:/tmp/  
└>ѳ> git clone -b parallel_rspec https://github.com/innogames/puppet-clickhouse
Cloning into 'puppet-clickhouse'...
remote: Enumerating objects: 134, done.
remote: Counting objects: 100% (134/134), done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 134 (delta 60), reused 116 (delta 43), pack-reused 0
Receiving objects: 100% (134/134), 27.86 KiB | 4.64 MiB/s, done.
Resolving deltas: 100% (60/60), done.
┌[17:23:25] felixoid@ig-fel:/tmp/  
└>ѳ> cd puppet-clickhouse 
┌[17:23:28] felixoid@ig-fel:/tmp/puppet-clickhouse/ (parallel_rspec|✔) 
└>±> pdk test unit
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.6
[✔] Preparing to run the unit tests.
[✔] Running unit tests.
  Evaluated 46 tests in 2.502751486 seconds: 0 failures, 0 pending.

Total resources:   16
Touched resources: 16
Resource coverage: 100.00%
┌[17:24:24] felixoid@ig-fel:/tmp/puppet-clickhouse/ (parallel_rspec|✔) 
└>±> pdk test unit --parallel
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.6
[✔] Preparing to run the unit tests.
[✔] Running unit tests in parallel.
  Evaluated 46 tests in 6.017412 seconds: 1 failures, 0 pending.

Total resources:   18
Touched resources: 17
Resource coverage: 94.44%

Untouched resources:
  Class[Clickhouse]
┌[17:24:43] felixoid@ig-fel:/tmp/puppet-clickhouse/ (parallel_rspec|✔) 
└>±> pdk --version
1.7.1

Does it look related or should I create a new issue? Here is Gemfile.lock

  remote: https://rubygems.org/
  specs:
    addressable (2.5.2)
      public_suffix (>= 2.0.2, < 4.0)
    ansi (1.5.0)
    ast (2.4.0)
    codecov (0.1.13)
      json
      simplecov
      url
    coderay (1.1.2)
    diff-lcs (1.3)
    docile (1.3.1)
    domain_name (0.5.20180417)
      unf (>= 0.0.5, < 1.0.0)
    facter (2.5.1)
    facterdb (0.5.2)
      facter
      jgrep
    fast_gettext (1.1.2)
    gettext (3.2.9)
      locale (>= 2.0.5)
      text (>= 1.3.0)
    gettext-setup (0.30)
      fast_gettext (~> 1.1.0)
      gettext (>= 3.0.2)
      locale
    hiera (3.4.5.13)
    hirb (0.7.3)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    jgrep (1.5.0)
    json (2.0.4)
    json-schema (2.8.0)
      addressable (>= 2.4)
    locale (2.1.2)
    mcollective-client (2.12.3)
      json
      stomp
      systemu
    metaclass (0.0.4)
    metadata-json-lint (2.2.0)
      json-schema (~> 2.8)
      spdx-licenses (~> 1.0)
    method_source (0.8.2)
    mime-types (3.2.2)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2018.0812)
    mocha (1.1.0)
      metaclass (~> 0.0.1)
    multi_json (1.13.1)
    net-scp (1.2.1)
      net-ssh (>= 2.6.5)
    net-ssh (4.2.0)
    net-telnet (0.2.0)
    netrc (0.11.0)
    parallel (1.12.1)
    parallel_tests (2.14.2)
      parallel
    parser (2.5.1.2)
      ast (~> 2.4.0)
    powerpack (0.1.2)
    pry (0.10.4)
      coderay (~> 1.1.0)
      method_source (~> 0.8.1)
      slop (~> 3.4)
    public_suffix (3.0.3)
    puppet (5.5.6)
      facter (> 2.0.1, < 4)
      fast_gettext (~> 1.1.2)
      hiera (>= 3.2.1, < 4)
      locale (~> 2.1)
      multi_json (~> 1.10)
    puppet-blacksmith (4.1.2)
      rest-client (~> 2.0)
    puppet-lint (2.3.6)
    puppet-module-posix-default-r2.4 (0.3.12)
    puppet-module-posix-dev-r2.4 (0.3.12)
      codecov (~> 0.1.10)
      gettext-setup (~> 0.26)
      metadata-json-lint (>= 2.0.2, < 3.0.0)
      mocha (>= 1.0.0, < 1.2.0)
      parallel_tests (>= 2.14.1, < 2.14.3)
      parser (~> 2.5.1.2)
      pry (~> 0.10.4)
      puppet-blacksmith (>= 3.4.0)
      puppet-lint (>= 2.3.0, < 3.0.0)
      puppet-strings (~> 2.0)
      puppet-syntax (>= 2.4.1, < 3.0.0)
      puppet_pot_generator (~> 1.0)
      puppetlabs_spec_helper (>= 2.9.0, < 3.0.0)
      rainbow (~> 2.0)
      rspec-puppet (>= 2.3.2, < 3.0.0)
      rspec-puppet-facts (~> 1.8)
      rspec_junit_formatter (~> 0.2)
      rubocop (~> 0.49.0)
      rubocop-i18n (~> 1.2.0)
      rubocop-rspec (~> 1.16.0)
      simplecov (>= 0.14.1, < 1.0.0)
      simplecov-console (~> 0.4.2)
      specinfra (= 2.67.3)
    puppet-strings (2.1.0)
      rgen
      yard (~> 0.9.5)
    puppet-syntax (2.4.1)
      rake
    puppet_pot_generator (1.0.1)
      puppet
    puppetlabs_spec_helper (2.11.0)
      mocha (~> 1.0)
      puppet-lint (~> 2.0)
      puppet-syntax (~> 2.0)
      rspec-puppet (~> 2.0)
    rainbow (2.2.2)
      rake
    rake (12.3.1)
    rest-client (2.0.2)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    rgen (0.8.2)
    rspec (3.8.0)
      rspec-core (~> 3.8.0)
      rspec-expectations (~> 3.8.0)
      rspec-mocks (~> 3.8.0)
    rspec-core (3.8.0)
      rspec-support (~> 3.8.0)
    rspec-expectations (3.8.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.8.0)
    rspec-mocks (3.8.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.8.0)
    rspec-puppet (2.7.1)
      rspec
    rspec-puppet-facts (1.9.0)
      facter
      facterdb (>= 0.5.0)
      json
      mcollective-client
      puppet
    rspec-support (3.8.0)
    rspec_junit_formatter (0.4.1)
      rspec-core (>= 2, < 4, != 2.12.0)
    rubocop (0.49.1)
      parallel (~> 1.10)
      parser (>= 2.3.3.1, < 3.0)
      powerpack (~> 0.1)
      rainbow (>= 1.99.1, < 3.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (~> 1.0, >= 1.0.1)
    rubocop-i18n (1.2.0)
      rubocop (~> 0.49.0)
    rubocop-rspec (1.16.0)
      rubocop (>= 0.49.0)
    ruby-progressbar (1.10.0)
    sfl (2.3)
    simplecov (0.16.1)
      docile (~> 1.1)
      json (>= 1.8, < 3)
      simplecov-html (~> 0.10.0)
    simplecov-console (0.4.2)
      ansi
      hirb
      simplecov
    simplecov-html (0.10.2)
    slop (3.6.0)
    spdx-licenses (1.2.0)
    specinfra (2.67.3)
      net-scp
      net-ssh (>= 2.7, < 5.0)
      net-telnet
      sfl
    stomp (1.4.4)
    systemu (2.6.5)
    text (1.3.1)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.5)
    unicode-display_width (1.4.0)
    url (0.3.2)
    xml-simple (1.1.5)
    yard (0.9.16)

PLATFORMS
  ruby

DEPENDENCIES
  fast_gettext
  json (<= 2.0.4)
  puppet
  puppet-module-posix-default-r2.4
  puppet-module-posix-dev-r2.4
  puppet-module-win-default-r2.4
  puppet-module-win-dev-r2.4
  xml-simple

BUNDLED WITH
   1.16.1
rodjek commented 5 years ago

@Felixoid It looks like a different issue so open a new issue please :)