rodjek / rspec-puppet

RSpec tests for your Puppet manifests
http://rspec-puppet.com
MIT License
364 stars 203 forks source link

Different amount of tested resources between parallel and serial tests #739

Closed Felixoid closed 5 years ago

Felixoid commented 5 years ago

Hello. We've met somehow the situation similar to #615. It counts the amount of tested resource differently with serial and parallel running. It also generates the wrong coverage report with parallel running. I've created a dedicated branch with a reproducible state https://github.com/innogames/puppet-clickhouse/tree/parallel_rspec

To reproduce:

git clone -b parallel_rspec https://github.com/innogames/puppet-clickhouse
cd puppet-clickhouse 
pdk test unit
pdk test unit --parallel

pdk test output

$ pdk test unit
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.6
[✔] Preparing to run the unit tests.
[✔] Running unit tests.
  Evaluated 46 tests in 2.531676031 seconds: 0 failures, 0 pending.

Total resources:   16
Touched resources: 16
Resource coverage: 100.00%
$ pdk test unit --parallel
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.6
[✔] Preparing to run the unit tests.
[✔] Running unit tests in parallel.
  Evaluated 46 tests in 6.10995585 seconds: 1 failures, 0 pending.

Total resources:   18
Touched resources: 17
Resource coverage: 94.44%

Untouched resources:
  Class[Clickhouse]

My Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.5.2)
      public_suffix (>= 2.0.2, < 4.0)
    ansi (1.5.0)
    ast (2.4.0)
    codecov (0.1.13)
      json
      simplecov
      url
    coderay (1.1.2)
    diff-lcs (1.3)
    docile (1.3.1)
    domain_name (0.5.20180417)
      unf (>= 0.0.5, < 1.0.0)
    facter (2.5.1)
    facterdb (0.5.2)
      facter
      jgrep
    fast_gettext (1.1.2)
    gettext (3.2.9)
      locale (>= 2.0.5)
      text (>= 1.3.0)
    gettext-setup (0.30)
      fast_gettext (~> 1.1.0)
      gettext (>= 3.0.2)
      locale
    hiera (3.4.5.13)
    hirb (0.7.3)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    jgrep (1.5.0)
    json (2.0.4)
    json-schema (2.8.0)
      addressable (>= 2.4)
    locale (2.1.2)
    mcollective-client (2.12.3)
      json
      stomp
      systemu
    metaclass (0.0.4)
    metadata-json-lint (2.2.0)
      json-schema (~> 2.8)
      spdx-licenses (~> 1.0)
    method_source (0.8.2)
    mime-types (3.2.2)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2018.0812)
    mocha (1.1.0)
      metaclass (~> 0.0.1)
    multi_json (1.13.1)
    net-scp (1.2.1)
      net-ssh (>= 2.6.5)
    net-ssh (4.2.0)
    net-telnet (0.2.0)
    netrc (0.11.0)
    parallel (1.12.1)
    parallel_tests (2.14.2)
      parallel
    parser (2.5.1.2)
      ast (~> 2.4.0)
    powerpack (0.1.2)
    pry (0.10.4)
      coderay (~> 1.1.0)
      method_source (~> 0.8.1)
      slop (~> 3.4)
    public_suffix (3.0.3)
    puppet (5.5.6)
      facter (> 2.0.1, < 4)
      fast_gettext (~> 1.1.2)
      hiera (>= 3.2.1, < 4)
      locale (~> 2.1)
      multi_json (~> 1.10)
    puppet-blacksmith (4.1.2)
      rest-client (~> 2.0)
    puppet-lint (2.3.6)
    puppet-module-posix-default-r2.4 (0.3.12)
    puppet-module-posix-dev-r2.4 (0.3.12)
      codecov (~> 0.1.10)
      gettext-setup (~> 0.26)
      metadata-json-lint (>= 2.0.2, < 3.0.0)
      mocha (>= 1.0.0, < 1.2.0)
      parallel_tests (>= 2.14.1, < 2.14.3)
      parser (~> 2.5.1.2)
      pry (~> 0.10.4)
      puppet-blacksmith (>= 3.4.0)
      puppet-lint (>= 2.3.0, < 3.0.0)
      puppet-strings (~> 2.0)
      puppet-syntax (>= 2.4.1, < 3.0.0)
      puppet_pot_generator (~> 1.0)
      puppetlabs_spec_helper (>= 2.9.0, < 3.0.0)
      rainbow (~> 2.0)
      rspec-puppet (>= 2.3.2, < 3.0.0)
      rspec-puppet-facts (~> 1.8)
      rspec_junit_formatter (~> 0.2)
      rubocop (~> 0.49.0)
      rubocop-i18n (~> 1.2.0)
      rubocop-rspec (~> 1.16.0)
      simplecov (>= 0.14.1, < 1.0.0)
      simplecov-console (~> 0.4.2)
      specinfra (= 2.67.3)
    puppet-strings (2.1.0)
      rgen
      yard (~> 0.9.5)
    puppet-syntax (2.4.1)
      rake
    puppet_pot_generator (1.0.1)
      puppet
    puppetlabs_spec_helper (2.11.0)
      mocha (~> 1.0)
      puppet-lint (~> 2.0)
      puppet-syntax (~> 2.0)
      rspec-puppet (~> 2.0)
    rainbow (2.2.2)
      rake
    rake (12.3.1)
    rest-client (2.0.2)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    rgen (0.8.2)
    rspec (3.8.0)
      rspec-core (~> 3.8.0)
      rspec-expectations (~> 3.8.0)
      rspec-mocks (~> 3.8.0)
    rspec-core (3.8.0)
      rspec-support (~> 3.8.0)
    rspec-expectations (3.8.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.8.0)
    rspec-mocks (3.8.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.8.0)
    rspec-puppet (2.7.1)
      rspec
    rspec-puppet-facts (1.9.0)
      facter
      facterdb (>= 0.5.0)
      json
      mcollective-client
      puppet
    rspec-support (3.8.0)
    rspec_junit_formatter (0.4.1)
      rspec-core (>= 2, < 4, != 2.12.0)
    rubocop (0.49.1)
      parallel (~> 1.10)
      parser (>= 2.3.3.1, < 3.0)
      powerpack (~> 0.1)
      rainbow (>= 1.99.1, < 3.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (~> 1.0, >= 1.0.1)
    rubocop-i18n (1.2.0)
      rubocop (~> 0.49.0)
    rubocop-rspec (1.16.0)
      rubocop (>= 0.49.0)
    ruby-progressbar (1.10.0)
    sfl (2.3)
    simplecov (0.16.1)
      docile (~> 1.1)
      json (>= 1.8, < 3)
      simplecov-html (~> 0.10.0)
    simplecov-console (0.4.2)
      ansi
      hirb
      simplecov
    simplecov-html (0.10.2)
    slop (3.6.0)
    spdx-licenses (1.2.0)
    specinfra (2.67.3)
      net-scp
      net-ssh (>= 2.7, < 5.0)
      net-telnet
      sfl
    stomp (1.4.4)
    systemu (2.6.5)
    text (1.3.1)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.5)
    unicode-display_width (1.4.0)
    url (0.3.2)
    xml-simple (1.1.5)
    yard (0.9.16)

PLATFORMS
  ruby

DEPENDENCIES
  fast_gettext
  json (<= 2.0.4)
  puppet
  puppet-module-posix-default-r2.4
  puppet-module-posix-dev-r2.4
  puppet-module-win-default-r2.4
  puppet-module-win-dev-r2.4
  xml-simple

BUNDLED WITH
   1.16.1

pdk version is 1.7.1