search

github / octocatalog-diff

Compile Puppet catalogs from 2 branches, versions, etc., and compare them
https://rubygems.org/gems/octocatalog-diff/
MIT License
239 stars 85 forks source link

Modifying existing resources crashes with Puppet 7 #261

Open supertassu opened 1 year ago

supertassu commented 1 year ago

Description of problem

Command used and debugging output

taavi@runko:~/src/jquery/infrastructure-puppet production(+1/-1) ± octocatalog-diff --environment production -n codeorigin-02.ops.jquery.net --debug
D, [2022-12-01T20:44:42.844628 #845048] DEBUG -- : Running octocatalog-diff 2.1.0-1 with ruby 3.0.4
D, [2022-12-01T20:44:42.844666 #845048] DEBUG -- : Command line arguments: ["--environment", "production", "-n", "codeorigin-02.ops.jquery.net", "--debug"]
D, [2022-12-01T20:44:42.844676 #845048] DEBUG -- : Running on host runko (x86_64-linux-gnu)
D, [2022-12-01T20:44:42.844688 #845048] DEBUG -- : Compiling catalogs for codeorigin-02.ops.jquery.net
D, [2022-12-01T20:44:42.844757 #845048] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for from-catalog
D, [2022-12-01T20:44:42.844767 #845048] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for to-catalog
D, [2022-12-01T20:44:42.844784 #845048] DEBUG -- : Initialized parallel task result array: size=2
D, [2022-12-01T20:44:42.845559 #845048] DEBUG -- : Launched pid=845049 for index=0
D, [2022-12-01T20:44:42.845790 #845049] DEBUG -- : Begin build_catalog for origin/staging
D, [2022-12-01T20:44:42.845870 #845049] DEBUG -- : Setting up Puppet catalog build for origin/staging
D, [2022-12-01T20:44:42.845895 #845049] DEBUG -- : Catalog for origin/staging will be built with OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:42.845941 #845049] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:42.846011 #845049] DEBUG -- : Start retrieving facts for codeorigin-02.ops.jquery.net from OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:42.846042 #845049] DEBUG -- : Retrieving facts from PuppetDB
D, [2022-12-01T20:44:42.846220 #845048] DEBUG -- : Launched pid=845050 for index=1
D, [2022-12-01T20:44:42.846672 #845050] DEBUG -- : Begin build_catalog for .
D, [2022-12-01T20:44:42.846802 #845050] DEBUG -- : Setting up Puppet catalog build for .
D, [2022-12-01T20:44:42.846825 #845050] DEBUG -- : Catalog for . will be built with OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:42.846870 #845050] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:42.846968 #845050] DEBUG -- : Start retrieving facts for codeorigin-02.ops.jquery.net from OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:42.847014 #845050] DEBUG -- : Retrieving facts from PuppetDB
D, [2022-12-01T20:44:43.468163 #845050] DEBUG -- : Success retrieving facts for codeorigin-02.ops.jquery.net from OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:43.468565 #845050] DEBUG -- : Symlinked /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/environments/production -> /home/taavi/src/jquery/infrastructure-puppet
D, [2022-12-01T20:44:43.469546 #845050] DEBUG -- : Installed hiera.yaml from /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/environments/production/test_data/hiera.yaml to /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/hiera.yaml
D, [2022-12-01T20:44:43.473271 #845050] DEBUG -- : Installed fact file at /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var/yaml/facts/codeorigin-02.ops.jquery.net.yaml
D, [2022-12-01T20:44:43.473641 #845050] DEBUG -- : Installed CA certificate in /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var/ssl/certs/ca.pem
D, [2022-12-01T20:44:43.489164 #845049] DEBUG -- : Success retrieving facts for codeorigin-02.ops.jquery.net from OctocatalogDiff::Catalog::Computed
D, [2022-12-01T20:44:43.489303 #845049] DEBUG -- : Begin git checkout /home/taavi/src/jquery/infrastructure-puppet:origin/staging -> /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-bootstrap-checkout-20221201-845049-9mfo6l
D, [2022-12-01T20:44:43.495187 #845049] DEBUG -- : ["Exit status: 0"]
D, [2022-12-01T20:44:43.495247 #845049] DEBUG -- : Success git archive /home/taavi/src/jquery/infrastructure-puppet:origin/staging
D, [2022-12-01T20:44:43.495261 #845049] DEBUG -- : Success git checkout /home/taavi/src/jquery/infrastructure-puppet:origin/staging -> /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-bootstrap-checkout-20221201-845049-9mfo6l
D, [2022-12-01T20:44:43.495291 #845049] DEBUG -- : Begin install bootstrap script in target directory
D, [2022-12-01T20:44:43.495468 #845049] DEBUG -- : Success: copied /home/taavi/src/jquery/infrastructure-puppet/test_data/bootstrap.sh to /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-bootstrap-checkout-20221201-845049-9mfo6l/test_data/bootstrap.sh
D, [2022-12-01T20:44:43.495487 #845049] DEBUG -- : Begin bootstrap with 'test_data/bootstrap.sh' in /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-bootstrap-checkout-20221201-845049-9mfo6l
D, [2022-12-01T20:44:43.505189 #845049] DEBUG -- : Success bootstrap in /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-bootstrap-checkout-20221201-845049-9mfo6l
D, [2022-12-01T20:44:43.505482 #845049] DEBUG -- : Symlinked /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/environments/production -> /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-bootstrap-checkout-20221201-845049-9mfo6l
D, [2022-12-01T20:44:43.506695 #845049] DEBUG -- : Installed hiera.yaml from /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/environments/production/test_data/hiera.yaml to /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/hiera.yaml
D, [2022-12-01T20:44:43.510244 #845049] DEBUG -- : Installed fact file at /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var/yaml/facts/codeorigin-02.ops.jquery.net.yaml
D, [2022-12-01T20:44:43.510558 #845049] DEBUG -- : Installed CA certificate in /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var/ssl/certs/ca.pem
D, [2022-12-01T20:44:43.994814 #845050] DEBUG -- : (to) Try 1 executing Puppet 7.20.0: /usr/bin/puppet catalog compile codeorigin-02.ops.jquery.net --no-storeconfigs --factpath=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var/yaml/facts --facts_terminus=yaml --no-daemonize --color=false --environment=production --hiera_config=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/hiera.yaml --environmentpath=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/environments --vardir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var --logdir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var --ssldir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var/ssl --confdir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l
D, [2022-12-01T20:44:43.995072 #845050] DEBUG -- : ["Execute: /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-scriptrunner20221201-845050-l867uc/puppet.sh catalog compile codeorigin-02.ops.jquery.net --no-storeconfigs --factpath\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var/yaml/facts --facts_terminus\\=yaml --no-daemonize --color\\=false --environment\\=production --hiera_config\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/hiera.yaml --environmentpath\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/environments --vardir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var --logdir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var --ssldir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l/var/ssl --confdir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845050-9mfo6l"]
D, [2022-12-01T20:44:44.012554 #845049] DEBUG -- : (from) Try 1 executing Puppet 7.20.0: /usr/bin/puppet catalog compile codeorigin-02.ops.jquery.net --no-storeconfigs --factpath=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var/yaml/facts --facts_terminus=yaml --no-daemonize --color=false --environment=production --hiera_config=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/hiera.yaml --environmentpath=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/environments --vardir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var --logdir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var --ssldir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var/ssl --confdir=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc
D, [2022-12-01T20:44:44.012826 #845049] DEBUG -- : ["Execute: /tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-scriptrunner20221201-845049-o6vqzm/puppet.sh catalog compile codeorigin-02.ops.jquery.net --no-storeconfigs --factpath\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var/yaml/facts --facts_terminus\\=yaml --no-daemonize --color\\=false --environment\\=production --hiera_config\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/hiera.yaml --environmentpath\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/environments --vardir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var --logdir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var --ssldir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc/var/ssl --confdir\\=/tmp/ocd-ipc-20221201-845048-fx3dm5/ocd-builddir-20221201-845049-l867uc"]
D, [2022-12-01T20:44:45.644370 #845050] DEBUG -- : ["Exit status: 0"]
D, [2022-12-01T20:44:45.644442 #845050] DEBUG -- : (to) Catalog succeeded on try 1 in 2.170742998 seconds
D, [2022-12-01T20:44:45.648713 #845050] DEBUG -- : Catalog for . successfully built with OctocatalogDiff::Catalog::Computed in 2.801828166 seconds
D, [2022-12-01T20:44:45.648798 #845050] DEBUG -- : Success build_catalog for .
D, [2022-12-01T20:44:45.659457 #845048] DEBUG -- : PID=845050 completed in 2.813199822 seconds, 358893 bytes
D, [2022-12-01T20:44:45.693843 #845049] DEBUG -- : ["Exit status: 0"]
D, [2022-12-01T20:44:45.693912 #845049] DEBUG -- : (from) Catalog succeeded on try 1 in 2.183305034 seconds
D, [2022-12-01T20:44:45.696800 #845049] DEBUG -- : Catalog for origin/staging successfully built with OctocatalogDiff::Catalog::Computed in 2.850866101 seconds
D, [2022-12-01T20:44:45.696839 #845049] DEBUG -- : Success build_catalog for origin/staging
D, [2022-12-01T20:44:45.704691 #845048] DEBUG -- : PID=845049 completed in 2.859106061 seconds, 358796 bytes
D, [2022-12-01T20:44:45.704718 #845048] DEBUG -- : All child processes completed with no exceptions raised
I, [2022-12-01T20:44:45.704779 #845048]  INFO -- : Catalogs compiled for codeorigin-02.ops.jquery.net
D, [2022-12-01T20:44:45.704794 #845048] DEBUG -- : Begin compute diffs between catalogs
D, [2022-12-01T20:44:45.704858 #845048] DEBUG -- : Entering catdiff; catalog sizes: 204, 204
D, [2022-12-01T20:44:45.704867 #845048] DEBUG -- : Entering preprocess_diff; catalog sizes: 204, 204
D, [2022-12-01T20:44:45.706920 #845048] DEBUG -- : Exiting preprocess_diff; added 0, removed 0
D, [2022-12-01T20:44:45.706935 #845048] DEBUG -- : Entering hashdiff_initial; catalog sizes: 204, 204
D, [2022-12-01T20:44:45.706944 #845048] DEBUG -- : HashDiff configuration: (use_lcs: true)
D, [2022-12-01T20:44:45.742015 #845048] DEBUG -- : Exiting hashdiff_initial; changes: 3, nested changes: 1
/usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter/compilation_dir.rb:72:in `new': tried to create Proc object without a block (ArgumentError)
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter/compilation_dir.rb:72:in `block in traverse'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter/compilation_dir.rb:72:in `map'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter/compilation_dir.rb:72:in `traverse'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter/compilation_dir.rb:63:in `remove_compilation_dir'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter/compilation_dir.rb:40:in `filtered?'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter.rb:62:in `block in filter'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter.rb:62:in `reject!'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter.rb:62:in `filter'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter.rb:42:in `block in apply_filters'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter.rb:42:in `each'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/filter.rb:42:in `apply_filters'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/differ.rb:207:in `catdiff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-diff/differ.rb:83:in `diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/cli/diffs.rb:34:in `diffs'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/api/v1/catalog-diff.rb:54:in `catalog_diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/api/v1.rb:19:in `catalog_diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/cli.rb:152:in `run_octocatalog_diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/cli.rb:126:in `cli'
    from /usr/bin/octocatalog-diff:34:in `<main>'

Platform and version information

Do the tests pass from a clean checkout?

No,

$ bundle exec rake
rake aborted!
SyntaxError: /home/taavi/src/octocatalog-diff/vendor/bundle/ruby/3.0.0/gems/mime-types-3.2.2/lib/mime/types/logger.rb:30: _1 is reserved for numbered parameter

Anything else to add that you think will be helpful?

supertassu commented 1 year ago

This seems to be an issue with a newer Ruby version, not with a newer Puppet version. The following patch seems to fix it, but I'm not very familiar with Ruby and therefore not sure if it's the correct or proper fix:

diff --git a/lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb b/lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb
index 5e2747d..b105f8a 100644
--- a/lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb
+++ b/lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb
@@ -66,12 +66,12 @@ module OctocatalogDiff
           value
         end

-        def traverse(a)
+        def traverse(a, &proc)
           case a
           when Array
-            a.map { |v| traverse(v, &Proc.new) }
+            a.map { |v| traverse(v, &proc) }
           when Hash
-            traverse(a.values, &Proc.new)
+            traverse(a.values, &proc)
           else
             yield a
           end
mika commented 1 year ago

I can confirm the problem, also seeing it with ruby 3.1.2p20 (as present in Debian/bookworm) and puppet-agent v7.23.0-1bullseye (from puppetlabs upstream).

Thanks @supertassu for providing a patch, this also works for me!