search

github / octocatalog-diff

Compile Puppet catalogs from 2 branches, versions, etc., and compare them
https://rubygems.org/gems/octocatalog-diff/
MIT License
239 stars 85 forks source link

Unable to determine Puppet version #8

Closed rnelson0 closed 7 years ago

rnelson0 commented 7 years ago

Description of problem

  • What did you do? After following the configuration instructions, and the settings[:puppetdb_ssl_client_pem] tweak in #6, I receive the error Unable to determine Puppet version
  • What happened? I receive the error Unable to determine Puppet version
  • What did you expect to happen? I expected octocatalog-diff to probably hang up on the bootstrap script (similar setup as described in #4)
  • How can someone reproduce the problem? My setup involves a private GitHub repo and nodes on my home LAN so I am unsure how much it can be replicated by others. The layout is similar to that of https://github.com/puppetinabox/controlrepo, and these steps were followed afterward:
# add octocatalog-diff to the Gemfile
bundle install --path vendor --without system_tests
curl https://raw.githubusercontent.com/github/octocatalog-diff/master/examples/octocatalog-diff.cfg.rb .octocatalog-diff.cfg.rb
# configure, see output below
bundle exec octocatalog-diff -d -n build03.nelson.va -o /tmp/catalog.json --bootstrapped-to-dir=$PWD --catalog-only

Command used and debugging output

$ bundle exec octocatalog-diff --config-test
Looking for config in: /home/rnelson0/puppet/controlrepo/.octocatalog-diff.cfg.rb
Loading config: /home/rnelson0/puppet/controlrepo/.octocatalog-diff.cfg.rb
:hiera_yaml_file => (String) "hiera.yaml"
:hiera_path_strip => (String) "/etc/puppetlabs/code/environments/"
:puppetdb_url => (String) "https://puppet.nelson.va:8081"
:puppetdb_ssl_ca => (String) "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
:puppetdb_ssl_client_pem => (String) "TWO REALLY LONG CERTIFICATE BLOCKS CONCATENATED TOGETHER"
:storeconfigs => (TrueClass) true
:puppet_binary => (String) "/opt/puppetlabs/puppet/bin/puppet"
:from_env => (String) "origin/production"
:header => (Symbol) :default
:cached_master_dir => (String) "/home/rnelson0/.octocatalog-diff-cache"
:safe_to_delete_cached_master_dir => (String) "/home/rnelson0/.octocatalog-diff-cache"
:basedir => (String) "/home/rnelson0/puppet/controlrepo"

$ bundle exec octocatalog-diff -d -n build03.nelson.va -o /tmp/catalog.json --bootstrapped-to-dir=$PWD --catalog-only
D, [2016-10-30T23:58:49.943424 #18490] DEBUG -- : Running octocatalog-diff 0.5.1 with ruby 2.0.0
D, [2016-10-30T23:58:49.943526 #18490] DEBUG -- : Command line arguments: ["-d", "-n", "build03.nelson.va", "-o", "/tmp/catalog.json", "--bootstrapped-to-dir=/home/rnelson0/puppet/controlrepo", "--catalog-only"]
D, [2016-10-30T23:58:49.943556 #18490] DEBUG -- : Running on host build03 (x86_64-linux)
D, [2016-10-30T23:58:49.943577 #18490] DEBUG -- : Compiling catalog --catalog-only for build03.nelson.va
D, [2016-10-30T23:58:49.943960 #18490] DEBUG -- : Initialized OctocatalogDiff::Catalog::Noop for from-catalog
D, [2016-10-30T23:58:49.944006 #18490] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for to-catalog
D, [2016-10-30T23:58:49.944054 #18490] DEBUG -- : Initialized parallel task result array: size=2
D, [2016-10-30T23:58:49.946238 #18492] DEBUG -- : Begin build_catalog for origin/production
D, [2016-10-30T23:58:49.946525 #18492] DEBUG -- : Setting up Puppet catalog build for origin/production
D, [2016-10-30T23:58:49.946616 #18492] DEBUG -- : Catalog for origin/production will be built with OctocatalogDiff::Catalog::Noop
D, [2016-10-30T23:58:49.946919 #18492] DEBUG -- : Catalog for origin/production successfully built with OctocatalogDiff::Catalog::Noop in 7.57e-06 seconds
D, [2016-10-30T23:58:49.946976 #18492] DEBUG -- : Success build_catalog for origin/production
D, [2016-10-30T23:58:49.947677 #18495] DEBUG -- : Begin build_catalog for .
D, [2016-10-30T23:58:49.947938 #18495] DEBUG -- : Setting up Puppet catalog build for .
D, [2016-10-30T23:58:49.947982 #18495] DEBUG -- : Catalog for . will be built with OctocatalogDiff::Catalog::Computed
D, [2016-10-30T23:58:49.948035 #18495] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2016-10-30T23:58:49.948118 #18495] DEBUG -- : Start retrieving facts for build03.nelson.va from OctocatalogDiff::Catalog::Computed
D, [2016-10-30T23:58:49.948119 #18490] DEBUG -- : Success build_catalog for origin/production
D, [2016-10-30T23:58:49.948173 #18495] DEBUG -- : Retrieving facts from PuppetDB
D, [2016-10-30T23:58:50.164067 #18495] DEBUG -- : Success retrieving facts for build03.nelson.va from OctocatalogDiff::Catalog::Computed
D, [2016-10-30T23:58:50.165255 #18495] DEBUG -- : Symlinked /tmp/d20161030-18495-t0lgqg/environments/production -> /home/rnelson0/puppet/controlrepo
D, [2016-10-30T23:58:50.165401 #18495] DEBUG -- : Installed puppetdb.conf file at /tmp/d20161030-18495-t0lgqg/puppetdb.conf
D, [2016-10-30T23:58:50.167107 #18495] DEBUG -- : Installed routes.yaml file at /tmp/d20161030-18495-t0lgqg/routes.yaml
D, [2016-10-30T23:58:50.178975 #18495] DEBUG -- : Installed fact file at /tmp/d20161030-18495-t0lgqg/var/yaml/facts/build03.nelson.va.yaml
D, [2016-10-30T23:58:50.179398 #18495] DEBUG -- : Installed CA certificate in /tmp/d20161030-18495-t0lgqg/var/ssl/certs/ca.pem
D, [2016-10-30T23:58:50.450442 #18495] DEBUG -- : Failed build_catalog for .: RuntimeError Unable to determine Puppet version: /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:95:in `block in materialize': Could not find rake-11.3.0 in any of the sources (Bundler::GemNotFound)
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:88:in `map!'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:88:in `materialize'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/definition.rb:140:in `specs'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/definition.rb:185:in `specs_for'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/definition.rb:174:in `requested_specs'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/environment.rb:19:in `requested_specs'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/runtime.rb:14:in `setup'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler.rb:95:in `setup'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/setup.rb:19:in `<top (required)>'
        from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'

bundler: failed to load command: octocatalog-diff (/home/rnelson0/puppet/controlrepo/vendor/ruby/bin/octocatalog-diff)
OctocatalogDiff::CatalogDiff::Cli::Catalogs::CatalogError: Catalog for 'to' (.) failed to compile with RuntimeError: Unable to determine Puppet version: /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:95:in `block in materialize': Could not find rake-11.3.0 in any of the sources (Bundler::GemNotFound)
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:88:in `map!'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:88:in `materialize'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/definition.rb:140:in `specs'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/definition.rb:185:in `specs_for'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/definition.rb:174:in `requested_specs'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/environment.rb:19:in `requested_specs'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/runtime.rb:14:in `setup'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler.rb:95:in `setup'
        from /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/setup.rb:19:in `<top (required)>'
        from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'

   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/puppetversion.rb:20:in `puppet_version'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:66:in `puppet_version'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:181:in `block in run_puppet'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:178:in `upto'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:178:in `run_puppet'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:114:in `build_catalog'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:59:in `build'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog.rb:60:in `build'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:221:in `build_catalog'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:26:in `call'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:26:in `execute'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:114:in `block in run_tasks_parallel'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:451:in `call'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:451:in `call_with_index'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:422:in `process_incoming_jobs'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:404:in `block in worker'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:395:in `fork'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:395:in `worker'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:386:in `block in create_workers'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:385:in `each'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:385:in `each_with_index'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:385:in `create_workers'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:327:in `work_in_processes'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:241:in `map'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/parallel-1.9.0/lib/parallel.rb:203:in `each'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:90:in `run_tasks_parallel'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:72:in `run_tasks'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:98:in `build_catalog_parallelizer'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:32:in `catalogs'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:180:in `catalog_only'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:88:in `cli'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/bin/octocatalog-diff:74:in `<top (required)>'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/bin/octocatalog-diff:23:in `load'
   /home/rnelson0/puppet/controlrepo/vendor/ruby/bin/octocatalog-diff:23:in `<top (required)>'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/cli/exec.rb:63:in `load'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/cli/exec.rb:63:in `kernel_load'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/cli/exec.rb:24:in `run'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/cli.rb:304:in `exec'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/cli.rb:11:in `start'
   /usr/local/share/gems/gems/bundler-1.12.5/exe/bundle:27:in `block in <top (required)>'
   /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/friendly_errors.rb:98:in `with_friendly_errors'
   /usr/local/share/gems/gems/bundler-1.12.5/exe/bundle:19:in `<top (required)>'
   /usr/local/bin/bundle:23:in `load'
   /usr/local/bin/bundle:23:in `<main>'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:209:in `add_parallel_result'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:118:in `block in build_catalog_parallelizer'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:112:in `each'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:112:in `build_catalog_parallelizer'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:32:in `catalogs'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:180:in `catalog_only'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:88:in `cli'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/gems/octocatalog-diff-0.5.1/bin/octocatalog-diff:74:in `<top (required)>'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/bin/octocatalog-diff:23:in `load'
  /home/rnelson0/puppet/controlrepo/vendor/ruby/bin/octocatalog-diff:23:in `<top (required)>'

Platform and version information

  • Your OS: CentOS 7
  • Your Ruby version: ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
  • Your version of Puppet: system: 4.5.2; gem: 4.7.0
  • Your version of octocatalog-diff: 0.5.1

Do the tests pass from a clean checkout?

Negative, these are the troubleshooting steps that are failing

Anything else to add that you think will be helpful?

You can find me on the Puppet Community slack most days and I can definitely set aside a chunk of time for contiguous work on this if it would help.

kpaulisse commented 7 years ago

The pertinent error is:

OctocatalogDiff::CatalogDiff::Cli::Catalogs::CatalogError: Catalog for 'to' (.) failed to compile with RuntimeError: Unable to determine Puppet version: /usr/local/share/gems/gems/bundler-1.12.5/lib/bundler/spec_set.rb:95:in `block in materialize': Could not find rake-11.3.0 in any of the sources (Bundler::GemNotFound)

and in particular:

Could not find rake-11.3.0 in any of the sources (Bundler::GemNotFound)

I suspect you have settings in your environment that are needed to allow Puppet to run (perhaps bundler settings or the like).

During our development we ran into a bunch of problems when a different version of Ruby was used to run octocatalog-diff than was used for Puppet, so when the Puppet binary is run, it's in a very clean environment by default. source

By default we are passing only HOME, PATH, and PWD to your Puppet binary. If you want to pass other variables from your environment, there is a command line option:

--pass-env-vars VAR1[,VAR2[,...]]          Environment variables to pass

Assuming /opt/puppetlabs/puppet/bin/puppet is the default binary that ships from Puppet, then I'd be very keen to make this work with all the default arguments. Could you confirm that's a stock Puppet install and let me know what version that comes with so I can try it out?

I'm happy to debug with you in real-time via Slack, but I wanted to identify here in public what I think the problem likely is.

rnelson0 commented 7 years ago

The agent is puppet-agent-1.5.2-1.el7.x86_64 and reports version 4.5.2. I do receive the same error if I specifically base the binary location (bundle exec octocatalog-diff -d -n build03.nelson.va -o /tmp/catalog.json --bootstrapped-to-dir=$PWD --catalog-only --puppet-binary /opt/puppetlabs/bin/puppet). Since I'm using bundler, maybe that's the interference point? I do have the rake gem, though.

$ be gem list

*** LOCAL GEMS ***

ansi (1.5.0)
bundler (1.12.5)
colorize (0.8.1)
diff-lcs (1.2.5)
diffy (3.1.0)
docile (1.1.5)
domain_name (0.5.20161021)
facter (2.4.6)
facterdb (0.3.8)
fast_gettext (1.1.0)
generate-puppetfile (0.9.11)
gettext (3.2.2)
gettext-setup (0.7)
hashdiff (0.3.0)
hiera (1.3.4)
hirb (0.7.3)
http-cookie (1.0.3)
httparty (0.14.0)
jgrep (1.4.1)
json (1.8.3)
json_pure (2.0.2)
locale (2.1.2)
mcollective-client (2.9.0)
metaclass (0.0.4)
metadata-json-lint (0.0.19)
mime-types (2.99.3)
mocha (1.2.1)
multi_xml (0.5.5)
netrc (0.11.0)
octocatalog-diff (0.5.1)
parallel (1.9.0)
puppet (3.8.7)
puppet-blacksmith (3.4.0)
puppet-lint (2.0.2)
puppet-lint-classes_and_types_beginning_with_digits-check (0.1.2)
puppet-lint-leading_zero-check (0.1.1)
puppet-lint-trailing_comma-check (0.3.2)
puppet-lint-unquoted_string-check (0.3.0)
puppet-lint-version_comparison-check (0.2.1)
puppet-syntax (2.1.1)
puppetlabs_spec_helper (1.2.2)
rake (11.3.0)
rest-client (1.8.0)
rspec (3.5.0)
rspec-core (3.5.4)
rspec-expectations (3.5.0)
rspec-mocks (3.5.0)
rspec-puppet (2.5.0)
rspec-puppet-facts (1.7.0)
rspec-support (3.5.0)
rugged (0.25.0b10)
semantic_puppet (0.1.4)
simplecov (0.12.0)
simplecov-console (0.3.1)
simplecov-html (0.10.0)
spdx-licenses (1.1.0)
stomp (1.4.3)
systemu (2.6.5)
text (1.3.1)
unf (0.1.4)
unf_ext (0.0.7.2)
kpaulisse commented 7 years ago

Hey @rnelson0,

I pulled a brand new clean CentOS virtualbox, installed Puppet on it, installed the base control repo on it, and ran the octocatalog-diff with what I believe is the closest to your settings. I did not experience the problem you reported and was able to generate a catalog successfully (albeit one that did absolutely nothing.) I'll arrange a screen share with you on slack to see if we can nail down the exact cause of this issue.

kpaulisse commented 7 years ago

The root cause of this is https://github.com/github/octocatalog-diff/issues/14