puppetlabs / puppet

Server automation framework and application
https://puppet.com/open-source/#osp
Apache License 2.0
7.43k stars 2.2k forks source link

Puppet agent throws random errors #9324

Closed lordwebbie closed 6 months ago

lordwebbie commented 6 months ago

Describe the Bug

For a project, we operate approx. 40 servers of exact same hardware specifications. On most of them, puppet runs stable. But on several others, puppet seems to be instable and throws random errors. We do not have any clue, what the problem could be.

Expected Behavior

Puppet agent should not throw random errors.

Steps to Reproduce

To reproduce, we can just use any simple puppet agent command and execute it multiple times after one another.

e.g. configprint:

root@host ~ # puppet agent --configprint ssldir
Error: Could not initialize global default settings: can't modify frozen File: #<File:/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/hocon-1.3.1/lib/hocon/impl/config_node_simple_value.rb>
root@host ~ # puppet agent --configprint ssldir
/etc/puppetlabs/puppet/ssl
root@host ~ # puppet agent --configprint ssldir
/etc/puppetlabs/puppet/ssl
root@host ~ # puppet agent --configprint ssldir
/etc/puppetlabs/puppet/ssl
root@host ~ #
root@host ~ # puppet agent --configprint ssldir
Traceback (most recent call last):
        21: from /opt/puppetlabs/puppet/bin/puppet:4:in `<main>'
        20: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        19: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        18: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
        17: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `require_relative'
        16: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `<top (required)>'
        15: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `require_relative'
        14: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `<top (required)>'
        13: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `require_relative'
        12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `<top (required)>'
        11: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `require_relative'
        10: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:1:in `<top (required)>'
         9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:120:in `<module:Puppet>'
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/model/ast.rb:4669:in `register_pcore_types'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loaders.rb:111:in `static_loader'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/pcore.rb:65:in `init'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:61:in `register_ptypes'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:61:in `each'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:65:in `block in register_ptypes'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:760:in `register_ptype'
         1: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:53:in `create_ptype'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/pcore.rb:103:in `create_object_type': can't modify frozen Hash: {} (FrozenError)

Example: confdir

root@host ~ # puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@host ~ # puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@host ~ # puppet agent --configprint confdir
Traceback (most recent call last):
        27: from /opt/puppetlabs/puppet/bin/puppet:4:in `<main>'
        26: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        25: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        24: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
        23: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `require_relative'
        22: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `<top (required)>'
        21: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `require_relative'
        20: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `<top (required)>'
        19: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `require_relative'
        18: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `<top (required)>'
        17: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `require_relative'
        16: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:1:in `<top (required)>'
        15: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:120:in `<module:Puppet>'
        14: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/model/ast.rb:4669:in `register_pcore_types'
        13: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loaders.rb:111:in `static_loader'
        12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/pcore.rb:65:in `init'
        11: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:68:in `register_ptypes'
        10: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:68:in `each'
         9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:68:in `block in register_ptypes'
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_meta_type.rb:48:in `resolve'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:738:in `_pcore_init_from_hash'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:738:in `map'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:738:in `each'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:744:in `block in _pcore_init_from_hash'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:744:in `new'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:286:in `initialize'
         1: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/type_asserter.rb:33:in `assert_instance_of'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/type_asserter.rb:41:in `report_type_mismatch': Puppet::Pops::Types::TypeAssertionError
root@host ~ # puppet agent --configprint confdir
/etc/puppetlabs/puppet

or even only version output:

root@host ~ # puppet agent --version
7.30.0
root@host ~ # puppet agent --version
Traceback (most recent call last):
        11: from /opt/puppetlabs/puppet/bin/puppet:4:in `<main>'
        10: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         9: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `require_relative'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:41:in `<top (required)>'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:129:in `<module:Puppet>'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/defaults.rb:231:in `initialize_default_settings!'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1019:in `define_settings'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1019:in `each'
         1: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1026:in `block in define_settings'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:763:in `newsetting': can't modify frozen Hash: {:default=>"", :desc=>"A command to run before every agent run.  If this command returns a non-zero\\n      return code, the entire Puppet run will fail.", :name=>:prerun_command, :section=>:main} (FrozenError)

Environment

Additional Context

Nothing.

joshcooper commented 6 months ago

@lordwebbie I can't reproduce this. There's something going on in your environment. If I had to guess you have a mix of ruby and/or puppet versions. You're better off reaching out in community slack to help track down the issue:

root@red-empathy:~# dpkg-query -W puppet-agent
puppet-agent    7.30.0-1bookworm
root@red-empathy:~# facter os
{
  architecture => "amd64",
  distro => {
    codename => "bookworm",
    description => "Debian GNU/Linux 12 (bookworm)",
    id => "Debian",
    release => {
      full => "12.5",
      major => "12",
      minor => "5"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Debian",
  release => {
    full => "12.5",
    major => "12",
    minor => "5"
  },
  selinux => {
    enabled => false
  }
}
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet