Closed mdechiaro closed 1 month ago
That error is coming from this line https://github.com/puppetlabs/puppet/blob/32499bca6e025a3ddfd7f0dd58d952baeba658c2/lib/puppet/functions/eyaml_lookup_key.rb#L10
For some reason Puppet.features.hiera_eyaml?
is returning false
. I can't reproduce that using puppetserver 8.6.0 and puppet-agent 8.8.1:
# /opt/puppetlabs/puppet/bin/facter os
{
architecture => "amd64",
distro => {
codename => "jammy",
description => "Ubuntu 22.04.3 LTS",
id => "Ubuntu",
release => {
full => "22.04",
major => "22.04"
}
},
family => "Debian",
hardware => "x86_64",
name => "Ubuntu",
release => {
full => "22.04",
major => "22.04"
},
selinux => {
enabled => false
}
}
# dpkg -l puppet-agent puppetserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-==========================================================================================>
ii puppet-agent 8.8.1-1jammy amd64 The Puppet Agent package contains all of the elements needed to run puppet, including ruby>
ii puppetserver 8.6.2-1jammy all Puppet Labs puppetserver
# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true
# /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/parser/parser.rb
Additionally the version of the hiera-eyaml
gem 3.4.0 is unchanged between puppet-agent 8.6.0 and 8.8.1, so seems unlikely this is caused by our agent.
Are you sure you did a full restart of puppetserver (as opposed to reload)?
Are you sure you installed puppet-agent from our repos? As opposed to Ubuntu/Debian repos?
@joshcooper Thank for you the fast response! Also, thanks for those extra troubleshooting commands. I think I found the problem, and it seems there's an issue with external hiera-eyaml gem, v4.1.0, with puppet versions 8.7.0+.
# dpkg -l puppet-agent puppetserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-======================================================================================================
ii puppet-agent 8.6.0-1jammy amd64 The Puppet Agent package contains all of the elements needed to run puppet, including ruby and facter.
ii puppetserver 8.6.2-1jammy all Puppet Labs puppetserver
# sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true
# sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
# sudo apt install puppet-agent=8.7.0-1jammy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
puppet-agent
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 35.9 MB of archives.
After this operation, 492 kB of additional disk space will be used.
Get:1 https://apt.puppetlabs.com jammy/puppet8 amd64 puppet-agent amd64 8.7.0-1jammy [35.9 MB]
Fetched 35.9 MB in 1s (71.7 MB/s)
(Reading database ... 152735 files and directories currently installed.)
Preparing to unpack .../puppet-agent_8.7.0-1jammy_amd64.deb ...
Unpacking puppet-agent (8.7.0-1jammy) over (8.6.0-1jammy) ...
Setting up puppet-agent (8.7.0-1jammy) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
# sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
false
# sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
This fixes the issue.
# sudo /opt/puppetlabs/server/bin/puppetserver gem uninstall hiera-eyaml
Select gem to uninstall:
1. hiera-eyaml-3.4.0
2. hiera-eyaml-4.1.0
3. All versions
> 2
Successfully uninstalled hiera-eyaml-4.1.0
# sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/parser/parser.rb
# sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true
It still works.
:~$ sudo /opt/puppetlabs/server/bin/puppetserver gem uninstall hiera-eyaml
Remove executables:
eyaml
in addition to the gem? [Yn] y
Removing eyaml
Successfully uninstalled hiera-eyaml-3.4.0
:~$ sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/parser/parser.rb
:~$ sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true
After removing all external installs of hiera-eyaml, we are now using the vendor_gem. That fixes the issue. It seems hiera-eyaml v4.1.0 was working until puppet agent 8.6.0, but fails on 8.7.0. We removed all external installs to use the vendor supplied gem to fix the issue.
Hey, do I read it correctly that puppetserver vendors hiera-eyaml 3.4.0 but you also install hiera-eyaml 3.4.0 and 4.1.0? Did you try uninstalling 3.4.0 and only have 4.1.0? I want to figure out if this is a conflict of between differeny hiera-eyaml versions or hiera-yaml and puppet.
I cannot reproduce it:
My test system is Ubuntu 22.04
root@puppetserver:~/# puppet --version
8.8.1
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver --version
puppetserver version: 8.6.2
root@puppetserver:~/#
The vanilla installation works:
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
2024-09-06 16:07:56,683 INFO [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
true
root@puppetserver:~/#
But it still works after installing hiera-eyaml 4.1.0
root@puppetserver:~/controlrepo# /opt/puppetlabs/server/bin/puppetserver gem install hiera-eyaml
2024-09-06 16:17:59,485 INFO [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
Fetching highline-2.1.0.gem
Fetching hiera-eyaml-4.1.0.gem
Successfully installed highline-2.1.0
Successfully installed hiera-eyaml-4.1.0
Parsing documentation for highline-2.1.0
Installing ri documentation for highline-2.1.0
Parsing documentation for hiera-eyaml-4.1.0
Installing ri documentation for hiera-eyaml-4.1.0
Done installing documentation for highline, hiera-eyaml after 12 seconds
2 gems installed
root@puppetserver:~/controlrepo# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
2024-09-06 16:18:57,221 INFO [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
true
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
2024-09-06 16:21:40,126 INFO [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
root@puppetserver:~/#
it even works after restarting puppetserver:
root@puppetserver:~/# systemctl restart puppetserver
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
2024-09-06 16:22:34,480 INFO [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
root@puppetserver:~/controlrepo# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
2024-09-06 16:22:46,146 INFO [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
true
root@puppetserver:~/#
any chance your umask isn't 0022 and the files, for example /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
, don't have 0644 permissions?
Another possibility is hiera-eyaml
depends on strscan
which I believe is a gem with native extensions. So you'd need a JRuby compatible version of strscan
in order to load hiera-eyaml
.
indeed there's a require 'strscan'
https://github.com/voxpupuli/hiera-eyaml/blob/dd86f485a35f4dd75d09334246ad097214d5b48a/lib/hiera/backend/eyaml/parser/parser.rb#L1
@mdechiaro you'll need to look closely at which gems are installed in JRuby. Recommend running these commands to try to narrow down the problem:
/opt/puppetlabs/server/bin/puppetserver gem list
/opt/puppetlabs/server/bin/puppetserver gem env
@joshcooper unfortunately I don't have time to troubleshoot this issue further. The fix for us was to use the vendor_gem version, and I am OK with closing this issue. Thanks for your help.
Thanks @mdechiaro, I'll close this
Describe the Bug
Puppetserver fails with
Function Load Error for function 'eyaml_lookup_key'
and produces error messages in the logs if puppet agent is upgraded to v8.7.0+. The latest version of puppet agent, v8.8.1, also produces this error.Downgrading puppet agent to v8.6.0 on system running puppetserver fixes the issue.
Expected Behavior
Hiera-eyaml shouldn't break when puppet agent is upgraded to 8.7.0+ on a system running puppetserver.
Steps to Reproduce
Steps to reproduce the behavior:
Function Load Error for function 'eyaml_lookup_key'
errors.Environment
Additional Context
Full error in puppetserver.log is below