Cannot decode output from Shim for Ruby Action

[mdechiaro]

OS: Ubuntu 22.04 Puppet: 8.3.1 Version: mod 'choria-mcollective_agent_shell', '1.0.6' Command: mco shell run 'echo foo' -W fqdn=puppetdev.domain.com

I tried setting the encoding as per the docs without success.

puppetdev:/etc/systemd/system$ cat /etc/systemd/system/choria-server.service.d/environment.conf 
[Service]
Environment="LC_ALL=en"

I get a Cannot decode output from Shim for Ruby action shell#run: EOF error on this system with simple commands. There are other Ubuntu 22.04 systems that do not have this issue, so it's strange that its occurring.

puppetdev.domain.com: ERROR: #<MCollective::RPC::Result:0x00007f3a2cd20628 @agent="shell", @action="run", @results={:sender=>"puppetdev.domain.com", :statuscode=>1, :statusmsg=>"Cannot decode output from Shim for Ruby action shell#run: EOF", :data=>{}}, @_ddl=#<MCollective::DDL::AgentDDL:0x00007f3a2c2aba08 @process_aggregate_functions=nil, @entities={"run"=>{:action=>"run", :input=>{:command=>{:prompt=>"Command", :description=>"Command to run", :type=>:string, :default=>nil, :optional=>false, :validation=>".*", :maxlength=>10240}, :user=>{:prompt=>"User", :description=>"User to run command as", :type=>:string, :default=>nil, :optional=>true, :validation=>".*", :maxlength=>1024}, :timeout=>{:prompt=>"Timeout", :description=>"Timeout to wait for the command to complete", :type=>:float, :default=>nil, :optional=>true}}, :output=>{:stdout=>{:description=>"stdout from the command", :display_as=>"stdout", :default=>nil}, :stderr=>{:description=>"stderr from the command", :display_as=>"stderr", :default=>nil}, :success=>{:description=>"did the process exit successfully", :display_as=>"success", :default=>nil}, :exitcode=>{:description=>"exit code of the command", :display_as=>"exitcode", :default=>nil}}, :display=>:always, :description=>"Run a command"}, "start"=>{:action=>"start", :input=>{:command=>{:prompt=>"Command", :description=>"Command to run", :type=>:string, :default=>nil, :optional=>false, :validation=>".*", :maxlength=>10240}, :user=>{:prompt=>"User", :description=>"User to run command as", :type=>:string, :default=>nil, :optional=>true, :validation=>".*", :maxlength=>1024}}, :output=>{:handle=>{:description=>"identifier to a running command", :display_as=>"handle", :default=>nil}}, :display=>:always, :description=>"Spawn a command"}, "status"=>{:action=>"status", :input=>{:handle=>{:prompt=>"Handle", :description=>"Handle of the command", :type=>:string, :default=>nil, :optional=>false, :validation=>"^[0-9a-z\\-]*$", :maxlength=>36}, :stdout_offset=>{:prompt=>"stdout_offset", :description=>"stdout_offset", :type=>:integer, :default=>nil, :optional=>true}, :stderr_offset=>{:prompt=>"stderr_offset", :description=>"stderr_offset", :type=>:integer, :default=>nil, :optional=>true}}, :output=>{:status=>{:description=>"status of the command", :display_as=>"status", :default=>nil}, :stdout=>{:description=>"stdout of the command", :display_as=>"stdout", :default=>nil}, :stderr=>{:description=>"stderr of the command", :display_as=>"stderr", :default=>nil}, :exitcode=>{:description=>"exitcode of the command", :display_as=>"exitcode", :default=>nil}}, :display=>:always, :description=>"Get status of managed command"}, "list"=>{:action=>"list", :input=>{}, :output=>{:jobs=>{:description=>"state of managed jobs", :display_as=>"jobs", :default=>nil}}, :display=>:always, :description=>"Get a list of all running commands"}, "kill"=>{:action=>"kill", :input=>{:handle=>{:prompt=>"Handle", :description=>"Handle of the command", :type=>:string, :default=>nil, :optional=>false, :validation=>"^[0-9a-z\\-]*$", :maxlength=>36}}, :output=>{}, :display=>:always, :description=>"Kill a command by handle"}}, @meta={:name=>"shell", :description=>"Run commands with the local shell", :author=>"Puppet Labs", :license=>"Apache-2.0", :version=>"1.0.6", :url=>"https://github.com/choria-plugins/shell-agent", :timeout=>180}, @usage="", @config=#<MCollective::Config:0x00007f3a2c4f8fe0 @configured=true, @subscribe=[], @pluginconf={"nrpe.conf_dir"=>"/etc/nrpe.d", "choria.discovery_host"=>"puppetdb.domain.com", "choria.discovery_port"=>"8085", "choria.middleware_hosts"=>"choria.domain.com:4222", "choria.puppetca_host"=>"puppetca.domain.com", "choria.puppetca_port"=>"8140", "choria.puppetdb_host"=>"puppetdb.domain.com", "choria.puppetdb_port"=>"8081", "choria.puppetserver_host"=>"puppetserver.domain.com", "choria.puppetserver_port"=>"8140", "choria.security.serializer"=>"json", "choria.tasks.wrapper_path"=>"/opt/puppetlabs/puppet/bin/execution_wrapper", "choria.use_srv"=>"false", "puppet.allow_server_override"=>"true", "puppet.resource_allow_managed_resources"=>"true", "puppet.resource_type_blacklist"=>"exec", "puppet.signal_daemon"=>"false", "actionpolicy.allow_unconfigured"=>"0", "actionpolicy.enable_default"=>"0"}, @connector="Nats", @securityprovider="Choria", @factsource="Yaml", @identity="client.domain.com", @registration="Base", @registerinterval=0, @registration_collective=nil, @registration_splay=false, @classesfile="/var/lib/puppet/state/classes.txt", @rpcaudit=false, @rpcauditprovider="", @rpcauthorization=false, @rpcauthprovider="", @configdir="/etc/choria", @color=true, @configfile="/etc/choria/client.conf", @logger_type="console", @keeplogs=5, @max_log_size=2097152, @rpclimitmethod=:first, @fact_cache_time=300, @loglevel="debug", @logfacility="user", @collectives=["mcollective"], @main_collective="mcollective", @ssl_cipher="aes-256-cbc", @direct_addressing=true, @direct_addressing_threshold=10, @default_discovery_method="mc", @default_discovery_options=[], @ttl=60, @mode=:client, @publish_timeout=2, @threaded=false, @soft_shutdown=false, @soft_shutdown_timeout=nil, @activate_agents=true, @connection_timeout=3, @default_batch_size=0, @default_batch_sleep_time=1>, @pluginname="shell", @plugintype=:agent, @requirements={}, @current_entity=nil>>

I tried enabling debug logs and even tried efforts done on 1215 without any luck. Any assistance in exposing more verbose logs would be helpful.

[ripienaar]

This means the shim crashes for some reason, you can try editing it to have the lines from this comment here and see if you get anything?

https://github.com/choria-io/go-choria/issues/1215#issuecomment-816798832

If you already did (as you mention) its difficult to say what it can be

[mdechiaro]

I think I found the issue, and support for Puppet 8 is missing in mcollective_choria module. This prevented a few packages from installing. When I downgraded to Puppet v7, the following changes were made:

Notice: /Stage[main]/Mcollective_agent_nettest/Mcollective::Module_plugin[mcollective_agent_nettest]/Package[net-ping]/ensure: created (corrective)
Notice: /Stage[main]/Mcollective_choria/Mcollective::Module_plugin[mcollective_choria]/Package[nats-pure]/ensure: created (corrective)
Notice: /Stage[main]/Puppetagent::Service/Service[puppet]: Triggered 'refresh' from 1 event
Info: Mcollective::Module_plugin[mcollective_agent_nettest]: Scheduling refresh of Service[choria-server]

After this change, upgrading to Puppet v8 works. This also explains why I couldn't get any logs with earlier testing, because necessary packages were missing.

[mdechiaro]

I am going to close this issue, since the issue was caused by missing gems. The module that installs these is currently missing Puppet 8 support, so opened an issue over there. The fix is:

---
mcollective_choria::gem_dependencies:
  "choria-mcorpc-support": "2.26.2"