search

aws / aws-codedeploy-agent

Host Agent for AWS CodeDeploy
https://aws.amazon.com/codedeploy
Apache License 2.0
329 stars 187 forks source link

NoMethodError while booting child #222

Closed matchbookmac closed 2 years ago

matchbookmac commented 5 years ago

Deployments started failing without getting to ApplicationStop event. Output of logs (below) indicates a NoMethodError is preventing the agent from functioning.

VERSION: agent_version: OFFICIAL_1.0-1.1597_deb

Logs:

2019-07-25 19:25:11 INFO  [codedeploy-agent(29051)]: master 29051: Spawned child 1/1
2019-07-25 19:25:12 ERROR [codedeploy-agent(29066)]: booting child: error during start or run: NoMethodError - undefined method `underscore' for Seahorse::Util:Module - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:51:in `command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:76:in `<class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:20:in `<module:CodeDeployPlugin>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:18:in `<module:Plugins>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:17:in `<module:InstanceAgent>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:16:in `<top (required)>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in `require_relative'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:17:in `block in load_plugins'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:13:in `each'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:13:in `load_plugins'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:29:in `prepare_run'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:in `block in prepare_run_with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:78:in `with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:in `prepare_run_with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in `block in spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:283:in `block (2 levels) in replace_terminated_children'
/opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2019-07-25 19:25:12 INFO  [codedeploy-agent(29051)]: master 29051: Received CHLD - cleaning dead child process
2019-07-25 19:25:12 INFO  [codedeploy-agent(29051)]: master 29051: been told to replace child 29066
2019-07-25 19:25:12 INFO  [codedeploy-agent(29051)]: master 29051: not enough child processes running - missing at least 1 - respawning
rohkat-aws commented 5 years ago

@matchbookmac did you install ruby sdk after installing the agent ?

matchbookmac commented 5 years ago

The instance this happened on has been running the agent with no problem since 11/8/2018. At that time, agent was installed with:

#!/bin/bash
CODEDEPLOY_INSTALLER_URL="https://aws-codedeploy-us-east-1.s3.amazonaws.com/latest/install"
RVM_INSTALLER_URL="https://raw.githubusercontent.com/wayneeseguin/rvm/stable/binscripts/rvm-installer"

curl -sSL https://rvm.io/mpapis.asc | gpg --import -
curl -sSL "$RVM_INSTALLER_URL" | bash -s stable --ruby={2.3.0,2.4.1,2.4.2,2.4.3} --gems=bundler,dotenv,pry
usermod -aG rvm root
usermod -aG rvm ubuntu

cd /home/ubuntu || true
wget "$CODEDEPLOY_INSTALLER_URL"
chmod +x ./install
bash -l -c "ruby ./install auto"
service codedeploy-agent start

The logs show the agent running normally until first instance of this error at 2019-07-24 01:16:07 server time.

2019-07-24 01:05:51 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:06:51 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.154567 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:06:52 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:07:52 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.154421 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:07:53 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:08:54 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.708441 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:08:55 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:09:55 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.232362 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:09:56 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:10:57 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.406749 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:10:58 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:11:58 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.677543 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:11:59 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:13:00 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.52441 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:13:01 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:14:01 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.248747 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:14:02 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:15:02 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 60.160273 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:15:03 INFO  [codedeploy-agent(1657)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-07-24 01:15:07 INFO  [codedeploy-agent(22326)]: Checking first if any deployment lifecycle event is in progress master 1653
2019-07-24 01:15:07 INFO  [codedeploy-agent(22326)]: Stopping master 1653
2019-07-24 01:15:07 INFO  [codedeploy-agent(1653)]: master 1653: Received TERM - stopping children and shutting down
2019-07-24 01:15:07 INFO  [codedeploy-agent(1657)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller of master 1653: Received TERM - setting internal shutting down flag and possibly finishing last run
2019-07-24 01:15:07 INFO  [codedeploy-agent(1657)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Gracefully shutting down agent child threads now, will wait up to 7200 seconds
2019-07-24 01:15:07 INFO  [codedeploy-agent(1657)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: All agent child threads have been shut down
2019-07-24 01:15:07 INFO  [codedeploy-agent(1657)]: agent exiting now
2019-07-24 01:16:04 INFO  [codedeploy-agent(1657)]: [Aws::CodeDeployCommand::Client 200 61.19316 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-1:368304857153:instance/i-0feb9a1ce7078f68f")

2019-07-24 01:16:05 INFO  [codedeploy-agent(1657)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller of master 1653: shutting down
2019-07-24 01:16:07 INFO  [codedeploy-agent(22462)]: master 22462: Spawned child 1/1
2019-07-24 01:16:07 ERROR [codedeploy-agent(22466)]: booting child: error during start or run: NoMethodError - undefined method `underscore' for Seahorse::Util:Module - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:51:in `command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:76:in `<class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:20:in `<module:CodeDeployPlugin>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:18:in `<module:Plugins>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:17:in `<module:InstanceAgent>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:16:in `<top (required)>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in `require_relative'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin.rb:2:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
mwjones-aws commented 2 years ago

Mitigated in https://github.com/aws/aws-codedeploy-agent/commit/165d2b2d0af88310ee14da17ff398a435cd41014