search

aws / aws-codedeploy-agent

Host Agent for AWS CodeDeploy
https://aws.amazon.com/codedeploy
Apache License 2.0
329 stars 187 forks source link

CodeDeploy agent does not work when IMDSv2 is required in EC2 #269

Closed sharath-sequoia closed 4 years ago

sharath-sequoia commented 4 years ago

For improving security, we made the use of IMDSv2 as required as per https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html. After doing this, all deployments are failing in EC2 instances. I looked at the agent logs, and it looks like it is repeatedly trying to start the service, but failing every time. Additional information: OS: Ubuntu 16.04 CodeDeploy agent version: 1.1.2-1855

Below are the logs from agent:

/var/lib/gems/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in block in create_with_logging_context' 2020-09-14 10:29:26 INFO [codedeploy-agent(1692)]: master 1692: Received CHLD - cleaning dead child process 2020-09-14 10:29:26 INFO [codedeploy-agent(1692)]: master 1692: been told to replace child 1704 2020-09-14 10:29:26 INFO [codedeploy-agent(1692)]: master 1692: not enough child processes running - missing at least 1 - respawning 2020-09-14 10:29:31 INFO [codedeploy-agent(1692)]: master 1692: Spawned child 1/1 2020-09-14 10:29:31 INFO [codedeploy-agent(1717)]: On Premises config file does not exist or not readable 2020-09-14 10:29:31 ERROR [codedeploy-agent(1717)]: booting child: error during start or run: RuntimeError - HTTP error from metadata service, code 401 - /opt/codedeploy-agent/lib/instance_metadata.rb:47:inblock in http_get' /usr/lib/ruby/2.3.0/net/http.rb:853:in start' /usr/lib/ruby/2.3.0/net/http.rb:584:instart' /opt/codedeploy-agent/lib/instance_metadata.rb:43:in http_get' /opt/codedeploy-agent/lib/instance_metadata.rb:55:indoc' /opt/codedeploy-agent/lib/instance_metadata.rb:21:in region' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:38:ininitialize' /opt/codedeploy-agent/lib/instance_agent/agent/base.rb:10:in new' /opt/codedeploy-agent/lib/instance_agent/agent/base.rb:10:inrunner' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:32:in block in prepare_run' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:78:inwith_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:31:in prepare_run' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:inblock in prepare_run_with_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:78:in with_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:inprepare_run_with_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:in start' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:inblock in spawn_child' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in fork' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:inspawn_child' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:283:in block (2 levels) in replace_terminated_children' /var/lib/gems/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:inblock in create_with_logging_context' 2020-09-14 10:29:31 ERROR [codedeploy-agent(1717)]: booting child: error during start or run: SystemExit - exit - /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:90:in exit' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:90:inrescue in with_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:77:in with_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:31:inprepare_run' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:in block in prepare_run_with_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:78:inwith_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:in prepare_run_with_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:instart' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in block in spawn_child' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:infork' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in spawn_child' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:283:inblock (2 levels) in replace_terminated_children' /var/lib/gems/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context' 2020-09-14 10:29:32 INFO [codedeploy-agent(1692)]: master 1692: Received CHLD - cleaning dead child process 2020-09-14 10:29:32 INFO [codedeploy-agent(1692)]: master 1692: been told to replace child 1717 2020-09-14 10:29:32 INFO [codedeploy-agent(1692)]: master 1692: not enough child processes running - missing at least 1 - respawning

philstrong commented 4 years ago

1.2.1 is adding IMDSv2 support and is being rolled out now in commercial regions. Expected to be rolled out completely next week.

sharath-sequoia commented 4 years ago

Thanks for the update @philstrong. Current latest release is 1.1.2. I believe what you mean is that 1.2.1 is expected to be released for everyone next week. Please correct me if I am wrong. And what does commercial regions mean?

brndnblck commented 4 years ago

CodeDeploy agent v1.2.1 is available now in all commercial regions and includes support for IMDSv2.

moloch-- commented 2 years ago

This does not appear to be fixed in the Windows version of the code deploy agent. The update script reports the latest version as 1.0.1.1597 and the agent fails to communicate with IMDSv2.