search

aws / aws-codedeploy-agent

Host Agent for AWS CodeDeploy
https://aws.amazon.com/codedeploy
Apache License 2.0
328 stars 188 forks source link

file already exists deployment failure and duplicate log entries #322

Open ericyancey opened 2 years ago

ericyancey commented 2 years ago

We have been successfully deploying our application to numerous Windows 2019 servers for some time now until this past weekend. A round of operating system updates were applied and it appears that SSM upgrade CodeDeploy at the same time. We attempted a deployment immediately after and it failed.

The error happens after the zip file is downloaded and during the extraction process. The error is:

Exception calling "ExtractToDirectory" with "2" argument(s): "The file 
'C:\ProgramData\Amazon\CodeDeploy\ce282efb-0b60-4d7f-8e29-233d4ce2710a\d-PMKMQMU3I\deployment-archive\appspec.yml' 
already exists."

I've seen numerous articles on rectifying this, but none seem to apply to our situation. What is odd is that every line in the CodeDeploy log file seems to be duplicated after the upgrade, almost as if there are two instances of it running. Here is a sample from the time the service is started:

2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: CodeDeploy Instance Agent Service: started
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: CodeDeploy Instance Agent Service: started
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: On Premises config file does not exist or not readable
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: On Premises config file does not exist or not readable
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Configuring deploy control client: Region="us-east-1"
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Configuring deploy control client: Region="us-east-1"
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Deploy control endpoint override=
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Deploy control endpoint override=
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Enable auth policy = false
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Enable auth policy = false
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Initializing Host Agent: Host Identifier = arn:aws:ec2:us-east-1:zzz:instance/zzz
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Initializing Host Agent: Host Identifier = arn:aws:ec2:us-east-1:z:instance/zzz
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Validating CodeDeploy Plugin Configuration
2022-06-12T06:09:36 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Validating CodeDeploy Plugin Configuration
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:36 INFO  [codedeploy-agent(1972)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:37 INFO  [codedeploy-agent(1972)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:37 INFO  [codedeploy-agent(1972)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:37 DEBUG [codedeploy-agent(1972)]: Current deploy control endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:37 DEBUG [codedeploy-agent(1972)]: Current deploy control endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2022-06-12T06:09:41 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: CodeDeploy Plugin Configuration is valid
2022-06-12T06:09:41 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: CodeDeploy Plugin Configuration is valid
2022-06-12T06:09:41 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2022-06-12T06:09:41 DEBUG [codedeploy-agent(1972)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2022-06-12T06:09:41 INFO  [codedeploy-agent(1972)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.3.2.1902_msi.
2022-06-12T06:09:41 INFO  [codedeploy-agent(1972)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.3.2.1902_msi.

After the initial error, a rollback is triggered that also fails. The AWS console reports the same error for both the initial deployment and the rollback:

Error Code: Unknown
Message:  No such file or directory @ rb_sysopen - C:\ProgramData/Amazon/CodeDeploy/ce282efb-0b60-4d7f-8e29-233d4ce2710a/d-OFHJUGT3I/deployment-archive/web\App_WebReferences\

Once it is all said and done, I checked the deployment folder and the zip file seems to have been extracted cleanly - the appsec file is there and the App_WebReferences.

Any help would be greatly appreciated, and please let me know if there is any other information I can provide to help remedy this.

Thanks

mwjones-aws commented 2 years ago

Can you please provide a little more information?

ericyancey commented 2 years ago

Sure thing! I actually have an update on this as well.

Here are answers to your questions:

  1. Does SSM update CodeDeploy Agent on a set schedule, or do you kick off the updates manually? Manual

  2. What version of CodeDeploy Agent were you running before that worked? 1.01

  3. What version of CodeDeploy Agent did SSM install recently? 1.3.2

  4. Are these EC2 or on-premise instances? If EC2, what region? What AMI? All of these are EC2 instances running in us-east-1 and the AMI for at least one instance was ami-05bb2dae0b1de90b3

  5. If EC2, are you using CodeDeploy's autoscaling group integration to deploy to your instances? We are not using autoscaling

  6. what file_exists_behavior setting are you using? fileExistsBehavior: 'OVERWRITE'

Additional Information: We triggered manual installations of CodeDeployAgent from SSM and cycled through the versions and tested the deployment until we got one to work. Deployments using versions 1.3.2, 1.3.1, 1.2.1 all failed with the same error mentioned in my original post.

But after installing 1.1.2, the deployments worked again!

Thank you for any help you can provide on this.

Eric

mwjones-aws commented 2 years ago

I wasn't able to find ami-05bb2dae0b1de90b3. Did you build it yourself, maybe using EC2 Image Builder?

If not, can you describe the filters/search parameters you used to find this AMI?

ericyancey commented 2 years ago

It looks like that particular AMI has been removed. The AMI was likely created after we hardened an AWS-provided image and then sysprepped it in the process of renaming it to join a domain.

However, this issue is happening with other instances from other AMIs, so I don't know that the base image has anything to do with it. I'm happy to gather any other information that you need.

thanks

ericyancey commented 1 year ago

Hi,

Sorry to reply on an old thread, but this issue still persists. We have been running version 1.0.1.1597 for some time now after not having any success with newer versions. I noticed a new version was released this spring so I installed it but the deployments fail.

This may not be an issue, but one thing I noticed is the log files for the newer agent have duplicated entries. In the log files for the old agent, I see this:

2023-07-14T05:35:02 DEBUG [codedeploy-agent(3020)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command =  nil
2023-07-14T05:35:03 DEBUG [codedeploy-agent(3020)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-07-14T05:35:03 INFO  [codedeploy-agent(3020)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.0.1.1597_msi.
2023-07-14T05:35:49 INFO  [codedeploy-agent(3020)]: [Aws::CodeDeployCommand::Client 200 45.691622 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-

However, for the new agent every line is duplicated throughout:

2023-07-17T13:10:50 DEBUG [codedeploy-agent(1804)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command =  nil
2023-07-17T13:10:50 DEBUG [codedeploy-agent(1804)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command =  nil
2023-07-17T13:10:51 DEBUG [codedeploy-agent(1804)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-07-17T13:10:51 DEBUG [codedeploy-agent(1804)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-07-17T13:10:51 INFO  [codedeploy-agent(1804)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.6.0.49_msi.
2023-07-17T13:10:51 INFO  [codedeploy-agent(1804)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.6.0.49_msi.
2023-07-17T13:11:36 INFO  [codedeploy-agent(1804)]: [Aws::CodeDeployCommand::Client 200 45.235753 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-e
2023-07-17T13:11:36 INFO  [codedeploy-agent(1804)]: [Aws::CodeDeployCommand::Client 200 45.235753 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-
ericyancey commented 1 year ago

We have been successfully deploying our application to numerous Windows 2019 servers for some time now until this past weekend. A round of operating system updates were applied and it appears that SSM upgrade CodeDeploy at the same time. We attempted a deployment immediately after and it failed.

The error happens after the zip file is downloaded and during the extraction process. The error is:

Exception calling "ExtractToDirectory" with "2" argument(s): "The file 
'C:\ProgramData\Amazon\CodeDeploy\ce282efb-0b60-4d7f-8e29-233d4ce2710a\d-PMKMQMU3I\deployment-archive\appspec.yml' 
already exists."

I found a solution to the original error message above. There was an issue in creating the bundle for the deployment where there were TWO appspec.yml files in the bundle in the root path. It is beyond me why the Powershell Compress-Archive command allows you to add mulitples of the same file into the same path, but when newer versions of CodeDeploy attempted to extract the archive the above error was generated.

Oddly enough, older versions of CodeDeploy didn't seem to mind that there were multiples.