High memory consumption and memory leak (still an issue)

[thehappycoder]

Opening a new issue because the problem described here https://github.com/aws/aws-codedeploy-agent/issues/32 still exists.

Basically, when codedeploy finishes, the agent doesn't release the memory back to the OS and the workaround is to restart the agent.

I've enabled verbose logging. Briefly looking at the logs, seems that unzip utility is used to unarchive the bundle:

$ cat codedeploy-agent.log | grep "LinuxUtil"
2021-10-10 09:58:26 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Dirs to delete: ["/opt/codedeploy-agent/deployment-root/e23efd0e-df20-42be-91d9-cd495d6e0abc/d-BWQ263SLC"]
2021-10-10 09:58:26 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Deleting dir: /opt/codedeploy-agent/deployment-root/e23efd0e-df20-42be-91d9-cd495d6e0abc/d-BWQ263SLC
2021-10-10 09:58:32 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Command status: pid 4311 exit 0
2021-10-10 09:58:32 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Command output: 
2021-10-10 09:58:34 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: extract_zip - dst : /opt/codedeploy-agent/deployment-root/e23efd0e-df20-42be-91d9-cd495d6e0abc/d-5ZDWO7ULC/deployment-archive
2021-10-10 09:58:34 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Executing unzip -qo /opt/codedeploy-agent/deployment-root/e23efd0e-df20-42be-91d9-cd495d6e0abc/d-5ZDWO7ULC/bundle.tar -d /opt/codedeploy-agent/deployment-root/e23efd0e-df20-42be-91d9-cd495d6e0abc/d-5ZDWO7ULC/deployment-archive
2021-10-10 09:58:46 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Command status: pid 4315 exit 0
2021-10-10 09:58:46 DEBUG [codedeploy-agent(4114)]: InstanceAgent::LinuxUtil: Command output:

Where can I securely share the 127MB codedeploy-agent.log file (generated from a single deployment)? I don't want to share it publicly.

My environment: EC2 instance running Ubuntu 20 ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [aarch64-linux-gnu] codedeploy-agent 1.3.2-1902

[erashdan]

+1

[shahawiz]

+1

[t0shiii]

Hi @thehappycoder,

I'm with the CodeDeploy team at AWS and I've done some investigation regarding the memory leak issue you have reported.

While I have successfully reproduced the issue you mentioned using the same environment you have described in your previous post in this thread, I would like to mention that this issue does not occur when using Amazon Linux 2.

Here are some memory usage graphs that highlight this behavior.

We are working to resolve this issue; at this time I cannot provide any ETAs on when this issue will be resolved.

I also understand that you want to securely share the log file. My suggestion would be to log your output to CloudWatch and open a support ticket with AWS Support.

[erashdan]

I have a droplet on digitalocean with 1GB / 25GB Disk I used to have a full memory issue on ubuntu 18.04 every ~15 minutes, but I create a swap space into VPS, and the problem was solved. Add Swap

[mashedkeyboard]

Two years later on and I've just hit this issue while trying to set up AWS CodeDeploy. Is this actually something on the roadmap to be fixed, or is the intention to only support CodeDeploy's use on Amazon Linux?

[DevGerry]

The issue also appears to be occurring on Amazon Linux 2023.

When I start codedeploy-agent at Amazon Linux 2023 (2023.4.20240528.0), it takes about 20MB. But, after several deploy it takes 450MB. Memory doesn't free when deploy is finished. In this form, memory occupancy is continuously increased, affecting other services.

Amazon Linux 2 EOL is scheduled for June 30, 25. However can't be migrated to Amazon Linux 2023, which, like Ubuntu, is difficult to memory leak is resolved.

My environment: EC2 instance running Amazon Linux 2023 (2023.4.20240528.0) ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] codedeploy-agent_version: OFFICIAL_1.7.0-92_rpm