aws / elastic-beanstalk-roadmap

AWS Elastic Beanstalk roadmap
https://aws.amazon.com/elasticbeanstalk/
Creative Commons Attribution Share Alike 4.0 International
283 stars 11 forks source link

Log messages are sent twice to CloudWatch #269

Open dennis-grunert opened 2 years ago

dennis-grunert commented 2 years ago

Community Note

Tell us about your request The elastic beanstalk has the following bug: Once an EC2 instance is terminated, the log file of the container is duplicated. This results in all messages to appear twice in CloudWatch. The fix consists in removing one unnecessary line in a systemd unit file (see below).

Is this request specific to an Elastic Beanstalk platform? Docker platform branch using the compose method.

Steps to reproduce:

  1. Create an elastic beanstalk with AMI amazon/aws-elasticbeanstalk-amzn-2.0.20220719.64bit-eb_docker_amazon_linux_2-hvm-2022-08-02T21-04 (probably also for later versions) and platform version aws-elasticbeanstalk-amzn-2.0.20220719.64bit-eb_docker_amazon_linux_2-hvm-.
  2. Deploy any docker image with the docker compose (!) method.
  3. SSH into the machine (or use SSM).
  4. Ensure that your application logs to /var/log/eb-docker/containers/eb-current-app/eb-stdouterr.log on the machine.
  5. Run systemctl restart eb-docker-compose-log.service (systemctl stop will happen during termination of an instance but we use this command here to still have access to the EC2 instance).
  6. Verify that the log messages has doubled now in the file /var/log/eb-docker/containers/eb-current-app/eb-stdouterr.log.

Reason for the bug: The systemd unit file /etc/systemd/system/eb-docker-compose-log.service contains the line

ExecStopPost=/bin/sh -c "exec sort -u /var/log/eb-docker/containers/eb-current-app/eb-stdouterr.log -o /var/log/eb-docker/containers/eb-current-app/eb-stdouterr.log"

which duplicates all log messages (and which will then be sent to CloudWatch).

Possible fix: Simply remove the above mentioned line from the systemd unit file.

Are you currently working around this issue? No.

Additional information: The bug was already reproduced by AWS Support in the support case 11043729491. They suggested to write this ticket to track the progress of fixing it.

zyv commented 1 year ago

Ping from case 12832694461.