Support for `--no-log-prefix` option when using docker-compose on Docker platform

[CatsMiaow]

https://repost.aws/questions/QUqAvwK8lhSCq79Tu27zf4Jw/is-there-a-way-to-remove-prefix-from-docker-compose-log-in-elastic-beanstalk

When a service is launched with docker-compose.yml on the Docker platform of AWS Elastic Beanstalk, the service name is added as a prefix in front of all logs.

When saving logs with Instance log streaming to CloudWatch Logs in the software configuration, prefixes are added to the logs in JSON format and stored as strings. e.g. {"timestamp": ... } to test-docker_1 | {"timestamp": ... }

Parsing JSON formatted logs will fail if logs are stored as strings in CloudWatch.

To solve this problem, need to add the --no-log-prefix option when the Docker service is started.

https://stackoverflow.com/questions/50360674/how-can-i-remove-prefix-with-service-name-from-logs

This option is not supported by docker-compose.yml and must be run by CLI.

https://docs.docker.com/engine/reference/commandline/compose_up/ docker-compose up --no-log-prefix

However, the docker-compose up command is automatically executed (docker-compose up -d) when the Docker service is started inside AWS EB.

When saving logs from AWS EB's Docker platform(Docker running on 64bit Amazon Linux 2/3.4.13) to CloudWatch, only the logs of the service without prefix should be recorded.

[dennis-grunert]

A workaround until AWS fixes this issue consists in modifying the systemd service eb-docker-compose-log with a elastic beanstalk predeploy hook to add the flag --no-log-prefix to the docker-compose logs command.

Add the below script remove_log_prefix.sh (with executable rights) in your deployment artifact under .platform/confighooks/predeploy and .platform/hooks/predeploy.

remove_log_prefix.sh:

#!/bin/bash
set -e

echo "Altering systemd service to print logs without docker compose container name prefix..."

if [ -f /opt/elasticbeanstalk/config/private/eb-docker-compose-log-start ]; then
  # AWS Linux 2 >=v3.6.2
  sed 's/\(docker[ -]compose logs\) --since/\1 --no-log-prefix --since/' -i /opt/elasticbeanstalk/config/private/eb-docker-compose-log-start
else
  # AWS Linux 2 <v3.6.2 && AWS Linux 2023 <=v4.0.1
  sed 's/\(docker[ -]compose logs\) -f/\1 --no-log-prefix -f/' -i /etc/systemd/system/eb-docker-compose-log.service
fi

systemctl daemon-reload
systemctl is-active --quiet eb-docker-compose-log && systemctl restart eb-docker-compose-log || echo "eb-docker-compose-log is not running..."

The script above is idempotent and does not restart the systemd service if it is not already running. Thus, it is save to run it both for a deployment and config hook.

(This is joint work with @mircohaug and @mvanderlee).

[mvanderlee]

@dennis-grunert the script caused my deployments to fail due to systemctl is-active --quiet service exiting with code 3.

I suspect you meant systemctl is-active --quiet eb-docker-compose-log

And to prevent the deployment from failing at all, I added an OR statement:

systemctl is-active --quiet eb-docker-compose-log && systemctl restart eb-docker-compose-log || echo "eb-docker-compose-log is not running..."

[dennis-grunert]

@mvanderlee You are correct, this was a typo. It is corrected in the original answer.

[dennis-grunert]

In newer AMIs, docker compose logs instead of docker-compose logs is used. I updated the sed command in the above code accordingly.

[dennis-grunert]

In the newest versions of AMIs, docker compose logs is run not from the systemd unit file but in /opt/elasticbeanstalk/config/private/eb-docker-compose-log-start. The above code was changed accordingly.

[mvanderlee]

@dennis-grunert It's more complicated than that. On the latest Linux 2 AMI (v3.6.2) you are correct but the new Linux 2023 (v4.0.1) still has the logic in /etc/systemd/system/eb-docker-compose-log.service

The new format causes an error on Linux 2023 because the file doesn't exist. This handles both cases

if [ -f /opt/elasticbeanstalk/config/private/eb-docker-compose-log-start ]; then
  # AWS Linux 2 >=v3.6.2
  sed 's/\(docker[ -]compose logs\) --since/\1 --no-log-prefix --since/' -i /opt/elasticbeanstalk/config/private/eb-docker-compose-log-start
else
  # # AWS Linux 2 <v3.6.2 && AWS Linux 2023 <=v4.0.1
  sed 's/\(docker[ -]compose logs\) -f/\1 --no-log-prefix -f/' -i /etc/systemd/system/eb-docker-compose-log.service
fi

[dennis-grunert]

@mvanderlee Thanks for the extension. I was so free to update the above script with your code and credited you. This way, others don't have to put the code pieces together after reading all the comments.