Add easy way to forward custom logs to CloudWatch

[bzuillsmith]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave "+1" or "me too" comments. They generate extra noise for issue followers and do not help prioritize the request.

Tell us about your request The new Amazon Linux 2023 platform does not have a clear way to forward custom logs to CloudWatch when used with EB. I have attempted changing the CloudWatch Agent configuration file of the but it gets overwritten by the EB startup script after config files and the prebuild hook run.

Either document a clear example of how, or give us a simple config or hook to specify custom log file paths.

The docs say we can have multiple apps/processes run on instances, but there should be easier ways to make logging work for all of them, not just the one identified as "web" in the Procfile.

I so far have not found a solution on the 2023 platform.

[arifb]

Can someone from AWS chime in here? Is this not a basic requirement for any production server, to be able to stream the logs to cloudwatch?

Previously on Amazon Linux 2, we used the following ebextension, which is a sample provided by AWS, but it does not work on AL2023

https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-streamtocloudwatch-linux.config

Thanks.

[simonweil]

Have you tried using aws cloudwatch agent for it?

[arifb]

Hey @simonweil, thanks. I haven't but I know that's what I need to do and it's on my list, hopefully this week. Just surprised there is no documentation or examples, like there was for AL2.

[simonweil]

I have not yet migrated to AL2023, but I'd expect the cloudwatch agent to work the same...

[arifb]

They've deprecated the CloudWatch Logs agent and expect you to move to the new unified agent, you can see this here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

Their docs / examples haven't been updated to use the new agent, for example https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-streamtocloudwatch-linux.config

[cybercussion]

This is getting quite confusing. You flip log streams on, try to add custom logs now and it steps on the whole config. New setup seems to create a beanstalk.json file if you turn log streams on, yet if I do a custom app log(s) then either try to append the config or fetch I seem to either end up with a config merge conflict or erase log streams and just end up with my custom app logs. Someone needs to write up better docs on this with examples.

[bananastalktome]

I don't know if this is "officially supported", but I have had success in AmazonLinux 2023 adding our own config to the CloudWatch Agent already configured/running as part of beanstalk. For example, to add an additional log group, I use the following in an .ebextensions file:

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/some_log_name_config.json":
    mode: "0644"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/some_log_name.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/some_log_name.log"]]}`",
                  "log_stream_name": "{instance_id}",
                  "retention_in_days": 3
                }
              ]
            }
          }
        }
      }
container_commands:
  append_some_log_name_cloudwatch_logs:
    command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/some_log_name_config.json
  remove_some_log_name_cloudwatch_logs_backup:
    command: "rm -f /opt/aws/amazon-cloudwatch-agent/etc/some_log_name_config.json.bak"
    ignoreErrors: true

We also add our own metrics collection, just replacing/adding to the content section of our above config using https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html as reference.

I wish the documentation at https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html#AWSHowTo.cloudwatchlogs.streaming.custom was updated to include info about the unified agent rather than still referencing the deprecated logs agent.

[cybercussion]

I don't know if this is "officially supported", but I have had success in AmazonLinux 2023 adding our own config to the CloudWatch Agent already configured/running as part of beanstalk. For example, to add an additional log group, I use the following in an .ebextensions file:

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/some_log_name_config.json":
    mode: "0644"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/some_log_name.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/some_log_name.log"]]}`",
                  "log_stream_name": "{instance_id}",
                  "retention_in_days": 3
                }
              ]
            }
          }
        }
      }
container_commands:
  append_some_log_name_cloudwatch_logs:
    command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/some_log_name_config.json
  remove_some_log_name_cloudwatch_logs_backup:
    command: "rm -f /opt/aws/amazon-cloudwatch-agent/etc/some_log_name_config.json.bak"
    ignoreErrors: true

We also add our own metrics collection, just replacing/adding to the content section of our above config using https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html as reference.

I wish the documentation at https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html#AWSHowTo.cloudwatchlogs.streaming.custom was updated to include info about the unified agent rather than still referencing the deprecated logs agent.

Thanks, this is almost identical with what I did. I had AL2 giving me a /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default file which conflicted with some of the force_flush_interval settings it had. It didn’t exhibit this on AL2023 but also moved to append config and a line to just remove the default file so they would no longer have config merge conflict.

[praegt]

Thanks for bringing this to our attention and sharing a workaround. We've updated the example ebextension and the doc page to reference the unified CloudWatch agent. Please let us know if you have other issues or suggestions.