files key in .ebextensions is ignored

[RaffaeleSgarro]

I need to use the old .ebextensions in order to create a CloudWatch rule that streams a file to a log group named after the Beanstalk environment name.

It seems that the content of .ebextensions (a single .config file attached below) is simply ignored:

# The structure of this file is documented here:
#   https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-commands
files:
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/webapp.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
             "files": {
               "collect_list": [
                 {
                    "file_path": "/home/webapp/logs/webapp.log",
                    "log_group_name": "/aws/elasticbeanstalk/my-env/home/webapp/logs/webapp.log",
                    "log_stream_name": "{instance_id}",
                    "timestamp_format": "%d-%m-%Y %H:%M:%S.%f",
                    "multi_line_start_pattern": "{timestamp_format}"
                 }
               ]
             }
          }
        }
      }
container_commands:
  01_restart_cloudwatch_agent:
    command: systemctl restart amazon-cloudwatch-agent.service

Expected behavior: connect with SSH and find a file named /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/webapp.json

Actual behavior: file /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/webapp.json does not exist

This is an example Java application that you can use to reproduce the issue: https://github.com/RaffaeleSgarro/cloudwatch-multiline-log-messages

[zzzhu0192]

Hi,

with the .ebextensions above, customers should see this file /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/webapp.json on their ec2 instance. The file customer mentioned above /aws/elasticbeanstalk/my-env/home/webapp/logs/webapp.log is the log_group_name, which should be seen in cloudwatch console.

Thanks

[RaffaeleSgarro]

@zzzhu0192 you are right, there was an error in the description. The description is fixed now, but the bug is still there.

[zzzhu0192]

okay. May I ask which platform version is used ?

I took a look at your repo and saw that you were using platform hooks feature to create the file. Just to clarify, platform hooks worked in this case but NOT with ebextensions. Is that the case ?

[RaffaeleSgarro]

I use the Java Corretto 11 platform. In the end I solved it with platform hooks, but the Beanstalk documentation states that .ebextension is still supported and that appears to not be the case.

[zzzhu0192]

Hi,

I am unable to reproduce the issue. Could you provide more details ? Bundle logs would be helpful.

ebextension

cat .ebextensions/s3.config
files:
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/webapp.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
             "files": {
               "collect_list": [
                 {
                    "file_path": "/var/log/message.log",
                    "log_group_name": "ecsdemo-crystal",
                    "log_stream_name": "{instance_id}",
                    "timestamp_format": "%d-%m-%Y %H:%M:%S.%f",
                    "multi_line_start_pattern": "{timestamp_format}"
                 }
               ]
             }
          }
        }
      }
container_commands:
  01_restart_cloudwatch_agent:
    command: systemctl restart amazon-cloudwatch-agent.service%

file is created successfully on instance

[root@ip-172-31-47-109 amazon-cloudwatch-agent.d]# cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/webapp.json
{
  "logs": {
    "logs_collected": {
       "files": {
         "collect_list": [
           {
              "file_path": "/var/log/message.log",
              "log_group_name": "ecsdemo-crystal",
              "log_stream_name": "{instance_id}",
              "timestamp_format": "%d-%m-%Y %H:%M:%S.%f",
              "multi_line_start_pattern": "{timestamp_format}"
           }
         ]
       }
    }
  }
}

Thanks

[dgagnon]

I had some of the same issue. turned out to be the aws cloudwatch agent version. There are different ones and the one being installed in AL1 is not the same as AL2. Just putting the files in the folder is not enough, you got to run some commands.

see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html

[RaffaeleSgarro]

@zzzhu0192 unfortunately bundle logs are gone because I deleted the demo app and environment. We could reproduce it in all of our Beanstalk environments and in the one created for testing purposes. May I ask if you use exactly the provided repository in a Java Corretto 11 environment?

Anyway we finally solved it with a shell script executed as a platform hook that creates the CloudWatch config file. Everything works and the demo repository has been updated with the solution:

https://github.com/RaffaeleSgarro/cloudwatch-multiline-log-messages

[zzzhu0192]

Hi,

not yet. I only used the ebextension above. I will use the source bundle from this repo to give it another shot.