[AWS] Support for reading Cloudwatch logs in Cross Account Observability Setup

[aydasraf]

Stack Info ES + ESAg: 8.8.0 AWS Integration: 1.42.0

Summary We are using the AWS integration for various AWS services, we mainly use the collect logs from CloudWatch option. This has been working so far so good as long as the cloudwatch log group is in the same account as the elastic agent.

Recently we moved to the CloudWatch cross-account observability which allows us to have centralised monitoring account where we can access the cloudwatch logs and metrics from all other accounts. We started centralising our elastic agent installation as well. Sadly we discovered that the current implementation the integration use to query cloudwatch logs is not optimal as it always uses the logGroupName to request data using the FilterLogEvents API whether you use Log Group ARN or Log Group Name which not ideal , as it this approach assumes that the cloudwatch log group is hosted in the same account - preventing any cross account calls.

This is the error we get when providing Log Group ARN for cloudwatch log that is in different account:

{"log.level":"error","@timestamp":"2023-06-19T13:43:27.827Z","message":"getLogEventsFromCloudWatch failed: error FilterLogEvents with Paginator: operation error CloudWatch Logs: FilterLogEvents, https response error StatusCode: 400, RequestID: d2e6e790-0224-4d6f-91e9-a9cb8d3b1c50, ResourceNotFoundException: The specified log group does not exist.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"aws-cloudwatch-3eefa390-8d17-11ed-a61c-2352394b4548","type":"aws-cloudwatch"},"log":{"source":"aws-cloudwatch-3eefa390-8d17-11ed-a61c-2352394b4548"},"log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.line":66,"file.name":"awscloudwatch/cloudwatch.go"},"service.name":"filebeat","id":"aws-cloudwatch-aws.route53_resolver_logs-1d5ef4d0-7797-4d7c-b6ee-0b32510f3d1a","ecs.version":"1.6.0","ecs.version":"1.6.0"}

This is the payload we see in cloudtrail:

{
  "logGroupName": "/aws/xxxxx/yyyyyyy",
  "startTime": 1687195032410,
  "endTime": 1687195092411,
  "unmask": false
}

This should be as follow instead (since Log Group ARN was supplied)

{
 "logGroupIdentifier": "arn:aws:logs:us-east-1:123456789012:log-group:abc-logGroup-1234:*"
  "startTime": 1687195032410,
  "endTime": 1687195092411,
  "unmask": false
}

What needs to be done

1. if the Log Group ARN is provided in the integration inputs, then the call must use the provided value and send a payload with logGroupIdentifier this will allow cross account access.
2. if the logGroupName the current logic stands and we can use the provided value and send the payload that contains logGroupName

[tommyers-elastic]

sorry we missed this one - i opened the following issue in the beats repo where this data collection code is implemented https://github.com/elastic/beats/issues/36642.

[hesingon]

May I check when can this PR be done? waiting for this feature.. @tommyers-elastic