search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
95 stars 4.92k forks source link

[aws]Add support for access point in aws integration #41494

Open narph opened 1 month ago

narph commented 1 month ago

Trying to use an access point as the bucket ARN in the S3 integration triggers the following permission issue:

expected: arn:aws:s3:::some-name-here Access Point: arn:aws:s3:us-east-....[REDACTED]/[REDACTED]

Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket: operation error S3: GetBucketLocation, https response error StatusCode: 403, RequestID: [REDACTED], HostID: [REDACTED], api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. The aws integration does not expect the Access Point (https://aws.amazon.com/s3/features/access-points/) format in the bucket ARN parameter atm.

elasticmachine commented 1 month ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

chemamartinez commented 1 week ago

Transferred to Beats repo as the fix is entirely attempted to be in the input side.

Already opened a draft PR with the proposed fix: https://github.com/elastic/beats/pull/41495. Although I am still working on an issue with the bucket region when reading from the bucket:

{"log.level":"warn","@timestamp":"2024-10-31T13:13:13.638+0100","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3PollerInput).readerLoop","file.name":"awss3/s3_input.go","file.line":198},"message":"Error when paginating listing.","service.name":"filebeat","id":"[REDACTED]","error":{"message":"operation error S3: ListObjectsV2, https response error StatusCode: 400, RequestID: [REDACTED], HostID: [REDACTED], api error AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'"},"ecs.version":"1.6.0"}

After fixing it and adding tests, it should be ready.

The current solution uses the current Bucket ARN config option to support Access Points. The advantage of this approach is that no changes are needed at integrations to get this working, apart from extending the documentation to let users know that Access Point ARN are also supported.

On the other hand, introducing Access Point ARN as a separate config option might be more self-explanatory for users.