AWS S3 Input custom endpoint handling broken in 8.15

[strawgate]

8.15 no longer has the fix for AWS S3 endpoint handling that was present in 8.14

8.15: https://github.com/elastic/beats/blob/2f0dda8c0d9a1bd644fcdf1ebd41f32bfc7a907d/x-pack/filebeat/input/awss3/input.go#L51-L60

8.14: https://github.com/elastic/beats/blob/7b6cfad6ad3f767ed277bd23317d63449173ec27/x-pack/filebeat/input/awss3/input.go#L83-L104

In AWS, the endpoint field is supposed to act kind of like a "base url" where service URLs are built using the value in the endpoint field. So when the SQS client makes a request, an endpoint field of s3.us-east1.amazonaws.com is transformed into sqs.us-east1.amazonaws.com, etc.

The 8.15 code forces all endpoints to use the value in the endpoint field instead of relying on the resolver to use the endpoint to "build" each service's endpoint (s3, sqs, etc). In the example above, this would cause the SQS client to directly query s3.us-east1.amazonaws.com

Even the 8.14 code has an issue that crops up with some customers. We should likely switch to only using a custom endpoint resolver when a user explicitly tells us to, for example by introducing a new setting called "static endpoint" or something similar, that when set to true, sets the endpoint resolver as it is set currently. This would be a breaking change.

An alternative would be introducing a setting called, "dynamic_endpoint" or something similar which, when set, sets the endpoint field without using a resolver.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

https://github.com/elastic/beats/pull/39709 was only in 8.14 and https://github.com/elastic/beats/pull/39722 to bring it to main was never merged or backported.

[strawgate]

#39709 was only in 8.14 and #39722 to bring it to main was never merged or backported.

39722 was not supposed to merge, it was just an example for how to include it post s3/sqs refactor, ill close the PR but add that as a comment

[predogma]

Issue was hit in version Elasticsearch 8.15.2, with at least these integrations.

• Crowdstrike (FDR) 1.42.2 section "Collect CrowdStrike Falcon Data Replicator logs (input: aws-s3) > Falcon Data Replicator logs"

• Cisco Umbrella 1.26.2. section "Collect logs from Cisco Umbrella > Cisco Umbrella Logs"

Workaround required deleting the value located in the "Endpoint" field

Are there any other integrations that we should be aware of that could be impacted?

[cmacknz]

Any integration that can use the aws-s3 input would be affected, here's a preliminary list I generated quickly:

```sh ❯ rg -g 'manifest.yml' 'aws-s3' -l --sort=path packages/amazon_security_lake/data_stream/event/manifest.yml packages/amazon_security_lake/manifest.yml packages/aws/data_stream/apigateway_logs/manifest.yml packages/aws/data_stream/cloudfront_logs/manifest.yml packages/aws/data_stream/cloudtrail/manifest.yml packages/aws/data_stream/ec2_logs/manifest.yml packages/aws/data_stream/elb_logs/manifest.yml packages/aws/data_stream/emr_logs/manifest.yml packages/aws/data_stream/firewall_logs/manifest.yml packages/aws/data_stream/guardduty/manifest.yml packages/aws/data_stream/route53_resolver_logs/manifest.yml packages/aws/data_stream/s3access/manifest.yml packages/aws/data_stream/vpcflow/manifest.yml packages/aws/data_stream/waf/manifest.yml packages/aws/manifest.yml packages/aws_bedrock/data_stream/invocation/manifest.yml packages/aws_bedrock/manifest.yml packages/aws_logs/data_stream/generic/manifest.yml packages/aws_logs/manifest.yml packages/canva/data_stream/audit/manifest.yml packages/canva/manifest.yml packages/carbon_black_cloud/data_stream/alert/manifest.yml packages/carbon_black_cloud/data_stream/alert_v7/manifest.yml packages/carbon_black_cloud/data_stream/endpoint_event/manifest.yml packages/carbon_black_cloud/data_stream/watchlist_hit/manifest.yml packages/carbon_black_cloud/manifest.yml packages/cisco_umbrella/data_stream/log/manifest.yml packages/cisco_umbrella/manifest.yml packages/cloudflare_logpush/data_stream/access_request/manifest.yml packages/cloudflare_logpush/data_stream/audit/manifest.yml packages/cloudflare_logpush/data_stream/casb/manifest.yml packages/cloudflare_logpush/data_stream/device_posture/manifest.yml packages/cloudflare_logpush/data_stream/dns/manifest.yml packages/cloudflare_logpush/data_stream/dns_firewall/manifest.yml packages/cloudflare_logpush/data_stream/firewall_event/manifest.yml packages/cloudflare_logpush/data_stream/gateway_dns/manifest.yml packages/cloudflare_logpush/data_stream/gateway_http/manifest.yml packages/cloudflare_logpush/data_stream/gateway_network/manifest.yml packages/cloudflare_logpush/data_stream/http_request/manifest.yml packages/cloudflare_logpush/data_stream/magic_ids/manifest.yml packages/cloudflare_logpush/data_stream/nel_report/manifest.yml packages/cloudflare_logpush/data_stream/network_analytics/manifest.yml packages/cloudflare_logpush/data_stream/network_session/manifest.yml packages/cloudflare_logpush/data_stream/sinkhole_http/manifest.yml packages/cloudflare_logpush/data_stream/spectrum_event/manifest.yml packages/cloudflare_logpush/data_stream/workers_trace/manifest.yml packages/cloudflare_logpush/manifest.yml packages/crowdstrike/data_stream/fdr/manifest.yml packages/crowdstrike/manifest.yml packages/f5_bigip/data_stream/log/manifest.yml packages/f5_bigip/manifest.yml packages/imperva_cloud_waf/data_stream/event/manifest.yml packages/imperva_cloud_waf/manifest.yml packages/jamf_protect/data_stream/alerts/manifest.yml packages/jamf_protect/data_stream/telemetry/manifest.yml packages/jamf_protect/data_stream/telemetry_legacy/manifest.yml packages/jamf_protect/data_stream/web_threat_events/manifest.yml packages/jamf_protect/data_stream/web_traffic_events/manifest.yml packages/jamf_protect/manifest.yml packages/lyve_cloud/data_stream/audit/manifest.yml packages/lyve_cloud/manifest.yml packages/sentinel_one_cloud_funnel/data_stream/event/manifest.yml packages/sentinel_one_cloud_funnel/manifest.yml packages/sublime_security/data_stream/audit/manifest.yml packages/sublime_security/data_stream/email_message/manifest.yml packages/sublime_security/data_stream/message_event/manifest.yml packages/sublime_security/manifest.yml packages/symantec_endpoint_security/data_stream/event/manifest.yml packages/symantec_endpoint_security/manifest.yml packages/tanium/data_stream/action_history/manifest.yml packages/tanium/data_stream/client_status/manifest.yml packages/tanium/data_stream/discover/manifest.yml packages/tanium/data_stream/endpoint_config/manifest.yml packages/tanium/data_stream/reporting/manifest.yml packages/tanium/data_stream/threat_response/manifest.yml packages/tanium/manifest.yml packages/trellix_edr_cloud/data_stream/event/manifest.yml packages/trellix_edr_cloud/manifest.yml ```