Regression in S3 Compatible APIs for Snapshot Repositories in 8.13

iveelsm commented 3 months ago

Elasticsearch Version

8.13.3

Installed Plugins

No response

Java Version

bundled

OS Version

Linux 31-prd-iad-elasticsearch 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

Problem Description

https://github.com/elastic/elasticsearch/pull/105044 introduced a regression in S3 Compatible APIs like R2. R2 does not support the use of x- parameters. Given the extremely specific nature of the x-purpose search parameter, either the documentation needs to be marked as not supporting S3 compatible services, or there needs to be another approach to address the custom log format for AWS.

Cloudflare error message (and stack trace) as an example:

[2024-06-03T23:35:14,506][ERROR][o.e.x.i.IndexLifecycleRunner] [redacted] policy [redacted] for index [redacted] failed on step [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}]. Moving to ERROR step
java.io.IOException: Exception when listing blobs by prefix [null]
    at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:356) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobs(S3BlobContainer.java:362) ~[?:?]
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$createSnapshotsDeletion$12(BlobStoreRepository.java:887) ~[elasticsearch-8.13.3.jar:?]
    at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:52) ~[elasticsearch-8.13.3.jar:?]
    at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:49) ~[elasticsearch-8.13.3.jar:?]
    at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95) ~[elasticsearch-8.13.3.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.3.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.3.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: ListObjectsV1 search parameter x-purpose not implemented (Service: Amazon S3; Status Code: 501; Error Code: NotImplemented; Request ID: null; S3 Extended Request ID: null; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$15(S3BlobContainer.java:403) ~[?:?]
    at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
    at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:403) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:347) ~[?:?]
    ... 10 more

Steps to Reproduce

Use a non-AWS S3 compatible storage.

Logs (if relevant)

No response

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner commented 3 months ago

The AWS S3 docs say specifically that S3 ignores all query-string parameters prefixed with an x-, so any service claiming to be S3-compatible must do the same.

elastic / elasticsearch