search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.26k stars 24.86k forks source link

[CI] S3SnapshotRepoTestKitIT testRepositoryAnalysis failing #102294

Closed kkrik-es closed 1 week ago

kkrik-es commented 1 year ago

Build scan: https://gradle-enterprise.elastic.co/s/d3hlyfatxvzxc/tests/:x-pack:plugin:snapshot-repo-test-kit:qa:s3:javaRestTest/org.elasticsearch.repositories.blobstore.testkit.S3SnapshotRepoTestKitIT/testRepositoryAnalysis Reproduction line:

./gradlew ':x-pack:plugin:snapshot-repo-test-kit:qa:s3:javaRestTest' --tests "org.elasticsearch.repositories.blobstore.testkit.S3SnapshotRepoTestKitIT.testRepositoryAnalysis" -Dtests.seed=1D93A9E8DAF6B824 -Dtests.locale=pl -Dtests.timezone=Pacific/Fiji -Druntime.java=21

Applicable branches: main

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.repositories.blobstore.testkit.S3SnapshotRepoTestKitIT&tests.test=testRepositoryAnalysis

Failure excerpt:

org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:40137], URI [/_snapshot/repository/_analyze?blob_count=10&seed=-5647455207957961422&max_blob_size=1mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"amazon_s3_exception","reason":"The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: T1W7Z1C34PV9DJ6F; S3 Extended Request ID: iGbxqm+wJy3A9IlT0Bv/h4ElYIVCwbj9St10IiRQ6MSJxmLudmUKV3OBZT3Y9ACPHYCUkwZQ9sU=; Proxy: null)"}],"type":"repository_verification_exception","reason":"[repository] analysis failed, you may need to manually remove [temp-analysis-sxfXgSufR5SOp3rmqPP1Ag]","caused_by":{"type":"amazon_s3_exception","reason":"The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: T1W7Z1C34PV9DJ6F; S3 Extended Request ID: iGbxqm+wJy3A9IlT0Bv/h4ElYIVCwbj9St10IiRQ6MSJxmLudmUKV3OBZT3Y9ACPHYCUkwZQ9sU=; Proxy: null)"}},"status":500}

  at __randomizedtesting.SeedInfo.seed([1D93A9E8DAF6B824:E4AE0886046D8BCD]:0)
  at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
  at org.elasticsearch.repositories.blobstore.testkit.AbstractSnapshotRepoTestKitRestTestCase.testRepositoryAnalysis(AbstractSnapshotRepoTestKitRestTestCase.java:35)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner commented 1 year ago

Hmm there's not much to go on here. I opened #102299 to make things more verbose.

DaveCTurner commented 1 year ago

I realised that the logs in the build scan don't include the node logs, which contain a little more detail:

[2023-11-16T13:11:32,130][WARN ][r.suppressed             ] [javaRestTest-0] path: /_snapshot/repository/_analyze, params: {seed=-5647455207957961422, max_blob_size=1mb, blob_count=10, repository=repository, timeout=120s, concurrency=4}, status: 500
org.elasticsearch.repositories.RepositoryVerificationException: [repository] analysis failed, you may need to manually remove [temp-analysis-sxfXgSufR5SOp3rmqPP1Ag]
Caused by: org.elasticsearch.transport.RemoteTransportException: [javaRestTest-0][127.0.0.1:40079][cluster:admin/repository/analyze/register]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: T1W7Z1C34PV9DJ6F; S3 Extended Request ID: iGbxqm+wJy3A9IlT0Bv/h4ElYIVCwbj9St10IiRQ6MSJxmLudmUKV3OBZT3Y9ACPHYCUkwZQ9sU=; Proxy: null)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853) ~[?:?]
    at com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52) ~[?:?]
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:618) ~[?:?]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1728) ~[?:?]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2899) ~[?:?]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605) ~[?:?]
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:114) ~[?:?]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542) ~[?:?]
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889) ~[?:?]
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825) ~[?:?]
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[?:?]
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224) ~[?:?]
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167) ~[?:?]
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502) ~[?:?]
    at com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316) ~[?:?]
    at com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313) ~[?:?]
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62) ~[?:?]
    at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44) ~[?:?]
    at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30) ~[?:?]
    at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$12(S3BlobContainer.java:815) ~[?:?]
    at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46) ~[?:?]
    at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
    at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:815) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:626) ~[?:?]
    at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:299) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$3(S3BlobContainer.java:624) ~[?:?]
    at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:173) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:305) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:846) ~[?:?]
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:620) ~[?:?]
    at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:173) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:331) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:121) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:117) ~[elasticsearch-core-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:69) ~[elasticsearch-core-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:116) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.core.Releasables$3.close(Releasables.java:142) ~[elasticsearch-core-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:151) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:146) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:36) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0-SNAPSHOT.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
DaveCTurner commented 1 year ago

Hmm Status Code: 0; should be the HTTP status code received from S3, but 0 is not a valid status code.

DaveCTurner commented 1 year ago

I've tried various ideas with our own simulated S3 test fixture to see if I can reproduce the Status Code: 0 seen in the exception message, but without success. I'm enabling wire logging (see #102331) and will then have to wait for this to happen again.

ywangd commented 4 months ago

@DaveCTurner I'd like check with you whether we can close this issue. It has not seen any failure for last 6 months. I think at this point it's OK to view it as not actionable and hence closing. I also think it is low-risk since it feels more infrastructureish (either our s3 fixuture or java's http server).

DaveCTurner commented 4 months ago

I believe this failure came from a test running against the real S3, not our own infra, so I remain concerned that there is some surprising real-life behaviour that we're not handling correctly.

ywangd commented 4 months ago

I see this in the console log

org.elasticsearch.repositories.blobstore.testkit.S3SnapshotRepoTestKitIT > testRepositoryAnalysis FAILED org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:40137], URI [/_snapshot/repository/_analyze?blob_count=10&seed=-5647455207957961422&max_blob_size=1mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]

Note the host is http://127.0.0.1:40137. So I don't think it is real s3?

ywangd commented 4 months ago

Nevermind. The above is host is for the elasticsearch API and the host is the ES node

metadaddy commented 2 weeks ago

I just encountered this issue, albeit using Backblaze B2, rather than Amazon S3. Running the analyzer with the default parameters given in the docs:

curl -X POST "localhost:9200/_snapshot/my_s3_repository/_analyze?blob_count=10&max_blob_size=1mb&timeout=120s&pretty"

yielded a similar No active upload for: <uploadId> error. After enabling the HTTP wire trace, and carefully analyzing the output, the issue seems to be in this sequence of operations:

I wrote a small Python app using aiobotocore to more easily replicate the sequence and compare Backblaze B2 against Amazon S3: https://gist.github.com/metadaddy/5658329ac44034ffed9b54d7fa5906ab

Running the app against Backblaze B2 results in around 97 failures out of 100 iterations, while running the app against S3 results in around 3 failures for the same number of iterations, enough to cause sporadic failures of both the Repository Analyzer and real-world snapshot creation.

Looking at the Amazon S3 docs for AbortMultipartUpload, it says:

[…] if any part uploads are currently in progress, those part uploads might or might not succeed.

The Repository Analyzer seems to assume that, when you abort a multipart upload, inflight part uploads will succeed, but this is in no way guaranteed by the S3 API.

I embarked on this investigation in response to a mutual customer encountering issues creating a snapshot on Backblaze B2. I'm guessing that the snapshot creation process behaves in the same way as the Repository Analyzer.

I think the fix would be to ignore the error return from UploadPart if the upload has been aborted. The analyzer code is pretty complex, though, so I could be way off!

DaveCTurner commented 2 weeks ago

Thanks for the input @metadaddy but that doesn't sound related. This test failure was because we got a Status Code: 0 response from S3, whereas you are talking about an unexpected 400 response. I believe we handle this scenario correctly with AWS S3, but if you can reproduce such a failure with AWS S3 then please share the output from the POST /_snapshot/my_s3_repository/_analyze in a fresh issue. Ideally add the ?error_trace=true query parameter so we get the full stack trace in the response.

metadaddy commented 2 weeks ago

Hi @DaveCTurner - Status Code: 0 is a weird one - I don't know how the SDK comes up with that!

Regardless, the error descriptions from the two services are semantically identical:

I'll run the analyzer in a loop against S3 for a while with ?error_trace=true and see if it results in a similar error.

metadaddy commented 2 weeks ago

Hi again, @DaveCTurner - I reproduced the original Status Code: 0 error in this issue running the repository analysis against Amazon S3, and I also figured out why it's reported as 0.

I hope this is helpful and on-topic. Let me know if it should actually be a new issue.

Setting up the keystore:

export ES_PATH_CONF=/Users/ppatterson/src/elasticsearch/build/testclusters/runTask-0/config
echo ${AWS_ACCESS_KEY_ID} | bin/elasticsearch-keystore add --stdin s3.client.s3-client.access_key -f
echo ${AWS_SECRET_ACCESS_KEY} | bin/elasticsearch-keystore add --stdin s3.client.s3-client.secret_key -f
bin/elasticsearch-keystore list
curl -X POST "localhost:9200/_nodes/reload_secure_settings?pretty" -H 'Content-Type: application/json' -d '{}'

Creating the S3 repository:

curl -X PUT "localhost:9200/_snapshot/my_s3_repository?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "metadaddy-elasticsearch2",
    "client": "s3-client",
    "region": "us-west-1",
    "endpoint": "s3.us-west-1.amazonaws.com"
  }
}
'

I wrote a script to repeatedly hit the Repository Analysis API until it encountered an error:

#! /bin/zsh

# Pass repo name as script argument
repo=$1
outfile=${repo}_output.txt
while true
do
    curl -s --fail-with-body -X POST "localhost:9200/_snapshot/${repo}/_analyze?blob_count=10&max_blob_size=1mb&timeout=120s&pretty&error_trace=true" &>> ${outfile}
    if [ $? -ne 0 ]
    then
        # Request failed
        break
    fi
    sleep 1
done

Eventually, the script terminated. Error response:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "amazon_s3_exception",
        "reason" : "The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null)",
        "stack_trace" : "com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)\n\tat com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)\n\tat com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)\n\tat com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/repo-analysis-api.html] for further information about repository analysis, and [https://www.elastic.co/guide/en/elasticsearch/reference/master/repository-s3.html#repository-s3-compatible-services] for further information about support for S3-compatible repository implementations. Elasticsearch attempted to remove the data it wrote at [temp-analysis-yBmseJExSNiitdirP1iyaw] but may have left some behind. If so, please now remove this data manually.",
    "caused_by" : {
      "type" : "amazon_s3_exception",
      "reason" : "The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null)",
      "stack_trace" : "com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)\n\tat com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)\n\tat com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)\n\tat com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"
    },
    "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [my_s3_repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/repo-analysis-api.html] for further information about repository analysis, and [https://www.elastic.co/guide/en/elasticsearch/reference/master/repository-s3.html#repository-s3-compatible-services] for further information about support for S3-compatible repository implementations. Elasticsearch attempted to remove the data it wrote at [temp-analysis-yBmseJExSNiitdirP1iyaw] but may have left some behind. If so, please now remove this data manually.\nCaused by: org.elasticsearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][cluster:admin/repository/analyze/register]\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)\n\tat com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)\n\tat com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)\n\tat com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"
  },
  "status" : 500
}

Here's the stack trace in a more digestible form (the three traces in the response are identical):

org.elasticsearch.repositories.RepositoryVerificationException: [my_s3_repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/repo-analysis-api.html] for further information about repository analysis, and [https://www.elastic.co/guide/en/elasticsearch/reference/master/repository-s3.html#repository-s3-compatible-services] for further information about support for S3-compatible repository implementations. Elasticsearch attempted to remove the data it wrote at [temp-analysis-yBmseJExSNiitdirP1iyaw] but may have left some behind. If so, please now remove this data manually.
Caused by: org.elasticsearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][cluster:admin/repository/analyze/register]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)
    at com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)
    at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)
    at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)
    at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)
    at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)
    at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)
    at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)
    at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
    at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)
    at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
    at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)
    at org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)
    at org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)
    at org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)
    at org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)
    at org.elasticsearch.base@9.0.0-SNAPSHOT/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)
    at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1570)

This is actually a different error from the one I was seeing with Backblaze B2 in that it occurs when CompleteMultipartUpload reports a "no active upload" error. The error I saw was related to UploadPart.

Why is the status code 0?

Looking at the stack trace, we can see that the operation being processed is CompleteMultipartUpload. We can also see that the response from S3 has HTTP status code 200, despite it actually containing an error, evidenced by the call to RequestExecutor.handleSuccessResponse(). This behavior is explained in the docs for CompleteMultipartUpload:

The processing of a CompleteMultipartUpload request could take several minutes to finalize. After Amazon S3 begins processing the request, it sends an HTTP response header that specifies a 200 OK response. While processing is in progress, Amazon S3 periodically sends white space characters to keep the connection from timing out. A request could fail after the initial 200 OK response has been sent. This means that a 200 OK response can contain either a success or an error. The error response might be embedded in the 200 OK response. If you call this API operation directly, make sure to design your application to parse the contents of the response and handle it appropriately. If you use AWS SDKs, SDKs handle this condition. The SDKs detect the embedded error and apply error handling per your configuration settings (including automatically retrying the request as appropriate). If the condition persists, the SDKs throw an exception (or, for the SDKs that don't use exceptions, they return an error).

At the top of the stack trace we see CompleteMultipartUploadHandler.doEndElement(). Looking at the source code, this method does not set the status code in the exception at all, so it keeps its default value of 0.

DaveCTurner commented 2 weeks ago

Aha that does explain it indeed. Thanks for the thorough investigation and writeup. I opened https://github.com/elastic/elasticsearch/pull/116212 to address this.

metadaddy commented 1 week ago

For the record, the root cause of the repository analyzer's failure with Backblaze B2 is that B2's UploadPart endpoint returns 400 for an invalid UploadId, rather than 404, as Amazon S3 does, and the S3 repository code expects. I verified this by tweaking S3BlobContainer.compareAndExchangeRegister() to handle the 400 the same as it does the 404 - this allowed the repository analysis to succeed with B2. I've opened a ticket with our engineering team to fix it in Backblaze B2.

Thanks for your patience as I got to the bottom of this, @DaveCTurner, and thanks for creating an awesome tool to check compatibility!

ywangd commented 1 week ago

@metadaddy Thanks for the great Investigation! TIL. I wonder whether it is worthwhile to also report this as a bug for aws-sdk-java. While the v1 client is on its way out, it seems they are still taking issue reports.

metadaddy commented 1 week ago

Hi @ywangd - I'm not sure it's a bug in aws-sdk-java so much as a quirk of the S3 API.