elastic / elasticsearch-cloud-aws

AWS Cloud Plugin for Elasticsearch
https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2
577 stars 181 forks source link

Snapshot creation fails with xml parse exception on AmazonS3Client.listObjects #162

Closed jloomis closed 9 years ago

jloomis commented 9 years ago

I'm running es 1.4.1 with cloud plugin 2.4.1. I'm using IAM with a policy following the one in the readme that uses a prefix. When I try to create a snapshot I get:

{"error":"AmazonClientException[Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK]; nested: AmazonClientException[Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler]; nested: SAXParseException[Premature end of file.]; ","status":500}

The full stacktrace on a given data node looks like:

2015-01-13T00:35:46.451Z WARN  [ventstore-es-data-d][snapshot][T#1]] [snapshots                           ]: [eventstore-es-data-d] [[bqdqy5oaaaa][3]] [lab-es-repository:snapshot_1] failed to create snapshot
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [bqdqy5oaaaa][3] Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$
ListBucketHandler). Response Code: 200, Response Text: OK
        at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:149)
        at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86)
        at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:818)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200
, Response Text: OK
        at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:758)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:417)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:245)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3722)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3675)
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:603)
        at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:118)
        at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobs(S3BlobContainer.java:136)
        at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:439)
        at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139)
        ... 5 more
Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
        at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:158)
        at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:293)
        at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:76)
        at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:73)
        at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
        at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
        at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:730)
        ... 15 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:145)
        ... 21 more

Any ideas on why this would happen? Seems like an s3 sdk issue but if that were the case I'd assume others would've run into it and am failing to find anything. In the bucket I wind up with a metadata-snapshot_1, a snapshot-snapshot_1 which is full references to the above error, and /indices with a file that contains mappings & settings.

jloomis commented 9 years ago

D'oh, so in case anyone else does the same thing...this happens if you mess up your repository settings and have "bucket": "mybucket/foo" rather than "bucket": "mybucket", "base_path": "foo".