I'm running es 1.4.1 with cloud plugin 2.4.1. I'm using IAM with a policy following the one in the readme that uses a prefix. When I try to create a snapshot I get:
{"error":"AmazonClientException[Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK]; nested: AmazonClientException[Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler]; nested: SAXParseException[Premature end of file.]; ","status":500}
The full stacktrace on a given data node looks like:
2015-01-13T00:35:46.451Z WARN [ventstore-es-data-d][snapshot][T#1]] [snapshots ]: [eventstore-es-data-d] [[bqdqy5oaaaa][3]] [lab-es-repository:snapshot_1] failed to create snapshot
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [bqdqy5oaaaa][3] Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$
ListBucketHandler). Response Code: 200, Response Text: OK
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:149)
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86)
at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:818)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200
, Response Text: OK
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:758)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:417)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:245)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3722)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3675)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:603)
at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:118)
at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobs(S3BlobContainer.java:136)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:439)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139)
... 5 more
Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:158)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:293)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:76)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:73)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:730)
... 15 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:145)
... 21 more
Any ideas on why this would happen? Seems like an s3 sdk issue but if that were the case I'd assume others would've run into it and am failing to find anything. In the bucket I wind up with a metadata-snapshot_1, a snapshot-snapshot_1 which is full references to the above error, and /indices with a file that contains mappings & settings.
D'oh, so in case anyone else does the same thing...this happens if you mess up your repository settings and have "bucket": "mybucket/foo" rather than "bucket": "mybucket", "base_path": "foo".
I'm running es 1.4.1 with cloud plugin 2.4.1. I'm using IAM with a policy following the one in the readme that uses a prefix. When I try to create a snapshot I get:
The full stacktrace on a given data node looks like:
Any ideas on why this would happen? Seems like an s3 sdk issue but if that were the case I'd assume others would've run into it and am failing to find anything. In the bucket I wind up with a metadata-snapshot_1, a snapshot-snapshot_1 which is full references to the above error, and /indices with a file that contains mappings & settings.