search

aws / aws-sdk-java

The official AWS SDK for Java 1.x. The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/
https://aws.amazon.com/sdkforjava
Apache License 2.0
4.13k stars 2.83k forks source link

Can't list buckets #895

Closed IRus closed 8 years ago

IRus commented 8 years ago

Hi, i tried to list all buckets using AmazonS3Client, and run in following error:

Exception in thread "main" com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListAllMyBucketsHandler
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:128)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListMyBucketsResponse(XmlResponsesSaxParser.java:320)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListBucketsUnmarshaller.unmarshall(Unmarshallers.java:38)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListBucketsUnmarshaller.unmarshall(Unmarshallers.java:34)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
    at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1428)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1135)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4039)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3976)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:867)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:873)
    at com.company.recruiting.commands.deploy.AmazonDeploy.listBuckets(AmazonDeploy.kt:213)
    at com.company.recruiting.commands.deploy.AmazonCommand.run(AmazonCommand.java:85)
    at io.bootique.run.DefaultRunner.run(DefaultRunner.java:20)
    at io.bootique.Bootique.run(Bootique.java:377)
    at io.bootique.Bootique.run(Bootique.java:355)
    at com.company.recruiting.Application.main(Application.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: org.xml.sax.SAXParseException; lineNumber: 164; columnNumber: 90; The entity "ntilde" was referenced, but not declared.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1465)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1896)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3058)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:504)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:114)
    ... 30 more

Dependency: "com.amazonaws:aws-java-sdk-s3:1.11.40"

IRus commented 8 years ago

Related https://github.com/aws/aws-sdk-java/issues/460

varunnvs92 commented 8 years ago

Do you see this error every time you call listBuckets() or occasionally?

Is it possible for you to provide the wire log when this error occurs. Instructions to enable wire logging: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html

IRus commented 8 years ago

Awww. I found that i inject bean AmazonS3Client instead of AmazonS3 and my DI in this case creates bean via default constructor, so credentials doesn't passed in constructor.

But anyway, error quite strange, amazon returned HTTP 200 OK and HTML page instead of 40x:

2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "GET /s3/ HTTP/1.1[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "Host: aws.amazon.com[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "User-Agent: aws-sdk-java/1.11.43 Linux/4.4.0-43-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.101-b13/1.8.0_101[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "amz-sdk-invocation-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "amz-sdk-retry: 1/0/500[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "Content-Type: application/octet-stream[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
2016-10-18 11:36:16,598 [main] DEBUG org.apache.http.wire -  http-outgoing-1 >> "[\r][\n]"
2016-10-18 11:36:16,855 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
2016-10-18 11:36:16,856 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Server: Server[\r][\n]"
2016-10-18 11:36:16,856 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Date: Tue, 18 Oct 2016 08:36:16 GMT[\r][\n]"
2016-10-18 11:36:16,856 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Content-Type: text/html;charset=UTF-8[\r][\n]"
2016-10-18 11:36:16,856 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Transfer-Encoding: chunked[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Connection: keep-alive[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "X-Frame-Options: SAMEORIGIN[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "X-Content-Type-Options: nosniff[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "x-amz-id-1: xxxxxxxxxxxxxxxxxxxxxxxx[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Last-Modified: Fri, 14 Oct 2016 20:21:28 GMT[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Vary: Accept-Encoding,User-Agent[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Access-Control-Allow-Origin: http://aws.amazon.com[\r][\n]"
2016-10-18 11:36:16,859 [main] DEBUG org.apache.http.wire -  http-outgoing-1 << "Set-Cookie: aws_lang=en; Domain=.amazon.com; Path=/[\r][\n]"

There are a lot of output, and i'm not sure if it is secure to put it all here.

varunnvs92 commented 8 years ago

Don't put any sensitive information here.

1) Do you have any buckets that have Non–DNS-Compliant Bucket Names? https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html 2) How often do you see the error? 3) In the log, you have removed the request ids. Please give me x-amz-request-id and x-amz-id-2 so that service team can look at their logs on why the error happened. This information is not sensitive, so feel free to post it here.

IRus commented 8 years ago

Ah, I'm not native speaker, so looks like you doesn't got what i say :)

0. Problem caused by not passing credentials nor via ProfileCredentialsProvider, nor via any other Provider in AWSCredentialsProviderChain. This is cause of problems with API, after i pass credentials properly - i become able to list buckets, etc. I expected some sort of validation here, but looks like it's not possible because anonymous access existing, and without breaking changes in sdk java api we can't do this. 1. No, all buckets names valid according documentation. 2. Every time, when i don't pass credentials :) 3. "Please give me x-amz-request-id and x-amz-id-2" I doesn't think that this is necessary, because obviously this is problem on my side :)

shorea commented 8 years ago

"Problem caused by not passing credentials nor via ProfileCredentialsProvider, nor via any other Provider in AWSCredentialsProviderChain. This is cause of problems with API, after i pass credentials properly - i become able to list buckets, etc. I expected some sort of validation here, but looks like it's not possible because anonymous access existing, and without breaking changes in sdk java api we can't do this. "

The client builder will give you this validation, it does not fallback to anonymous access. https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3ClientBuilder.java

IRus commented 8 years ago

@shorea i used it :) Problem was in DI, that create instance of AmazonS3 but I injected AmazonS3Client. And my DI framework (Guice) doesn't found and instance of AmazonS3Client and silently injected newly created AmazonS3Client via default constuctor -> in anonymous mode.

shorea commented 8 years ago

Oh I see. Darn you autowire!

Shivamx20167725 commented 3 years ago

You should add IAM role in s3. This will solve your problem