Amazon CloudSearch upload fails with 1.11.95, works with 1.11.93

[sbailliez]

Cannot provide a full testcase yet as I had to quickly revert the code but something makes the cloudsearch server crash with 1.11.95 and the server replies with a 500 hard and a text/plain message, given it works with 1.11.93, it could be related to some json refactoring present in 1.11.94 ?

com.amazonaws.protocol.json.JsonContent: Unable to parse HTTP response content
! com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Internal': was expecting 'null', 'true', 'false' or NaN  
! at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
! at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3464)
! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3442)
! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._matchToken(UTF8StreamJsonParser.java:2766)
! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2612)
! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:854)
! at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
! at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
! at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
! at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2355)
! at com.amazonaws.protocol.json.JsonContent.parseJsonContent(JsonContent.java:72)
! at com.amazonaws.protocol.json.JsonContent.<init>(JsonContent.java:64)
! at com.amazonaws.protocol.json.JsonContent.createJsonContent(JsonContent.java:54)
! at com.amazonaws.http.JsonErrorResponseHandler.handle(JsonErrorResponseHandler.java:61)
! at com.amazonaws.http.JsonErrorResponseHandler.handle(JsonErrorResponseHandler.java:33)
! at com.amazonaws.http.AwsErrorResponseHandler.handleAse(AwsErrorResponseHandler.java:50)
! at com.amazonaws.http.AwsErrorResponseHandler.handle(AwsErrorResponseHandler.java:38)
! at com.amazonaws.http.AwsErrorResponseHandler.handle(AwsErrorResponseHandler.java:24)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1561)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1249)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
! at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
! at com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient.doInvoke(AmazonCloudSearchDomainClient.java:507)
! at com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient.invoke(AmazonCloudSearchDomainClient.java:483)
! at com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient.uploadDocuments(AmazonCloudSearchDomainClient.java:447)

Cannot paste the content of the json being sent but the answer I get is.

16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "HTTP/1.1 500 Server Error[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "Content-Type: text/plain[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "Date: Fri, 24 Feb 2017 21:23:00 GMT[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "x-amzn-RequestId: 6b0cea86-fad7-11e6-919e-2b1758bca830[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "Content-Length: 21[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "Connection: keep-alive[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "[\r][\n]"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.wire: http-outgoing-7 << "Internal Server Error"  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.headers: http-outgoing-7 << HTTP/1.1 500 Server Error  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.headers: http-outgoing-7 << Content-Type: text/plain  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.headers: http-outgoing-7 << Date: Fri, 24 Feb 2017 21:23:00 GMT  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.headers: http-outgoing-7 << x-amzn-RequestId: 6b0cea86-fad7-11e6-919e-2b1758bca830  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.headers: http-outgoing-7 << Content-Length: 21  
16:23:01.954   DEBUG [2017-02-24 21:23:00,277] org.apache.http.headers: http-outgoing-7 << Connection: keep-alive  

[shorea]

taking a look

[shorea]

I think I see the bug. What content type are you setting? Seems we may be blowing it away with the latest refactoring.

[sbailliez]

Was just looking at the request log in details, apparently content-type is not set

16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "POST /2013-01-01/documents/batch?format=sdk HTTP/1.1[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "Host: xxxxxxxxxxx-k4twi2hvwnwandapy4dho4435q.us-east-1.cloudsearch.amazonaws.com[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXX/20170224/us-east-1/cloudsearch/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-date, Signature=63423d51d91964992083a329c41c6695cbe1abd293aa0a562942b715c55b4f47[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "X-Amz-Date: 20170224T212300Z[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "User-Agent: aws-sdk-java/1.11.95 Linux/4.4.44-39.55.amzn1.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.111-b14/1.8.0_111 scala/2.11.8[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "amz-sdk-invocation-id: c8fb41ca-d010-2989-4133-31fed4a74f35[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "amz-sdk-retry: 2/35/460[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "Content-Type: [\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "Content-Length: 149792[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"  
16:23:03.210   DEBUG [2017-02-24 21:23:00,408] org.apache.http.wire: http-outgoing-7 >> "[\r][\n]"  

[sbailliez]

My code is setting it explicitly to application/json

request.setContentType(ContentType.Applicationjson);

[shorea]

Yeah for REST-JSON services we send empty content to handle some weird behavior in our service stacks. We should be respecting an explicit content type set via a custom header or a modeled field in this case but the recent refactoring broke that. I'll work on a fix and get it out today.

[sbailliez]

Thanks @shorea !

[shorea]

Fix is in code review.

[shorea]

@sbailliez thanks for reporting this! A fix will be released shortly as part of version 1.11.97.

[shorea]

Just for future reference. This bug affects version 1.11.94, 1.11.95, and 1.11.96. The fix was released in 1.11.97.