DynamoDB unmarshalling issue with big json responses

[FedericoLaGreca]

When the plugin tries to parse big json responses from a DynamoDB request, it can fail to unmarshall the response. We noticed this on iOS (always) and Android (on low RAM devices).

If needed I can help quantify a "big" json, but it can usually fail with something over 1000/2000 characters.

The only solution we found is to split the request using the Limit parameter, but that makes it slower as it makes more requests to get the same number of elements.

We noticed that if this case occurs, the Limit parameter needs to be VERY low: eg: on device where everything goes ok, the request can return up to 8000 items. on a device where it fails to unmarshall, the Limit needs to be 10 or lower elements, or it will fail again.

[karthiksaligrama]

Thanks @FedericoLaGreca for the feedback. I was wondering if you have specific information on the device ram and cpu for android?

Also i'm not really sure if there is a valid use case for downloading 1k items on a mobile device.

[FedericoLaGreca]

The device is a Huawei Ascend G600 (U8950N-1): 768 MB RAM, Qualcomm Snapdragon MSM8260 / Scorpion processor

A couple of details: It's not necessarily 1k items, it's 1k characters in the json response. Our case is a leaderboard that also contains user data (the "culprit" is the user profile image, which is saved as url on the database). We end up with a lot of elements downloaded per request because we noticed that if we limit the query to the number of elements we want (usually 150) we won't get all the top elements from the DB. Our query uses GSI, specifying leaderboard ID (that is the range key on the table) and the score value (for > 0) with backward search. No other conditions, but as I said we noticed that by limiting the query to check for 150 elements on the DB, we aren't sure to get 150 elements back from the query.

Thank you

[issfire]

I tested this on 2 different phones. iPhone 5 sees errors with unmarshalling response while my Samsung Note 3 doesn't have issues.

The item I was querying for was 2kb per row and I had to limit my query to 1 result per query for it to work on iPhone 5.

[karthiksaligrama]

@issfire @FedericoLaGreca do you have the stacktrace where it fails to unmarshall?

[FedericoLaGreca]

@karthiksaligrama sorry for the late response, I was on holiday and then it took a while to catch back to work ;)

here is the exception message and stacktrace:

Error unmarshalling response back from AWS. Request ID: AIEKKO3GOEPGFPMAR2F7D8LQ2RVV4KQNSO5AEMVJF66Q9ASUAAJG, Last Parsed Path: /Item/fr/SS/ at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall (Amazon.Runtime.Internal.Transform.UnmarshallerContext input) [0x00056] in /AWSSDK/src/Core/Amazon.Runtime/Internal/Transform/ResponseUnmarshallers.cs:190 at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse (Amazon.Runtime.Internal.Transform.UnmarshallerContext context) [0x00000] in /AWSSDK/src/Core/Amazon.Runtime/Internal/Transform/ResponseUnmarshallers.cs:64 at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse (Amazon.Runtime.Internal.Transform.UnmarshallerContext context, IRequestContext requestContext) [0x00016] in /AWSSDK/src/Core/Amazon.Runtime/Pipeline/Handlers/Unmarshaller.cs:211 at Amazon.Runtime.Internal.Unmarshaller.Unmarshall (IExecutionContext executionContext) [0x00050] in /AWSSDK/src/Core/Amazon.Runtime/Pipeline/Handlers/Unmarshaller.cs:137

the DB entry on which this happened (on Android, same device mentioned before) contained 5258 characters when viewed as a json from the aws console.

let me know if you need anything else

[karthiksaligrama]

Thanks @FedericoLaGreca

I was able to replicate the problem on iOS. It doesn't seems like performance bug but its related to crc32 checks specifically in the JsonUnmarshallerContext.cs file. I observed that this is primarily due to the difference in the WWW implementation for the zipped response.

Unfortunately i wasn't able to replicate the problem on android.

I'm pasting the updated code here, can you take a look and see if it fixes your problem?

        /// <summary>
        /// Wrap the jsonstring for unmarshalling.
        /// </summary>
        /// <param name="responseStream">Stream that contains the JSON for unmarshalling</param>
        /// <param name="maintainResponseBody"> If set to true, maintains a copy of the complete response body as the stream is being read.</param>
        /// <param name="responseData">Response data coming back from the request</param>
        public JsonUnmarshallerContext(Stream responseStream, bool maintainResponseBody, IWebResponseData responseData)
        {
            if (maintainResponseBody)
            {
                this.WrappingStream = new CachingWrapperStream(responseStream, AWSConfigs.LoggingConfig.LogResponsesSizeLimit);
                responseStream = this.WrappingStream;
            }

            this.WebResponseData = responseData;
            this.MaintainResponseBody = maintainResponseBody;

            long contentLength;
            bool parsedContentLengthHeader = long.TryParse (responseData.GetHeaderValue ("Content-Length"), out contentLength);

            //possible scenario in unity where the content length in header does not match responseData.ContentLength
            //responseData.ContentLength represents actual bytes downloaded header value represents the length sent from the server.
            //we will only try to setup crc32 in case the responseData.ContentLenth is same as the content lenght from the header.
            //failing to do so may result in the stream being cut off in the middle (since the zip stream length is less than the responseData.ContentLength)
            //or may result in a crc32 exception since the crc32 calcuated value for an unzipped stream will differ from the crc32 values for a zipped stream.
            if (parsedContentLengthHeader && responseData.ContentLength.Equals (contentLength)) 
            {
                base.SetupCRCStream(responseData, responseStream, contentLength);
            }

            if (this.CrcStream != null)
                streamReader = new StreamReader(this.CrcStream);
            else
                streamReader = new StreamReader(responseStream);

            jsonReader = new JsonReader(streamReader);
        }

[FedericoLaGreca]

@karthiksaligrama I can confirm that the code you posted fixes the issue on our Android device.

[karthiksaligrama]

awesome.. i'll include this patch with the next release.

[linithos]

Hello, I have the same issue.

I've updated the code you posted, but I have these errors, there is something that I missing?

thank you

Lina

1.error CS1061: Type Amazon.Util.LoggingConfig' does not contain a definition forLogResponsesSizeLimit' and no extension method LogResponsesSizeLimit' of typeAmazon.Util.LoggingConfig' could be found (are you missing a using directive or an assembly reference?)

1. error CS1729: The type Amazon.Runtime.Internal.Util.CachingWrapperStream' does not contain a constructor that takes2' arguments

[karthiksaligrama]

Fixed in version 2.0.0.5