search

aws / aws-sdk-net

The official AWS SDK for .NET. For more information on the AWS SDK for .NET, see our web site:
http://aws.amazon.com/sdkfornet/
Apache License 2.0
2.05k stars 852 forks source link

Modify CloudSearch Search to use POST instead of GET #900

Closed joel-cass closed 1 week ago

joel-cass commented 6 years ago

Any search request using AmazonCloudSearchDomain.SearchAsync(searchRequest) or AmazonCloudSearchDomain.Search(searchRequest) that has a large search string will end up with a 413 Full Head message being returned from the AWS service.

This exception is not handled gracefully by the Marshaller and will return a Null Pointer Exception that obfuscates the actual cause of the issue. I don't care much for this; it would be better that the actual issue above is addressed.

Expected Behavior

Searching for a long string should return results (create a search longer than 4000 characters for example).

The same search can be run in the console with no problems as it is using post

Current Behavior

Instead it returns ``[NullReferenceException: Object reference not set to an instance of an object.] Amazon.Runtime.Internal.Transform.JsonErrorResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context) +874 Amazon.CloudSearchDomain.Model.Internal.MarshallTransformations.SearchResponseUnmarshaller.UnmarshallException(JsonUnmarshallerContext context, Exception innerException, HttpStatusCode statusCode) +40 Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.UnmarshallException(UnmarshallerContext input, Exception innerException, HttpStatusCode statusCode) +61 Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) +282

[AmazonUnmarshallingException: Error unmarshalling response back from AWS. ] Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) +1011 Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) +275``

And using Fiddler we see the response from the server: HTTP/1.1 413 FULL head Content-Length: 0 Connection: keep-alive

Possible Solution

SearchRequestMarshaller has hard-coded the method of GET for requests.

https://github.com/aws/aws-sdk-net/blob/1a5187e0dcddadba10a6b90fd827f2e485af0f9c/sdk/src/Services/CloudSearchDomain/Generated/Model/Internal/MarshallTransformations/SearchRequestMarshaller.cs#L58

Change it to POST, and the problem will probably go away.

Steps to Reproduce (for bugs)

Create a search index with the field 'type' Run test:

[Test] public void Test_AWSCloudSearch_GivenReallyLongSearchString() { var search = new AmazonCloudSearchDomainClient("<your-search-domain-here>"); var result = search.Search(new SearchRequest { QueryParser = QueryParser.Structured, Query = "(and type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' )", Start = 1, Size = 15, Sort = "_score desc,objectid asc", Return = "_score,_all_fields" }); Assert.AreNotEqual(result, null); }

Context

We are trying to do a search where words in any field are matched using an or statement. Because there is no way to group terms by field, or an ability to restrict fields conditionally in simple search, we have had to use structured search in this manner, when a long list of keywords are used across a range of fields then the string can become too long for the service to handle.

This is just the first search where we have hit this limit, I am concerned that even if we are able to work around the issue, it exposes a limitation in the ASP.Net library, which could impact whether we choose to use this service in production

Your Environment

sstevenkang commented 6 years ago

Interesting... thanks for the detailed write up. It looks like CloudSearchDoomain is recommending using GET(the code snipped you included is generated code based on model provided by CSD team). Let me follow up with the service team if it's safe to default to POST.

Also, it looks like we have some gaps where we should re-throw a meaningful exception.

EyeDreamt commented 5 years ago

I also have the same issue. I see this was adjusted to use POST in the Java SDK. Are there plans to update the .Net SDK soon?

tpund commented 5 years ago

This has become a production issue for me as well. Workarounds are failing at this point. Please look into t his.

franzcars commented 5 years ago

This has become an issue for me as well. Please look into this.

CerebralUnit commented 5 years ago

This has become an issue for our team. At least 50% of our queries are failing because we're requesting all of our fields to be returned and it's too much for a GET request. Have you guys made progress on this since March?

joel-cass commented 5 years ago

I gave up on AWS helping me out on this a while ago, checked out their code and created a modified version that uses POST instead of GET. It's very easy to swap in if you're using an IOC framework such as Windsor. This should help if you're in a pinch.

aws-sdk-net-issue-900.zip

klaytaybai commented 5 years ago

@joel-cass, @EyeDreamt, @tpund, @franzcars, @CerebralUnit, and others, sorry we let this issue get lost in the backlog. It looks like this was an issue faced by almost all of our SDKs because we depend on the service models to generate a lot of our code. They specify the HTTP method, which is defaulted to GET for this operation, but the service models don't provide a mechanism for alternative HTTP methods. We should have escalated this as an issue that affects all SDKs when first discovered, but I don't think we did. I'll raise this as a discussion point and see if we can get a fix out for .NET relatively soon.

phillip-haydon commented 4 years ago

@sstevenkang @klaytaybai

https://github.com/aws/aws-sdk-java/issues/660

This was fixed in the Java SDK in 2016

https://aws.amazon.com/releasenotes/release-aws-sdk-for-java-1-11-32/

It's really disappointing to see this isn't resolved, we've been running with our own implementation for like 4 years now and I don't want to maintain code that the AWS SDK should be doing for me.

brandongregoryscott commented 3 years ago

Would also love to see this resolved for the .NET SDK ☝️ We just started running into this with a longer filter query, expecting the underlying request to go out as a POST - two years to the day when it was escalated as a global issue, or supposedly.

github-actions[bot] commented 2 years ago

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

Kralizek commented 2 years ago

@ashishdhingra

This bot has a really unhealthy setup trying to close issues that haven't been solved just because the AWS team forgot about them. Could you please check with your team on how to change its settings so that issues like this one are not closed by mistake?

phillip-haydon commented 2 years ago

The fact that we can implement our own class to resolve this ourselves makes me absolutely baffled why AWS doesn’t want to fix this issue.

phillip-haydon commented 2 years ago

4 years and unresolved…

joel-cass commented 2 years ago

Well AWS isnt exactly kicking goals with this one.

Ive created a PR in an attempt to expedite the process, am not holding my breath in anticipation.

bhoradc commented 1 week ago

Hello @joel-cass,

Cloudsearch is moving to Opensearch, and therefore most unlikely we would be doing this feature request. For reference - https://aws.amazon.com/blogs/big-data/transition-from-amazon-cloudsearch-to-amazon-opensearch-service/.

We will go ahead and close this out, feel free to provide your thoughts.

Regards, Chaitanya

github-actions[bot] commented 1 week ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.