Open hendryluk opened 1 month ago
Looks like a bug :( Do you think you can try to turn your repro into a (failing) unit test with a server mock response?
The issue seems to be really related to quota setting (it is not reproducible locally at least):
Create an AWS OpenSearch Service domain, with data nodes of an instance-type with 10MB network quota, e.g. m6g.large.search
@dblock I may need your help here, wondering if we could have an AWS OpenSearch Service for a few hours to troubleshot the issue?
@reta I can help offline, yes.
Do you think you can try to turn your repro into a (failing) unit test with a server mock response?
@dblock Unfortunately it's not easily replicable with a mock server that returns a specific error response. The bug only occurs with the specific low level network behavior of AWS when we reach the network quota, in which case we'll get a specific sequence of events that's different from normal, e.g. it closes the request stream (preventing it from sending more data) before writing the status to the response.
So I think the best way would be to test this bug is with actual AWS.
Force-closing the request stream in an intercepted call is usually a solution, but I am talking without trying.
@hendryluk @dblock interesting, the issue is related to HTTP protocol chosen:
RestClientTransport
for 2.x is always using HTTP/1.1, (HC4), butApacheHttpClient5Transport
talks HTTP/2.0 and there is no response callback (looking why now)Setting HttpVersionPolicy
for the client fixes the issue, but I am not sure where exactly the problem is: OS 2.x does not support HTTP/2.0, why protocol is negotiated this way? gateway or LB in front?
ApacheHttpClient5TransportBuilder.builder(HttpHost.create(OPENSEARCH_URL))
.setHttpClientConfigCallback(builder -> builder.setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_1))
A bit more on that: the behaviour vary if the payload is chunked (buffer size) or not (sent at once), 400
vs 413
is returned.
Thanks @reta for looking into the issue and for the workaround - we'll give that a try.
What is the bug?
When using java client with ApacheHttpClient5Transport, large requests that exceeds AWS network limit (e.g. 10MB on
m6g.large.search
) will cause the request to hang and permanently blocks the thread. In actuality, AWS responds with 400 (or 413 if thecontent-length
header is present in the request, which is missing in this case, but that's another issue).This bug only affects
ApacheHttpClient5Transport
, but works correctly on the legacyRestClientTransport
(i.e. throws an exception). It also works with curl (i.e. returns a 400 or 413 response).How can one reproduce the bug?
Steps to reproduce the behavior.
m6g.large.search
Make a bulk request >10MB, with the following junit test:
What is the expected behavior?
Throws
OpenSearchException
orResponseException
with 400 or 413 error.What is the actual behavior?
The thread hangs forever. No exceptions, no timeouts.
What is your host/environment?
m6g.large.search
opensearch-java
client version: 2.10.1Do you have any screenshots?
No. See the junit code above.
Do you have any additional context?
Only affects
ApacheHttpClient5Transport
. Works correctly withRestClientTransport
, curl, postman, etc, i.e. produces 400 or 413 errors.