Open AhmadMS1988 opened 1 year ago
dblock
commented
1 year ago Looks like a client-side problem, I'll move to opensearch-java.
This seems like it could be the reason: https://stackoverflow.com/questions/71611560/got-cancellationexception-request-execution-cancelled-always-when-throwing-an
reta
commented
1 year ago @AhmadMS1988 could you please confirm:
wbeckler
commented
1 year ago This is from the rest high level client, which is in the opensearch core repo. What do you think about moving the issue back?
dblock
commented
1 year ago This is from the rest high level client, which is in the opensearch core repo. What do you think about moving the issue back?
The user is using opensearch-java though. It's a transitive dependency, but still.
<dependency>
<groupId>org.opensearch.client</groupId>
<artifactId>opensearch-java</artifactId>
<version>2.4.0</version>
</dependency>
AhmadMS1988
commented
1 year ago @reta no special timeouts is configured. We use the client with its default configuration. The client works as expected then suddenly all opensearch requests will return the previous error immediately. I do not know if this information is relevant, but we faced the same issue on elasticsearch 7.10.3 java clients too.
thisisananth
commented
1 year ago Hi, I am also facing the same issue. We are using ES High level rest client version 7.13.4. We are running a process to query the documents created in the past 6 months and then upload them to S3 and delete them. It is working fine for the first time. When we run that request again, we are getting the Request exection cancelled exception. Unfortunately after that the client throws that exception for every request and it becomes pretty useless. We need to restart the server to be able to get it working again.
dblock
commented
1 year ago @thisisananth Generally I don't have much insights into Elasticsearch (OpenSearch forked at 7.10), however this may be a bug that we've carried since so it's worth pursuing. What version of the server are you using, does it match the client version (7.13.4) and did you open an issues with Elasticsearch about this?
Then, is this reproducible, anything helpful if you enable DEBUG level logs, and is this caused/correlated with a server-side error (look at server-side logs/HTTP responses)?
reta
commented
1 year ago The only theory I have is that the client is closed somewhere:
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:114)Sadly the close() just sets the status, so it not possible to understand where it was called from these stack traces. There is also a possibility the async client failed, looking for "I/O reactor terminated abnormally" in logs could help to prove (or disprove) that.
Describe the bug Occasionally, when we stress out application APIs that are requesting Opensearch, the application starts
throwing java.lang.RuntimeException: Request execution cancelled, and will never work again until we restart the application.To Reproduce Randomly stressing search request using rest and high level rest clients.
Host/Environment (please complete the following information):
Client information:
Additional context Stacktrace:
Can we know what is the cause of this error? and is there any a fix or a workaround to may prevent this problem. Thank you