[BUG] "No server available to handle the request" thrown by AWS Opensearch Service

[uttsap]

What is the bug?

It doesn't happen regularly, but I was able to catch it in my debugger: And then, because of that trailing comma, things further complicate. That error is not even well formatted JSON, which makes it throw DeserializationError, which is how it reaches my code. We can ignore this one, I just needed to bitch about it a little bit.

How do we find why is the "No server available to handle the request" happening? How to mitigate this?

If this is the expected behavior of an overloaded ES cluster, how do we properly handle this error specifically?

2024-01-11T00:02:35.360Z    9b8c0ae7-b375-5af1-849e-54fe813d8649    ERROR   [dd.trace_id=3870422671396660654 dd.span_id=2735995719494683067] Invoke Error   {"stack":"DeserializationError: Unexpected token } in JSON at position 60
    at e.exports.deserialize (/var/task/webhose-normalizer.js:2:1032096)
    at M (/var/task/webhose-normalizer.js:2:1014135)
    at IncomingMessage.l (/var/task/webhose-normalizer.js:2:1013396)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)","message":"Unexpected token } in JSON at position 60","name":"DeserializationError","data":"{
  \"message\": \"No server available to handle the request\",
}
"}

How can one reproduce the bug?

It doesn't happen regularly. Error is seen when there is high CPU usage.

What is the expected behavior?

Well formated error that can be caught from our application.

What is your host/environment?

_AWS Lambda using sdk v3, opensearch-js 2.2.0

Do you have any screenshots?

Do you have any additional context?

Add any other context about the problem.

[dblock]

I think there are two problems here.

1. The fact that AOS is throwing the error. The best path is to open a ticket with AWS customer support.
2. The fact that the client is not handling the error gracefully. I think it needs to be catching the JSON parsing error and re-throwing a different server error with the body as text.

I think (2) is definitely a problem for this client. If you want to try to fix this, start by writing a test that reproduces this behavior?

[uttsap]

@dblock It seems that the existing test case test('DeserializationError', ...) in the opensearch-js client code attempts to reproduce a JSON parsing error scenario.

are you proposing doing something like this?

try {
  result.body = this.serializer.deserialize(payload);
} catch (err) {
  if (err.name === 'SyntaxError') {
    // JSON parsing error, handle it gracefully
    const errorResponse = {
      message: 'Error parsing JSON response',
      originalError: {
        message: err.message,
        name: err.name,
        position: err.at,
      },
      data: payload, // Include the payload for further analysis if needed
    };

    this.emit('response', errorResponse, result);
    return callback(errorResponse, result);
  } else {
    // Other errors, re-throw the original error
    this.emit('response', err, result);
    return callback(err, result);
  }
}

[dblock]

No, something like rethrow the original error without the message being parsed. Basically whatever happens if the server returns a text error instead of json.