SQS receiveMessage throws an error that goes around catch block

[yhaskell]

Describe the bug

We have a setup with multiple accounts. During one of the maintenance windows, DevOps engineers accidentally enabled KMS encryption for the SQS queue my app was listening in.

Surprisingly, instead of just logging the error, the app crashed, despite the receiveMessage call being awaited inside of the try-catch block.

Expected Behavior

• The error gets caught in the catch statement around receiveMessage call
• nothing else happens

Current Behavior

• The error gets logged
• The error gets logged second time, but not by the app
• The app crashes with an uncaught error

Error that gets uncaught:

KMS.AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access. (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: 5ebf3f88-a721-45a0-9e1b-65981878f1e5; proxy: null)
    at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'KMS.AccessDeniedException',
  time: 2022-07-01T13:44:17.639Z,
  requestId: '3fbf0dc7-7117-5e72-aad1-56ff8d7bdac7',
  statusCode: 400,
  retryable: false,
  retryDelay: 10.318890537814163
}

Reproduction Steps

import { SQS } from "aws-sdk";
const QueueName = "<insert queue name here>";
const QueueOwnerAWSAccountId = "<insert account id here>";

const sqs = new SQS({ region: "us-east-1" });

async function main(sqs: SQS) {
  try {
    const { QueueUrl } = await sqs.getQueueUrl({ QueueOwnerAWSAccountId, QueueName }).promise();

    console.log("Acquired QueueUrl", QueueUrl);

    if (!QueueUrl) {
      console.log("Empty QueueUrl");

      return;
    }

    const Message = await sqs
      .receiveMessage({
        QueueUrl,
        WaitTimeSeconds: 10,
        MaxNumberOfMessages: 1,
      })
      .promise();

    console.log("Received Message", Message);
  } catch (err) {
    console.log("Cannot receive message", err); // does not happen
  }
}

main(sqs).then(() => process.exit(0)); // logs uncaught error, crashes

Possible Solution

No response

Additional Information/Context

No response

SDK version used

2.875.0

Environment details (OS name and version, etc.)

node 16.13, dockerised, inside of EKS

[ajredniwja]

Hey @yhaskell thanks for opening this issue, using the code above I was not able to reproduce the issue, I used something like:

var AWS = require("aws-sdk");
const QueueName = "myQue";
// const QueueOwnerAWSAccountId = "";

const sqs = new AWS.SQS({ region: "us-west-2" });

async function main(sqs) {
    try {
        const { QueueUrl } = await sqs.getQueueUrl({ QueueName }).promise();

        console.log("Acquired QueueUrl", QueueUrl);

        if (!QueueUrl) {
            console.log("Empty QueueUrl");

            return;
        }

        const Message = await sqs
            .receiveMessage({
                QueueUrl,
                WaitTimeSeconds: 10,
                MaxNumberOfMessages: 1,
            })
            .promise();

        console.log("Received Message", Message);
    } catch (err) {
        console.log("Cannot receive message", err); // does not happen
    }
}

main(sqs).then(() => process.exit(0));

Response:

Acquired QueueUrl https://sqs.us-west-2.amazonaws.com/021102071791/myQueName
Received Message {
  ResponseMetadata: { RequestId: 'd389280kdj8' },
  Messages: [
    {
      MessageId: 'x',
      ReceiptHandle: 'ldj',
      MD5OfBody: '411a4a47dd82fs546f99570asf423',
      Body: 'asdlkad'
    }
  ]
}

Are there any additional details that might come into play? Do you see similar error with the latest version of the SDK too?

[yhaskell]

@ajredniwja to reproduce the issue, the message must come from another account, and the queue must be set up to encode with the key that doesn't exist in the account, where the message is received.

The happy path definitely works ;)

[ajredniwja]

@yhaskell apologies this fell out of queue but I had troubles reproducing this. I followed some of the steps from here as well: https://aws.amazon.com/premiumsupport/knowledge-center/sns-topic-sqs-queue-sse-kms-key-policy/

Can you also check for permissions as mentioned here: https://docs.aws.amazon.com/sns/latest/dg/sns-key-management.html#sns-what-permissions-for-sse

[yhaskell]

@ajredniwja please note that the problem is NOT the fact, that the message cannot be decoded. This is expected, and this is not a problem.

Problem is, that the error about decoding is thrown outside of the standard event loop and breaks the application, getting out of try-catch.

To reproduce, you need 2 accounts, SQS queue created in 1 account and being read from another account. The access to the 2nd account must be granted, but not to the KMS key, resulting in the error being thrown

[aBurmeseDev]

Hi there - checking in. Apologies for the delay. I can't reproduce this behavior with recent version in the scenario that you described. If this's still an issue for you, I would advise adding a handler explicitly for KMS.AccessDeniedException to prevent from crashing your application.

async function main(sqs: SQS) {
  try {
    const { QueueUrl } = await sqs.getQueueUrl({ QueueOwnerAWSAccountId, QueueName }).promise();

    console.log("Acquired QueueUrl", QueueUrl);

    if (!QueueUrl) {
      console.log("Empty QueueUrl");
      return;
    }

    const Message = await sqs
      .receiveMessage({
        QueueUrl,
        WaitTimeSeconds: 10,
        MaxNumberOfMessages: 1,
      })
      .promise();

    console.log("Received Message", Message);
  } catch (err) {
    if (err.code === "KMS.AccessDeniedException") {
      console.error("Access denied to KMS key for this SQS queue. Please check your KMS key configuration.");
      // You can add additional error handling logic here, such as logging, retrying, or graceful shutdown
    } else {
      console.error("Cannot receive message", err);
    }
  }
}

main(sqs).catch((err) => {
  console.error("Uncaught error:", err);
  process.exit(1); // Exit with a non-zero status code to indicate an error
});