bbc / sqs-consumer

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate
https://bbc.github.io/sqs-consumer/
Other
1.71k stars 330 forks source link

Configure consumer to use a SQS client that gets credentials using sts.assumeRole and post refresh #373

Closed razzpoker50 closed 1 year ago

razzpoker50 commented 1 year ago

Describe the bug

Created a node service application that is responsible to send messages to a SQS queue. The reason the processing is not done as part of the main API and offloaded to the sqs consumer ( that runs in a separate container ) is to unblock the API and run the processing in the background which will be responsible to call 3rd party API's and transform data and store the transformed data in the DB.

In order to do so I have to call the AWS.sts.assumeRole to get the temporary credentials which will be passed along to the sqs client. This is working as expected when the pod starts running but once the temporary credentials expire we start running into the error.

SQS receive message failed: The security token included in the request is expired, undefined

One way to stop seeing this error is to restart the pod which is not really ideal. What would be the best way to refresh the credentials and pass it to the sqs client so that we don't run into the above error.

async function handleMessage(message: AWS.SQS.Message) {
  const messageId = message.MessageId;

  // process the message 
}

async funtion getServiceObjectsWIthCrossAccountCredentials() {
     // call to AWS.sts.assumeRole to get temporary credentials

    // use the credentials to create the SQS client
}

const main = async () => {
  const { sqsClient } = await getServiceObjectsWIthCrossAccountCredentials();

  logger.info('START SQS consumer for datawarehouse');

  const consumer = Consumer.create({
    queueUrl,
    visibilityTimeout: 10,
    handleMessage: handleMessage,
    sqs: sqsClient,
  });

  consumer.on(
    'message_received',
    (message: AWS.SQS.Message) => {
      logger.info({
        message: `Message received with messageId: ${messageId}`,
      });
    },
  );

  consumer.on(
    'message_processed',
    async (message: AWS.SQS.Message) => {
      const messageId = message.MessageId;
      logger.info({
        message: `Message processed with messageId: ${messageId}`,
      });
    },
  );

  consumer.start();
};

main();

Trying to figure out the cleanest way to keep the application processing the messages without running into the invalid credentials error. Is stopping and restarting the container once the credentials expire the best way to go or are they any other solutions ? Any pointers on this regard would be great.

Your minimal, reproducible example

none

Steps to reproduce

None

Expected behavior

I expect the sqs consumer to process the messages with out any invalid credentials error.

How often does this bug happen?

Every time

Screenshots or Videos

No response

Platform

macOS

Package version

v5.8.0

AWS SDK version

v2.1178.0

Additional context

No response