bbc / sqs-consumer

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate
https://bbc.github.io/sqs-consumer/
Other
1.74k stars 333 forks source link

Batch Consumer: some messages are not deleted from queue on success #315

Closed SaraPrager closed 1 year ago

SaraPrager commented 1 year ago

Working with the batch consumer in a high load I get occasional messages that are not deleted from the queue - with no error message.

The relevant code:

  const sqs = new SQS({ apiVersion: '2012-11-05', httpOptions: { agent: new https.Agent({ keepAlive: true }) } });
  const consumer = Consumer.create({ queueUrl, handleMessageBatch, sqs, batchSize });

  consumer.on('error', (err: any) => {
      error(`error on SQS batch message ${err?.message}`);
  });

  consumer.on('processing_error', (err: any) => {
      error(`processing_error ${err?.message}`);
  });

  consumer.on('timeout_error', (err: any) => {
      error(`timeout_error ${err?.message}`);
  });

  consumer.on('message_received', (message: SQSMessage) => {
      debug(`message_received ${JSON.stringify(message)}`);
  });

  consumer.on('message_processed', (message: SQSMessage) => {
      debug(`message_processed ${JSON.stringify(message)}`);
  });

  consumer.on('response_processed', () => {
      debug(`response_processed`);
  });

  debug(`Start consuming batch messages from queue: ${queueUrl}`);
  consumer.start();

The actual logs: `

message_received...

message_processed...

response_processed...

` But the message is not deleted from queue. Any idea? any way to track what's going on? thanks

SaraPrager commented 1 year ago

The issue was a performance problem on my side - handler execution took more time than visibility timeout. Still I would expect to get some error message when the delete fails, but I guess it is related to this bug: https://github.com/bbc/sqs-consumer/issues/193