Is it possible not to delete a message from the queue when the callback is successful?

[ypicard]

I would like to use this library to trigger long running jobs which are themselves responsible of deleting the original message from the SQS queue.

Is it possible to not delete the received message when the handler receives successfully?

[crh3675]

Sounds like you need an additional message to be sent to the Queue for a log running job. You can tweak the acknowledgment visibility timeout and I believe it is 12 hours now but it seems like an extreme usage of the SQS service.

[crh3675]

I think that devs are mis-using AWS technologies and asking for features in libraries to do things that they SHOULD NOT do. I see so many situations where a dev is answering a problem expecting a single AWS service to provide that. AWS does not provide solutions, just a hoard of tools to make your own (SQS, SNS, ElasticCache, Kinesis, Lambda)

[crh3675]

@ypicard If you are able to disclose more of the "problem you are trying to to fix", perhaps insight can be given as to a better means to meet those goals

[ypicard]

Indeed, I would love to see if another approach would better suit my problem, thank you.

I need an event to trigger a long running job and make sure this job is successfully processed.

To make sure the job is successfully processed, I see 2 options:

1. Trigger the job from the handler in the sqs-consumer and poll Airflow continuously to fetch the job's state. When successful, return from the handler successfully, which will delete the SQS message from the queue.
2. Trigger the job from the handler in the sqs-consumer, not delete the message from the queue, and let the job delete the message himself when finished. I am not sure this is a way I should go with because it brings responsibility to the job and couples it to the communication and event layer of my application, but it looks very reliable.

I am open to any advice and suggestion: other technologies, tools, architecture.

Thank you

[DocCaliban]

I have a use for this functionality as well

I am receiving a message that tells me to create and / or update and entry in a DB. If I receive an update message before receiving the create message I don't have a record to update. I could insert and the when I get the create just use it as an update, but this causes other potential problems.

The other potential issue we have is we have linked records that could come in before the parent record is created, again we could try and insert them anyway, but it's not the ideal solution.

The ideal case would be to take an action, and if it doesn't return properly, allow me to not delete the message so it will get processed in the next batch, until such time that it hits a threshold or the criteria is satisfied.

@crh3675 to your point

'AWS does not provide solutions, just a hoard of tools to make your own (SQS, SNS, ElasticCache, Kinesis, Lambda)'

You are correct, but AWS does provide a way to not delete the message, it's this library that automatically deletes the message, so the answer to the original question is 'not with this library', but I think it would be a great feature, one that I could use for sure!

[DocCaliban]

FYI @ypicard Also seem like this feature request already exists here https://github.com/bbc/sqs-consumer/issues/189

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[nicholasgriffintn]

Duplicates this ticket: https://github.com/bbc/sqs-consumer/issues/189