We saw this behavior in ATS on 2023-08-15 when two messages were sent to SQS for the same events (Transcribe job end). The messages were sent 2 minutes apart, so there is some kind of delay. This is most likely related to the massive scalability and redundancy that AWS builds into these managed services behind the scene.
If we want to prevent processing of the same JSON fille multiple times, we would need to build in some kind of logic to check to see if a given job has already been processed. This would be best accomplished by using a step function and adding a step to write a record to DynamoDB, which we could then check in another step.
Under some circumstances, EventBridge may generate duplicate events for the a triggered rule:
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-troubleshooting.html#eb-rule-triggered-more-than-once
We saw this behavior in ATS on 2023-08-15 when two messages were sent to SQS for the same events (Transcribe job end). The messages were sent 2 minutes apart, so there is some kind of delay. This is most likely related to the massive scalability and redundancy that AWS builds into these managed services behind the scene.
If we want to prevent processing of the same JSON fille multiple times, we would need to build in some kind of logic to check to see if a given job has already been processed. This would be best accomplished by using a step function and adding a step to write a record to DynamoDB, which we could then check in another step.