pressly / qmd

[DEPRECATED] Async script processing web service
MIT License
11 stars 3 forks source link

Dedup #10

Closed dkua closed 10 years ago

dkua commented 10 years ago

This is for issue #5 the duplicated message problem. The solution uses Redis as a locking mechanism whereas handlers who consume the message will check for and create a key similar to Job #:id. All handlers after the first will be locked out until 900 seconds (15 minutes) until the key expires.

900 seconds is how long NSQ will allow a message to be in the system by default. If the NSQ max-message-timeout was increased or our JOB_TIMEOUT was decreased then it would be possible for QMD to have duplicate messages in the system. Clock skew could also be an issue if there is a huge skew.