Turns out through much hair-pulling-and-face-palming I discovered that the PriorityQueue I wrote did not have atomic pops. D'oh! We discovered this when we had two processes running and using a shared PriorityQueue and they were very occasionally pulling off the same key.
This pull request includes the fix to make the PriorityQueue pop atomic, along with some initial thinking I'm doing on having a worker decorator. If you'd prefer, I can remove the co-mingling of the two changes.
Sorry for the bug, but we just discovered it yesterday after several days of teeth-gnashing :-(
Hey again,
Turns out through much hair-pulling-and-face-palming I discovered that the PriorityQueue I wrote did not have atomic pops. D'oh! We discovered this when we had two processes running and using a shared PriorityQueue and they were very occasionally pulling off the same key.
This pull request includes the fix to make the PriorityQueue pop atomic, along with some initial thinking I'm doing on having a
workerdecorator. If you'd prefer, I can remove the co-mingling of the two changes.Sorry for the bug, but we just discovered it yesterday after several days of teeth-gnashing :-(