Closed balchua closed 5 months ago
I found the issue here, the different executors / workers instances are picking up the same rows from the database.
The way I fixed it is to use SELECT FOR UPDATE SKIP LOCKED
, this way, those items already picked up by other process will simply be ignored by the other process. This is done on both GetOrchestrationWorkItem
and GetActivityWorkItem
.
The code is in this repo.
This repo is a copy of the original PR #33, just wanted to see how it works with postgres db.
I noticed a strange behavior when i run multiple instances of the worker (say 3) all pointing to the same database. Currently took the postgres implementation with some changes.
Here's the screenshot![strange-traces](https://github.com/microsoft/durabletask-go/assets/28750768/93f6d320-5a41-447b-95db-85415b49c10c)
You can see that it contains several
orchestration:SimpleOrchestration
spans. This happens also to activities.I also see several of these logs.
I think this happens while the other workers are all processing the work items, while one of them has already transitioned or completed the work item.