Closed sermojohn closed 1 year ago
Hey @sermojohn!
I think you are onto something here, this purge function presented the issue you describe in earlier versions of the lib, but after running your snippet I wasn't able to reproduce the deadlock. I'll take a closer look over the weekend, but in the meantime, if you could help me come up with a sample that enters a deadlock in every run that would greatly help me debug this issue.
I made some modifications to your snippet to log how many tasks are pending after the program completes, how many idle/running workers are there before submitting another task, and also reduced the IdleTimeout to 1 microsecond to ensure the purge
function runs more often (to force the deadlock). Here's the modified version: https://go.dev/play/p/hw1iGibUVDb
Thanks for opening this issue! 🙂
Hey @sermojohn,
I was able to reproduce this issue within a unit test and found a solution for this scenario by ensuring maybeStartWorker
is synchronised with purge
, as you pointed out.
This is the PR with the fix: https://github.com/alitto/pond/pull/34 and I just released version v1.8.1
, which includes it.
Please check it out if you have some time and let me know how that goes :slightly_smiling_face:
Thanks!
@alitto, thank you for your prompt fix! This library is proved to be the right tool for the job. Feel free to close this issue, it looks fixed from my perspective.
I hope to manage to contribute back to this library in the future!
I experienced a deadlock when using pond in the following way (https://go.dev/play/p/eJLX1vc3C81).
Minimum workers defaults to 0 and
purge
can stop idle workersIdleWorkers() > 0
. At the same time,workerpool.Submit
can add a task but not start a worker becauseIdleWorkers() > 0
. If thepurger
managed to signal the worker to stop before the newly submitted job is consumed by the worker, the workerpool ends up with a non-empty task channel and no workers to process the tasks.Possible solutions:
purge
andmaybeStartWorker
to avoid such cases.