Open saurabhnanda opened 6 years ago
I am pretty sure that code will result in multiple threads sharing connections.
withResource
will hand over a connectionasync
will spawn a thread & that thread will have a reference to the conn
async
will return immediately, so withResource
will return the conn
to the poolIt's better to spawn the thread first and then have the thread ask for a resource:
forever $ async $ withResource dbPool $ \conn -> doSomething conn
Here's a gist that illustrates the difference between your pasted use of withResource
and the one I suggested above:
https://gist.github.com/ramirez7/ede1bc97d851d01e4c603d17da5f738d
The code you pasted in the issue can give currently-allocated resources to new workers, which is exactly what you don't want.
Since this issue is about using pools in concurrent environments in general, I'd like to share an example of a deadlock that can occur with subtly wrong use of resource-pool
:
https://gist.github.com/ramirez7/17602611ba15a74c50a4aaffeebca25a
Nested calls to withResource
are what cause the issue. If the first call successfully allocates resources in multiple threads but then exhausts the pool, the nested calls will block forever.
When using a pool concurrently, if you have more threads than resources, some threads will end up waiting. You can then control the number of concurrent workers by changing your pool size.
A poorly configured pool can easily become the bottleneck in a concurrent system. This resource-pool
PR exposes some useful metrics about pools that can be useful for debugging & scaling.
I'm trying to write a job-queue backed by an RDBMS and have everything up & running, but not designed perfectly. I'm happy to share the entire design/source-code, because I plan to open-source it once it's battle-tested in production, but for now, here's the short version of the question --
How is the following going to behave:
Will
pool
be able to track usage of a resource in an async thread and put it back in the pool only when the thread has exited? Under no circumstance, do I want multiple threads sharing the same DB connection!