Ability to remove workers after delay, conversion of workers to queue, callback when a worker is checked in and full pool status

devinus / poolboy

A hunky Erlang worker pool factory

http://github.com/devinus/poolboy

ISC License

1.56k stars 348 forks source link

Ability to remove workers after delay, conversion of workers to queue, callback when a worker is checked in and full pool status #106

Open zugolosian opened 6 years ago

zugolosian commented 6 years ago

https://github.com/devinus/poolboy/pull/83 has been in use in a production environment for a while but there are some possible improvements for making it more efficient. I realise this pull request has a lot of changes but it would be great if we could figure out some of the common feature requests together so there aren't so many active forks. This pull request:

Converts the workers to a queue for better lifo performance
Adds the ability to remove workers after a delay (reasoning detailed in https://github.com/devinus/poolboy/pull/83)
Adds an optional callback so callers can perform any task they wish with the worker before checkin like a database connection reset, also gives the ability to kill/restart a worker on checkin if desired
Adds a full_status function which provides more information should someone want to graph pool changes
Adds a total workers started counter for detecting churn over time

zugolosian commented 6 years ago

I should also mention because of the use of monotonic time pre 17 is not supported

zugolosian commented 6 years ago

Also happy to split this up a bit so each change can be looked at separately.

Vagabond commented 6 years ago

Yeah, we should really address this. I'm sorry we've taken so long on it.

I'm very busy and I'm not sure @devinus is around either, so anything you can do to make it easy to test would be appreciated.

@getong since you're also interested in helping maintain poolboy, can you pitch in here?

getong commented 6 years ago

At first glance, it uses queue to handle workers, it is a very good idea. Since some code was fixed in #104 , would you rebase your branch to master? @zugolosian

zugolosian commented 6 years ago

Happy to rebase, will likely split the changes into:

Conversion to use a queue for workers
Full status information
Worker callback
Ability to remove workers after delay

That sound reasonable?

Vagabond commented 6 years ago

Thank you, yes. Also make sure you resolve the merge conflicts :)

getong commented 6 years ago

rebar3_eqc has a new version 0.1.0, you can update to it.

zugolosian commented 6 years ago

Sorry I haven't touched this in a while, been pretty busy myself. Shooting for sometime on Friday to separate into different pull requests.

zugolosian commented 6 years ago

@getong I've added a pull request for the conversion to a queue, let's work through that first and I'll start making the others so there aren't a bunch of conflicts to resolve