This provides a more robust implementation of a process executor which includes things like recycling idle processes. This should hopefully reduce our memory leak issues.
Additionally, this commit changes the way we manage processes so that there's only one general purpose process pool for offloading tasks rather than several. This makes it easier to control server resources and also it seems that with Loky you can only have 1 process pool running at once.
Annoyingly this seems to break my ability to debug functions running in the process pools in PyCharm but heyho. Maybe we could add something that switches the pool in use depending on settings/env.
This provides a more robust implementation of a process executor which includes things like recycling idle processes. This should hopefully reduce our memory leak issues.
Additionally, this commit changes the way we manage processes so that there's only one general purpose process pool for offloading tasks rather than several. This makes it easier to control server resources and also it seems that with Loky you can only have 1 process pool running at once.
Annoyingly this seems to break my ability to debug functions running in the process pools in PyCharm but heyho. Maybe we could add something that switches the pool in use depending on settings/env.
Closes #30 🤞🏼