search

nspcc-dev / neofs-node

NeoFS is a decentralized distributed object storage integrated with the Neo blockchain
https://fs.neo.org
GNU General Public License v3.0
31 stars 38 forks source link

Release ants worker pools #901

Open cthulhu-rider opened 2 years ago

cthulhu-rider commented 2 years ago

Mentioned by @fyrchik in https://github.com/nspcc-dev/neofs-node/pull/899.

We use ants.Pool as an implementation of limited worker pools. Each instance should be closed (Release method) after there is no more need in pool (almost always on application shutdown).

  1. I'd be better to concentrate the implementation of util.WorkerPool interface in a single package (e.g. util/antspool) and use it instead of direct calls like in https://github.com/nspcc-dev/neofs-node/pull/899.
  2. Extend util.WorkerPool interface with io.Closer.
  3. Close all pools in application onShutdown or in Close methods of each component.
fyrchik commented 2 years ago

I have checked ants.Pool and it doesn't wait until submitted workers have returned in Release. This is crucial to graceful shutdown, because we want all pending operations to finish before the storage is closed. So some kind of sync.WaitGroup seems inevitable at least in shard. Another way to fix is to implement a wrapper over ants.Pool in util and require Release method to wait for all currently-being-executed workers to finish.

cthulhu-rider commented 2 years ago

Lets gather all the use-cases and describe common mechanism of pool allocation and free-ing.