NerdWalletOSS / kinesis-python

Low level, multiprocessing based AWS Kinesis producer & consumer library
Other
118 stars 50 forks source link

Use of offpsring eventually leads to too many open files if there are closed shards #24

Open jeffglover opened 5 years ago

jeffglover commented 5 years ago

While doing some testing of kinesis-python I was evaluating how well it handles live resharding. As old shards remained kinesis-python would spin up a ShardReader and shut it down for each inactive shard. Offspring keeps track of these instances only to cleanup on exit, but they build up overtime and eventually resulting in [Errno 24] Too many open files

Offspring looks dead, but I submitted a pull request that would fix it. https://github.com/borgstrom/offspring/pull/3