fix(multiprocessing): Reset pool if tasks are not completed

getsentry / arroyo

A library to build streaming applications that consume from and produce to Kafka.

https://getsentry.github.io/arroyo/

Apache License 2.0

39 stars 6 forks source link

fix(multiprocessing): Reset pool if tasks are not completed #315

Closed lynnagara closed 6 months ago

lynnagara commented 6 months ago

If multiprocessing tasks are not completed within the timeout specified, we need to reset the pool to avoid state being carried over between assignments.

untitaker commented 6 months ago

also please add a counter metric when the pool is being recreated, i imagine that with this change, we will have to fine-tune join timeout again to improve consumer performance.