twosigma / Cook

Fair job scheduler on Kubernetes and Mesos for batch workloads and Spark
Apache License 2.0
337 stars 63 forks source link

Parallel watch processing. #2169

Closed scrosby closed 2 years ago

scrosby commented 2 years ago

Changes proposed in this PR

Why are we making these changes?

Processing watches was previously done by a single thread which could impact the rate Cook could process job completions. Doing it in parallel allows us to handle a higher job completion throughput.