search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

Worker-Scheduler SSL causing performance pain #100

Closed mrocklin closed 3 years ago

mrocklin commented 3 years ago

I'm running a few experiments at moderate scale (100 4-core workers with many small tasks) and I'm finding that reading/writing on SSL sockets is the primary bottleneck. How can we handle this?

We could remove TLS security from workers/schedulers . This is currently redundant with our use of security groups. We've taken a belt-and-suspenders approach to network security so far, but maybe this performance issue encourages us to drop it, and just go with security groups. We would continue to use TLS/SSL between the client and scheduler.

mrocklin commented 3 years ago

https://github.com/dask/distributed/issues/4443

mrocklin commented 3 years ago

I'm going to work with OSS folks to figure this out before bugging people here. Closing for now.