Just thought I would surface a couple of issues I have been having- they seem to be known- but complex, to do with memory allocation. A few workarounds have been proposed, but I've not attempted to implement any.
Symptom:
distributed.utils_perf - WARNING - full garbage collections took 63% CPU time recently (threshold: 10%)
#859, mem: 125.76 mb
Also, workers failing to clear memory and becoming stuck.
Just thought I would surface a couple of issues I have been having- they seem to be known- but complex, to do with memory allocation. A few workarounds have been proposed, but I've not attempted to implement any.
Symptom:
Also, workers failing to clear memory and becoming stuck.
Discussion: https://github.com/dask/distributed/issues/2068 https://github.com/dask/dask/issues/3530
No feeling on how we should handle this currently- but worth keeping an eye on I think.