[Stack Overflow] Dask Kubernetes strange behavior of adapt method

[github-actions[bot]]

I have a Dask cluster on AKS and I want to run a function f in parallel, but have this function run in a single process allocated in a single pod. Acording to the documentation on Worker Resources …

---

Would you like to know more?

Read the full article on the following website:

https://stackoverflow.com/questions/71280180/dask-kubernetes-strange-behavior-of-adapt-method

[pavithraes]

@scharlottej13 I'm not super familiar with Dask deployment, shall we verify this one with Jim in our pair-issue-answering session?

[scharlottej13]

I think that's a great idea! For instance, I'm not quite sure what behavior they mean by this:

If instead of adapt method I set manually cluster.scale(5) the execution of f most of the time they run as I wish. I say most of the time because sometimes the behavior is similar to that of the adapt method. This behavior seems very strange to me.

Here are some references that might be helpful for this one:

• https://coiled.io/blog/under-the-hood-with-dasks-cluster-autoscaling/ (especially Dask adaptive scaling APIs)
• https://docs.dask.org/en/stable/how-to/adaptive.html
• http://distributed.dask.org/en/stable/resources.html

[scharlottej13]

• question title should be more 'how do resources work'
• answer w/ combined info from aaron's answer + their CLI template (process is just a string, fyi, but have to match between the two); also note that this will run 5 tasks in parallel (once one task starts running another won't be scheduled until it completes)