What might cause a a dask graph to freeze/hang when there's only 1 worker but work with no problems when there's 2 workers or above

[CMCDragonkai]

In my application where I've written a test fixture to setup a dask cluster.

I noticed that when my local cluster only had 1 worker in it. The dask graph would compute up to a point and then just hang forever with no errors. The dashboard just shows the whole graph waiting on several tasks in the middle of the whole graph.

Debugging this led to nowhere. So I just added a second worker. As soon as the cluster had 2 workers, the graph completed.

The workers have no memory limit or --memory-limit=0 and there are no constraints on the execution.

Unfortunately the graph is quite large, and so I'm not able to replicate it in this issue.

[jcrist]

Hi @CMCDragonkai,

The workers have no memory limit or --memory-limit=0 and there are no constraints on the execution.

I'm having a bit of trouble parsing this sentence. Do you mean:

• You passed in --memory-limit=0
• You didn't pass anything for --memory-limit, which defaults to auto, a memory limit based on the memory available on the system?

The dask graph would compute up to a point and then just hang forever with no errors. The dashboard just shows the whole graph waiting on several tasks in the middle of the whole graph.

Can you describe how it hangs? Are the tasks still computing, or is nothing happening? How is your cluster set up? Do you have multiple threads per worker?

[CMCDragonkai]

The memory limit is 0. There are still pending tasks. Some tasks have completed. The cluster was 1 scheduler, 1 worker with 1 thread. This is when it always freezed. Then with 2 workers each with 1 thread, it worked with no problems.

On 21 April 2020 08:37:38 GMT+10:00, Jim Crist-Harif notifications@github.com wrote:

Hi @CMCDragonkai,

The workers have no memory limit or --memory-limit=0 and there are no constraints on the execution.

I'm having a bit of trouble parsing this sentence. Do you mean:

• You passed in --memory-limit=0
• You didn't pass anything for --memory-limit, which defaults to auto, a memory limit based on the memory available on the system?

The dask graph would compute up to a point and then just hang forever with no errors. The dashboard just shows the whole graph waiting on several tasks in the middle of the whole graph.

Can you describe how it hangs? Are the tasks still computing, or is nothing happening? How is your cluster set up? Do you have multiple threads per worker?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/dask/distributed/issues/3730#issuecomment-616847875

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[jcrist]

What happens if you give the one worker 2 threads?

[CMCDragonkai]

Nope with 1 process 2 threads, it still doesn't work:

[CMCDragonkai]

With 2 processes it then works perfectly:

[quasiben]

Is it possible to call wait/persist earlier in the task generation and batch the job into one worker to help debug ?