Cluster failed to start: Timed out trying to connect

dask / dask-kubernetes

Native Kubernetes integration for Dask

https://kubernetes.dask.org

BSD 3-Clause "New" or "Revised" License

312 stars 148 forks source link

Cluster failed to start: Timed out trying to connect #920

Open shreyaslr opened 6 days ago

shreyaslr commented 6 days ago

Describe the issue:

I am using prefect and dask getting below error

RuntimeError: Cluster failed to start: Timed out trying to connect to tcp://dask-root-d.default:8786 after 30 s

even though the environment value is set to 300 seconds

ENV DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT=300s

Environment:

dask-kubernetes==2023.7.0
dask==2023.7.0
distributed==2023.7.0
prefect==1.2.4

jacobtomlinson commented 4 days ago

To help you troubleshoot this we are going to need a minimum reproducible example. It would also help if you can share the logs from the Dask scheduler Pod.