scottyhq
commented
5 years ago moving comment from other issue tracker:
Okay, I can reproduce the dask pods and nodes in limbo by logging onto esip.pangeo.io and launching a dask cluster. if no worker nodes are available it seems to be taking too long to scale up and there are network errors (scroll to bottom):
kubectl describe pod dask-scottyhq-527e3b25-ag48fq -n esip-prod
Name: dask-scottyhq-527e3b25-ag48fq
Namespace: esip-prod
Priority: 0
Node: ip-192-168-78-224.us-west-2.compute.internal/192.168.78.224
Start Time: Wed, 04 Sep 2019 19:55:16 -0700
Labels: app=dask
component=dask-worker
dask.org/cluster-name=dask-scottyhq-527e3b25-a
user=jovyan
Annotations: kubernetes.io/psp: eks.privileged
Status: Pending
IP:
Containers:
dask-scottyhq:
Container ID:
Image: 783380859522.dkr.ecr.us-west-2.amazonaws.com/pangeo-esip:e6ec0b9
Image ID:
Port: <none>
Host Port: <none>
Args:
dask-worker
--nthreads
2
--no-dashboard
--memory-limit
7GB
--death-timeout
60
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 1750m
memory: 7G
Requests:
cpu: 1
memory: 7G
Environment:
DASK_SCHEDULER_ADDRESS: tcp://192.168.4.168:35189
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l8cgz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-l8cgz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-l8cgz
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: k8s.dask.org/dedicated=worker:NoSchedule
k8s.dask.org_dedicated=worker:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 7m42s (x2 over 7m42s) default-scheduler 0/2 nodes are available: 1 Insufficient cpu, 2 Insufficient memory.
Normal TriggeredScaleUp 7m32s cluster-autoscaler pod triggered scale-up: [{eksctl-pangeo-esip-nodegroup-dask-worker-v2-NodeGroup-UKEXL45VXVA7 0->1 (max: 100)}]
Warning FailedScheduling 6m12s (x3 over 6m14s) default-scheduler 0/3 nodes are available: 1 Insufficient cpu, 1 node(s) had taints that the pod didn't tolerate, 2 Insufficient memory.
Normal Scheduled 6m4s default-scheduler Successfully assigned esip-prod/dask-scottyhq-527e3b25-ag48fq to ip-192-168-78-224.us-west-2.compute.internal
Warning FailedCreatePodSandBox 6m2s kubelet, ip-192-168-78-224.us-west-2.compute.internal Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "77c187a18ff7974198302f452bfb89cf126689939ff7fe72bb4598c55ea4d360" network for pod "dask-scottyhq-527e3b25-ag48fq": NetworkPlugin cni failed to set up pod "dask-scottyhq-527e3b25-ag48fq_esip-prod" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused", failed to clean up sandbox container "77c187a18ff7974198302f452bfb89cf126689939ff7fe72bb4598c55ea4d360" network for pod "dask-scottyhq-527e3b25-ag48fq": NetworkPlugin cni failed to teardown pod "dask-scottyhq-527e3b25-ag48fq_esip-prod" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"]then it becomes a kubernetes autoscaler problem (see my first comment) where "the node cannot be removed because the pod is not replicated". This seems relevant kubernetes/autoscaler#351
TomAugspurger
jacobtomlinson
tjcrone
jhamman
dask-worker nodes were left running on the aws clusters even after user-notebook pods and nodes shut down. Here are some relevant kubectl outputs (dask worker node running for 9 days!)
kubectl get pods --all-namespaces -o widekubectl describe pod dask-rsignell-usgs-0b2bf9f0-1kc48v -n esip-prodkubectl get nodes -o widekubectl logs cluster-autoscaler-6575548656-74955 | grep daskand finally,
Originally posted by @scottyhq in https://github.com/pangeo-data/pangeo/issues/712#issuecomment-528165345