kubelet-executor hangs on DCOS CM.3, under load

mesosphere / kubernetes-mesos

A Kubernetes Framework for Apache Mesos

637 stars 92 forks source link

kubelet-executor hangs on DCOS CM.3, under load #664

Open jdef opened 8 years ago

jdef commented 8 years ago

xref https://github.com/kubernetes/kubernetes/issues/17525

jdef commented 8 years ago

data points:

looks like coreos ships docker 1.8.3 starting with 833.0.0
latest CM.x image builder uses coreos 766.4 on AWS

jdef commented 8 years ago

people talk about rebooting the node to resolve the problem. such a solution may be OK for dedicated k8s nodes, but not for mesos clusters running mixed workloads

jdef commented 8 years ago

@air this is a problem

jdef commented 8 years ago

tagged as "blocked" since we have to wait for dcos-image to upgrade to a version of coreos that uses a sane version of docker

spiffxp commented 8 years ago

@jdef FWIW in the xref'ed issue, I confirmed that I'm seeing this with docker 1.8.3; CoreOS alpha channel just moved to docker 1.9.1

jdef commented 8 years ago

@spiffxp thanks for clarifying. skimming issues too quickly!

ravilr commented 8 years ago

fyi, docker 1.9.1 also has issues reported with 'docker ps hanging' https://github.com/docker/docker/issues/17720#issuecomment-158640449