Open jlewi opened 4 years ago
Issue-Label Bot is automatically applying the label kind/bug
to this issue, with a confidence of 0.89. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
I also observed kaniko jobs launched by skaffold getting stuck. Symptom was kaniko container started but no logs were emitted.
I kicked the node metadata servers
kubectl -n kube-system delete pods -l k8s-app=gke-metadata-server
At that appears to have caused things to start.
I'm running 1.14.9-gke.2
I'm new to kubeflow and this has caused me all kinds of grief. On top of the other issues with kfctl delete and reapply this really makes things seem unusable. Glad to see it's a Google problem though ;-)
See attached logs. Some of the worker pods for the label microservice are returning 500s when they first try to contact the metadata server.
It appears to be able to get credentials though since it is able to verify the pubsub subscription exists. Takes about 4 minutes.
label-bot-worker-5c8967dc7c-rgv9b.pod.logs.txt
Not seeing the same errors reported in kubeflow/kubeflow#4607 in the metadata server logs.
gke-metadata-server.logs.txt
Note I think a lot of the K8s errors in the logs are because the master was temporarily unavailable while it was upgrading.
My cluster is 1.14.9-gke.2