apache / openwhisk-deploy-kube

The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
https://openwhisk.apache.org/
Apache License 2.0
297 stars 231 forks source link

prewarm runtime pods are not completely deleted #652

Closed ningyougang closed 3 years ago

ningyougang commented 3 years ago

If the number of prewarm runtime pods is small, after executed helm delete(or uninstall) xxx, all prewarm runtime pods are deleted completely.

But if the number of prewarm runtime pods is big, after executed helm delete(or uninstall) xxx, some prewarm runtime pods are not deleted completely.(e.g. maybe 1/3 prewarm pods are deleted, 2/3 prewarm pods are not deleted, still Running)

The reason is After analized the kubernets-client-4.4.2.jar source codes, when invoke below codes

        kubeRestClient
          .inNamespace(kubeRestClient.getNamespace)
          .pods()
          .withLabels(labels.asJava)
          .delete()

Finally, it will invoke below codes image We can see, it deletes the runtime pods in a for statement, on the other hand, h.delete(client, ...) method is a sync method, due to some configuration of timeout is low, it leads to can't delete all runtime pods(When timeout happens, above for statement is not executed finished)

I will open 2 prs to solve this issue

if increase the timeout value, can delete all runtime pods successfully, e.g. values.yaml

invoker:
  runtimeDeleteTimeout: "5 minutes"
akka:
  actorSystemTerminateTimeout: "5 m"