Newly created osiris pods cannot activate already scaled to zero deployments

[hiroshi]

Environment:

• Kubernetes distribution (e.g. AKS, ACS Engine, GKE, EKS, etc.): GKE
• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-dispatcher", GitCommit:"f2a77f678d6baccda27740d700f6cba2754dfacf", GitTreeState:"clean", BuildDate:"2020-04-21T04:44:03Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.12-gke.2", GitCommit:"fb7add51f767aae42655d39972210dc1c5dbd4b3", GitTreeState:"clean", BuildDate:"2020-06-01T22:20:10Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}

• Osiris version (specify by SHA or semver):

  kubectl get deployment/osiris-osiris-edge-zeroscaler  -n osiris-system -oyaml | grep image:
      image: osiris.azurecr.io/osiris:2bf13af

  I think you should tell us how to find it.

• Install method (specify exact helm install command used): Sorry, I'm don't remember it. It will be great if install method is recorded as changecause or some annotations.

What happened?

• A osiris enabled deployment/service was scaled to zero after some idling.
• Osiris pods are re-scheduled by some reason (I'm using GKE preemptive nodes. So pods are constantly evicted).
• Make a request to the osiris enabled deployment/service, but it was not activated again.
• I see a log in the edge-zero-scaler:

  $ kubectl logs -f deployment/osiris-osiris-edge-zeroscaler  -n osiris-system
  ...
  I0804 11:44:34.857072       1 zeroscaler.go:77] Notified about new or updated Osiris-enabled deployment my-deployment in namespace default
  I0804 11:44:34.857077       1 zeroscaler.go:94] Osiris-enabled deployment my-deployment in namespace default is running zero replicas OR more than the minimum number of replicas; ensuring NO metrics collection
  ...

What you expected to happen? Osiris activate it again.

How to reproduce it (as minimally and precisely as possible): See the what happen section.

Anything else that we need to know? Please leave a comment if you have a question.

[hiroshi]

After I did kubectl scale deployment/my-deployment --replicas=1 for the deployment in the lifespan of newly created osiris pods, it scale-to-zero and activate the deployment again as expected.

Other workaround might be:

• Keep osiris pods alive in non-preemptive nodes (with pod affinity) as long as the issue resolved.

[hiroshi]

Sorry, it can be a mistake. Some pods of old deployments long gone was activated...