search

pusher / k8s-spot-rescheduler

Tries to move K8s Pods from on-demand to spot instances
Apache License 2.0
311 stars 42 forks source link

Pods created from Deployment are considered "non-replicated" #65

Open harshal-shah opened 4 years ago

harshal-shah commented 4 years ago

Trying to move workloads from on-demand instances to spot instances on a test cluster with guestbook installed with 20 frontend replicas.

The rescheduler fails to move pods with the following error message: 

k8s-spot-rescheduler-6b85d59bf5-gxlm7 k8s-spot-rescheduler E1004 11:37:47.920923       1 rescheduler.go:256] Failed to get pods for consideration: staging/frontend-69859f6796-jx7zs is not replicated
k8s-spot-rescheduler-6b85d59bf5-gxlm7 k8s-spot-rescheduler E1004 11:37:47.920929       1 rescheduler.go:256] Failed to get pods for consideration: staging/frontend-69859f6796-ppz8p is not replicated
k8s-spot-rescheduler-6b85d59bf5-gxlm7 k8s-spot-rescheduler E1004 11:37:47.920935       1 rescheduler.go:256] Failed to get pods for consideration: staging/frontend-69859f6796-fpzp6 is not replicated

adding the flag --delete-non-replicated-pods=true solves this but that is not the right way. Any suggestions on why this happens? Also is there any plans to support eviction of pods that contain a specific label? This would help to do a lot of controlled eviction

JoelSpeed commented 4 years ago

Hey @harshal-shah, could you please give a bit more detail about the particular environment you are running in, what version of K8s etc

I've not seen this bug manifested before so this is certainly an interesting one 🤔

Also is there any plans to support eviction of pods that contain a specific label? This would help to do a lot of controlled eviction

No plans, though PR welcome if you want to add the feature 🙂

harshal-shah commented 4 years ago

Sure, We are running on EKS, 

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

I deployed the app using the deploy dir and updated the image tag to v0.3.0

Let me know if you need any further information