pusher / k8s-spot-rescheduler

Tries to move K8s Pods from on-demand to spot instances
Apache License 2.0
311 stars 42 forks source link

Is the moving of pods logged? #58

Open max-rocket-internet opened 5 years ago

max-rocket-internet commented 5 years ago

Because all I see is this:

I0502 14:44:16.294689       1 rescheduler.go:247] No nodes to process.
I0502 14:44:26.443325       1 rescheduler.go:247] No nodes to process.
I0502 14:44:36.574006       1 rescheduler.go:247] No nodes to process.
I0502 14:44:46.724541       1 rescheduler.go:247] No nodes to process.
I0502 14:44:56.940652       1 rescheduler.go:247] No nodes to process.
I0502 14:45:07.118794       1 rescheduler.go:247] No nodes to process.
I0502 14:45:17.301454       1 rescheduler.go:247] No nodes to process.
I0502 14:45:27.464235       1 rescheduler.go:247] No nodes to process.

And sometimes this:

I0502 14:46:28.288639       1 rescheduler.go:205] Waiting for unschedulable pods to be scheduled.
sercanacar commented 5 years ago

@max-rocket-internet did you get to the bottom of this?

max-rocket-internet commented 5 years ago

It's not logged but it should definitely be.

JoelSpeed commented 5 years ago

Hey @max-rocket-internet and @sercanacar, which log setting do you have? The rescheduler uses glog and we typically run with level 2 -v=2 at @pusher. You are right that there is not specific log line for each individual pod being evicted, though i don't see why we couldn't add one, probably somewhere within this method https://github.com/pusher/k8s-spot-rescheduler/blob/b46cd362a3fb07159226b8d5a9872ad6f0de8dab/scaler/scaler.go#L71

max-rocket-internet commented 5 years ago

which log setting do you have?

I'm just using the chart: https://github.com/helm/charts/tree/master/stable/k8s-spot-termination-handler

there is not specific log line for each individual pod being evicted

I would say this is very important to log at default log level. Much more important than No nodes to process. 😄

JoelSpeed commented 5 years ago

Yeah you're probably right... Levelled logging decisions are hard 😅

aimanparvaiz commented 4 years ago

Any update on this please?