search

kubecost / cluster-turndown

Automated turndown of Kubernetes clusters on specific schedules.
Apache License 2.0
259 stars 23 forks source link

turndown sets own deployment to 0 #22

Closed chap-dr closed 4 years ago

chap-dr commented 4 years ago

Im running this on a cluster spun up by our current GKE terraform files, only deployment running on the cluster is teardown, the cluster currently has two node pools both with autoscaling enabled.

The very first thing turndown does, is turn its own deployment to 0, Im assuming this is why the turnup i have configured a 20 minutes after after turndown does not trigger. (poc schedule for seeing how turndown works)

This is the controller log:

cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:22:01.122236       1 validator.go:39] Validating Provider
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:22:01.127017       1 namedlogger.go:24] [GKEClusterProvider] Loading node pools for: [ProjectID: xxxxxxxx, Zone: xxxxxxx, ClusterID: xxxxxxx]
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:22:01.345562       1 reflector.go:122] Starting reflector *v1alpha1.TurndownSchedule (30s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:22:01.345602       1 reflector.go:160] Listing and watching *v1alpha1.TurndownSchedule from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:22:01.345705       1 schedulecontroller.go:109] Starting TurndownSchedule controller
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.345558       1 turndownscheduler.go:404] -- Scale Down --
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.356117       1 namedlogger.go:24] [TurndownScheduler] Already running on correct turndown host node. No need to setup environment.
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.356152       1 namedlogger.go:24] [Turndown] Scaling Down Cluster Now
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.364449       1 namedlogger.go:24] [GKEClusterProvider] Loading node pools for: [ProjectID: xxxxxxx, Zone: xxxxxx, ClusterID: xxxxxxx]
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.398052       1 namedlogger.go:24] [Turndown] Found Cluster-AutoScaler. Flattening Cluster...
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.398147       1 namedlogger.go:32]   [Flattener] Starting to Flatten All Deployments...
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:01.589561       1 namedlogger.go:32]   [Flattener] Starting to Flatten All DaemonSets...
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:02.010862       1 namedlogger.go:32]   [Flattener] Starting to Suspend All Jobs...
cluster-turndown-5c77649bff-67tng cluster-turndown I0607 18:23:02.021614       1 namedlogger.go:24] [Turndown] Resizing all non-autoscaling node groups to 0...

Any input on why this is happening?

mbolt35 commented 4 years ago

Hey @chap-dr! Thanks for the input. This behavior is definitely not expected (flattening it's own deployment). The flattener does have a blacklist, and for a while that blacklist included its own deployment. However, it does appear that bit of code no longer exists. I'm going to investigate a bit more, and will hopefully have a fix here soon. Thanks again for the report!

mbolt35 commented 4 years ago

Released in https://github.com/kubecost/cluster-turndown/releases/tag/v1.2.1