search

kubecost / cluster-turndown

Automated turndown of Kubernetes clusters on specific schedules.
Apache License 2.0
259 stars 23 forks source link

Turndown scaledown not works ( EKS) #42

Open billmetangmo opened 2 years ago

billmetangmo commented 2 years ago

Hello,

I wanted to use turndown to stop my dev cluster during off hours. It's an EKS cluster. I succeeded to create the example schedule . However when the schedule is supposed to happen, it failed with the following error :

E0525 11:47:00.907323       1 schedulecontroller.go:190] TurndownSchedule 'example-schedule' in work queue no longer exists
I0525 11:47:14.319434       1 namedlogger.go:24] [TurndownScheduler] Schedule Created: &{Current:scaledown ScaleDownID:1adae7c3-6f07-4837-baec-a5ed8a8efad0 ScaleDownTime:2022-05-25 11:55:00 +0000 UTC ScaleDownMetadata:map[repeat:daily type:scaledown] ScaleUpID:afcc2f9c-4324-4a29-880d-a532546857ef ScaleUpTime:2022-05-25 12:45:00 +0000 UTC ScaleUpMetadata:map[repeat:daily type:scaleup]}
I0525 11:47:14.339321       1 event.go:282] Event(v1.ObjectReference{Kind:"TurndownSchedule", Namespace:"", Name:"example-schedule", UID:"a6b0311e-8430-423a-ae1f-26935b7279ee", APIVersion:"kubecost.k8s.io/v1alpha1", ResourceVersion:"1476199", FieldPath:""}): type: 'Normal' reason: 'ScheduleTurndownSuccess' Successfully scheduled turndown
I0525 11:55:00.000175       1 turndownscheduler.go:404] -- Scale Down --
I0525 11:55:00.009267       1 namedlogger.go:24] [TurndownScheduler] Turndown Pod does not exist on expected host node. Preparing environment...
I0525 11:55:00.009306       1 namedlogger.go:24] [Turndown] Creating or Getting the Target Host Node...
I0525 11:55:00.419785       1 namedlogger.go:24] [MasterlessStrategy] Finite node backed cluster. Creating singleton nodepool for turndown.
I0525 11:55:00.665213       1 namedlogger.go:48] [Error] Failed to prepare current turndown environment. Cancelling. Err=AccessDeniedException: 
    status code: 403, request id: e5777622-11c8-4beb-aed1-1c1073c1925d
I0525 11:55:00.665544       1 scheduler.go:188] Job was cancelled: afcc2f9c-4324-4a29-880d-a532546857ef
I0525 11:55:00.685926       1 namedlogger.go:24] [TurndownScheduler] Turndown Schedule Successfully Cancelled

The error seems to be an "Access Denied" however i created the credentials with the suggested policy https://github.com/kubecost/cluster-turndown#eks--aws-kops-setup Do you have any idea ?

Regards