search

pusher / k8s-spot-rescheduler

Tries to move K8s Pods from on-demand to spot instances
Apache License 2.0
313 stars 43 forks source link

RBAC related error during draining #48

Closed kimxogus closed 6 years ago

kimxogus commented 6 years ago

I've got and error log from running k8s-spot-rescheduler created with helm stable chart.

I0613 08:22:25.885787       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-10-103-64.ap-northeast-2.compute.internal", UID:"e554192f-6e5b-11e8-83f1-02cebe856fe6", APIVersion:"v1", ResourceVersion:"4970345", FieldPath:""}): type: 'Normal' reason: 'Rescheduler' marked the node as drained/schedulable
E0613 08:22:25.887363       1 event.go:200] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ip-10-10-103-64.ap-northeast-2.compute.internal.1537a706144e99db", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"56289", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-10-103-64.ap-northeast-2.compute.internal", UID:"e554192f-6e5b-11e8-83f1-02cebe856fe6", APIVersion:"v1", ResourceVersion:"4970345", FieldPath:""}, Reason:"Rescheduler", Message:"marked the node as drained/schedulable", Source:v1.EventSource{Component:"rescheduler", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63664471043, loc:(*time.Location)(0x290da60)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbec052f074c9f2d8, ext:4923937149480, loc:(*time.Location)(0x290da60)}}, Count:4, Type:"Normal"}': 'events "ip-10-10-103-64.ap-northeast-2.compute.internal.1537a706144e99db" is forbidden: User "system:serviceaccount:kube-system:spot-rescheduler-k8s-spot-rescheduler" cannot patch events in the namespace "default"' (will not retry!)

It seems patch verb need to be added below https://github.com/pusher/k8s-spot-rescheduler/blob/master/deploy/clusterrole.yaml#L21, but I'm not a rbac expert, so not sure if it will fixes this issue. I added patch verb, but my clusters already already autoscaled so that action doesn't executed again.

mthssdrbrg commented 6 years ago

This will indeed fix the issue (AFAICT at least...) and was added in #52.