AWS API Call Rate limit exceeded

[riponbanik]

Is there way to reduce the api call limit to prevent the following? It looks like sleep time is 10s, so every minute it makes 6 api calls.

W0207 23:20:14.956666 1 handler.go:28] Can not get instance status and auto scaling group name, will try again: Throttling: Rate exceeded status code: 400, request id: e29ab693-33b8-401a-b73a-85567b8559bd

[pawelprazak]

It should be possible to override the sleep time in a config file, it is not an advertised feature at the moment, but it exists, you can pass --config param.

TODO:

• document --config and the config schema
• add a flag for sleep time

[pawelprazak]

I've added --loop-sleep-time flag, will release a version soon.

[pawelprazak]

https://github.com/VirtusLab/kubedrainer/releases/tag/v0.0.10 was released

[riponbanik]

@pawelprazak Thanks. So need to pass this as args in deployment manifest?

[pawelprazak]

yes, adding flags to the manifest on the args field, should work

Entrypoint in the Dockerfile is set to kubedrainer serve, and k8s translates this to the command field in the manifest, so yeah, adding flags to args in manifest should work

[mtaylor98]

Hello all - we ran into an issue setting this flag in our Kubernete's manifest and found that the flag value is being set but not passed through to the running container parsed settings properly, see output of debug logs below (removed sensitive information as necessary):

{"level":"info","time":1623442470,"message":"Running as server"}
{"level":"debug","time":1623442470,"message":"All keys: [insecure-skip-tls-verify selector region timeout profile ignore-daemonsets force context kubeconfig request-timeout help server cluster user cache-dir drain-delay loop-sleep-time grace-period node debug shutdown-sleep pod-selector client-certificate token client-key instance-id delete-local-data certificate-authority config]"}
{"level":"debug","time":1623442470,"message":"All settings: map[cache-dir:/.kube/http-cache certificate-authority: client-certificate: client-key: cluster: config: context: debug:true delete-local-data:true drain-delay:0s force:false grace-period:-1 help:false ignore-daemonsets:true insecure-skip-tls-verify:false instance-id: kubeconfig: loop-sleep-time:60s node:xxxxxxx pod-selector: profile: region: request-timeout:0 selector: server: shutdown-sleep:6m0s timeout:1m0s token: user:]"}
{"level":"debug","time":1623442470,"message":"'cache-dir' -> flag: '/.kube/http-cache' | setting: '/.kube/http-cache'"}
{"level":"debug","time":1623442470,"message":"'certificate-authority' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'client-certificate' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'client-key' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'cluster' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'config' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'context' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'debug' -> flag: 'true' | setting: 'true'"}
{"level":"debug","time":1623442470,"message":"'delete-local-data' -> flag: 'true' | setting: 'true'"}
{"level":"debug","time":1623442470,"message":"'drain-delay' -> flag: '0s' | setting: '0s'"}
{"level":"debug","time":1623442470,"message":"'force' -> flag: 'false' | setting: 'false'"}
{"level":"debug","time":1623442470,"message":"'grace-period' -> flag: '-1' | setting: '-1'"}
{"level":"debug","time":1623442470,"message":"'help' -> flag: 'false' | setting: 'false'"}
{"level":"debug","time":1623442470,"message":"'ignore-daemonsets' -> flag: 'true' | setting: 'true'"}
{"level":"debug","time":1623442470,"message":"'insecure-skip-tls-verify' -> flag: 'false' | setting: 'false'"}
{"level":"debug","time":1623442470,"message":"'instance-id' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'kubeconfig' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'loop-sleep-time' -> flag: '60s' | setting: '60s'"}
{"level":"debug","time":1623442470,"message":"'node' -> flag: '' | setting: 'xxxxxxxxxxx'"}
{"level":"debug","time":1623442470,"message":"'pod-selector' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'profile' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'region' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'request-timeout' -> flag: '0' | setting: '0'"}
{"level":"debug","time":1623442470,"message":"'selector' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'server' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'shutdown-sleep' -> flag: '6m0s' | setting: '6m0s'"}
{"level":"debug","time":1623442470,"message":"'timeout' -> flag: '1m0s' | setting: '1m0s'"}
{"level":"debug","time":1623442470,"message":"'token' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"'user' -> flag: '' | setting: ''"}
{"level":"debug","time":1623442470,"message":"Settings: {Kubernetes:{ConfigFlags:{CacheDir:/.kube/http-cache KubeConfig: ClusterName: AuthInfoName: Context: Namespace:<nil> APIServer: Insecure:false CertFile: KeyFile: CAFile: BearerToken: Impersonate:<nil> ImpersonateGroup:<nil> Username:<nil> Password:<nil> Timeout:0 clientConfig:<nil> lock:{state:0 sema:0} usePersistentConfig:true}} Drainer:{Node: Force:false GracePeriodSeconds:-1 IgnoreAllDaemonSets:true Timeout:1m0s DeleteLocalData:true Selector: PodSelector: DrainDelay:0s} AWS:{InstanceID: Region: Profile: LoopSleepTime:10s ShutdownSleep:6m0s ForceLoopBreak:false}}"}
{"level":"debug","time":1623442470,"message":"Parsed settings: {Kubernetes:{ConfigFlags:{CacheDir:/.kube/http-cache KubeConfig: ClusterName: AuthInfoName: Context: Namespace:<nil> APIServer: Insecure:false CertFile: KeyFile: CAFile: BearerToken: Impersonate:<nil> ImpersonateGroup:<nil> Username:<nil> Password:<nil> Timeout:0 clientConfig:<nil> lock:{state:0 sema:0} usePersistentConfig:true}} Drainer:{Node:xxxxxxxxxxxxxxxx Force:false GracePeriodSeconds:-1 IgnoreAllDaemonSets:true Timeout:1m0s DeleteLocalData:true Selector: PodSelector: DrainDelay:0s} AWS:{InstanceID: Region: Profile: LoopSleepTime:10s ShutdownSleep:6m0s ForceLoopBreak:false}}"}
{"level":"debug","time":1623442470,"message":"Context: "}
{"level":"debug","time":1623442470,"message":"Configured Host: https://xxxxxxxxx"}
{"level":"debug","time":1623442470,"message":"Configured AuthProvider: <nil>"}
{"level":"debug","time":1623442470,"message":"Configured ExecProvider: <nil>"}
{"level":"debug","time":1623442470,"message":"Server version: v1.19.8"}
{"level":"debug","time":1623442470,"message":"Getting node information"}
{"level":"debug","time":1623442470,"message":"Using default AWS API credentials profile"}
{"level":"info","time":1623442471,"message":"Running node drainer on node 'xxxxxxxxxxx' on instance 'xxxxxxxx' in region 'xxxxxxx' and profile 'default'"}
{"level":"info","time":1623442471,"message":"Sleeping 10s seconds"}

The container args are being set as follows:

        - name: kubedrainer
          image: quay.io/virtuslab/kubedrainer:v0.0.10
          args: ["--debug", "--loop-sleep-time", "60s"]

Let us know how we can properly override this value and if there is any other information that we can provide. Thank you!

[mannkind]

After a bit more testing/digging... It looks like the option parsing issue was fixed in #14. A new release -- 0.0.11 or whatever -- needs to be made before the option will work as expected.