search

crossplane-contrib / provider-aws

Crossplane AWS Provider
Apache License 2.0
427 stars 367 forks source link

Huge number of AWS API calls #847

Closed robertgates55 closed 2 years ago

robertgates55 commented 2 years ago

Is there anything that can limit the number of calls being made to the AWS API? Since enabling this provider in a dev cluster our AWS bill jumped noticeably - which turned out to be because of crossplane calls being logged by cloudwatch. We were seeing 70qps hitting the AWS APIs, causing rate limiting!

Is that expected? Is there any way to dial down the chattiness?

chlunde commented 2 years ago

@robertgates55 there has been some issues which has triggered frequent reconciles, but you should not see 70 qps. Can you check

You can normally dial down the chattiness with a ControllerConfig, but 70 qps might be a different issue:

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: provider-aws-config
spec:
  args:
  - --poll=5m
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: crossplane-provider-aws
  namespace: crossplane-system
spec:
...
  controllerConfigRef:
    name: provider-aws-config
robertgates55 commented 2 years ago

I went through and cleaned up old resources and provider configs in the cluster, and think that seems to have stopped it - now back down to a much more sensible and manageable poll rate.

I think the way that a previous installation had been removed had left things in a funny state. Thanks for the confirmation - pointed me in the right direction just knowing that that was unexpected.

Closing now.

chlunde commented 2 years ago

@robertgates55 if you can provide a short summary of what resource types this was, and what kind of error states you saw, that could be helpful.

nagavijayan-nagarathinam commented 1 year ago

I am experiencing same issue like if managed resource count goes above 30, i started to get this error message on all resource, is there workaround for this issue ?

Warning CannotObserveExternalResource 18m managed/iam.aws.upbound.io/v1beta1, kind=policy cannot run refresh: refresh failed: Warning CannotObserveExternalResource 15m managed/iam.aws.upbound.io/v1beta1, kind=policy cannot run plan: plan failed: Warning CannotObserveExternalResource 11m (x3 over 29m) managed/iam.aws.upbound.io/v1beta1, kind=policy cannot run plan: plan failed: Warning CannotUpdateManagedResource 10m managed/iam.aws.upbound.io/v1beta1, kind=policy Operation cannot be fulfilled on policies.iam.aws.upbound.io "cplane-123-autoscaling-policy-vjv75-4": the object has been modified; please apply your changes to the latest version and try again Warning CannotObserveExternalResource 8m56s (x5 over 28m) managed/iam.aws.upbound.io/v1beta1, kind=policy cannot run plan: plan failed: Warning CannotObserveExternalResource 8m34s managed/iam.aws.upbound.io/v1beta1, kind=policy cannot run refresh: refresh failed: Warning CannotObserveExternalResource 8m29s (x3 over 20m) managed/iam.aws.upbound.io/v1beta1, kind=policy cannot run refresh: refresh failed:

Conditions: Last Transition Time: 2023-02-02T12:25:28Z Message: observe failed: cannot run plan: plan failed: Reason: ReconcileError Status: False Type: Synced Last Transition Time: 2023-02-02T11:55:57Z Reason: Available Status: True Type: Ready Last Transition Time: 2023-02-02T11:54:36Z Reason: Finished Status: True Type: AsyncOperation Last Transition Time: 2023-02-02T11:54:36Z Reason: Success Status: True Type: LastAsyncOperation

controller config `spec: args:

Crossplane-v1.10.0 upbound/provider-aws - v0.28.0

akundrock commented 1 year ago

Not a fix, but here are the related flags with descriptions from the code base in case anyone else finds this issue in the future. https://github.com/crossplane-contrib/provider-aws/blob/master/cmd/provider/main.go#L50