Huge number of API Calls

cest-pas-faux commented 2 years ago

What happened?

Hello, we found out that most of the calls logged by Cloudtrail are from provider-aws, and it significantly impacts our Guarduty bill.

After some digging we tried the solution from #847 and added --poll=5m, which drastically reduced the calls number overall, however we think that the call rate is still very high, considering our setup.

Here is the summary of the last hour calls, about 80% of AssumeRole, and 100%-ish of ModifyDBInstance and AddTagsToResource are from crossplane-provider-aws.

'AddTagsToResource': 1812,
'AssignPrivateIpAddresses': 59,
'AssumeRole': 8662,
'AssumeRoleWithWebIdentity': 55,
'ModifyDBInstance': 2015,
'ModifyDBParameterGroup': 66,
'PutBucketAcl': 26,
'PutBucketPolicy': 26,
'PutEvaluations': 77,
'UnassignPrivateIpAddresses': 59,
'UpdateInstanceInformation': 275

We have a EKS cluster and are starting to migrate our clients from metal and we currently have created a small amount of RDS instances (we expect to have at least 3x this number), so we are affraid of an exponential increase of the calls to AWS in the next weeks :

 $ kubectl get rdsinstances.database.aws.crossplane.io -A | wc -l
26 # all are Ready, only ONE is not Synced

Playing around with max-reconcile-rate didn't change anything and sync default to 1 hour shouldn't have much impact.

Is there any other ways to reduce the calls number ?
We could increase again the poll argument but we are not sure if this is a good idea to reduce the crossplane reactivity
Why the provider keeps rewriting the tags using AddTagsToResource ? (there is no changes to the tags once the instance is created)
2000 ModifyDBInstance per hour for 26 RDS Instances seems a bit intense IMO

Thanks in advance and let me know if you need more informations.

How can we reproduce it?

Standard install of crossplane + provider-aws, create a bunch of rds instances and check cloudtrail

What environment did it happen in?

Crossplane version: 1.7.0
provider-aws : 0.26.0

Kubernetes version (use kubectl version)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.13-eks-84b4fe6", GitCommit:"e1318dce57b3e319a2e3fecf343677d1c4d4aa75", GitTreeState:"clean", BuildDate:"2022-06-09T18:22:07Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes distribution : EKS 1.21.5

chlunde commented 2 years ago

Might be related to #828 - but I don't understand how you have only so many AssumeRole and so few AssumeRoleWithWebIdentity. What kind of ProviderConfig do you use?

Could you also take a look into what kind of resource is tagged with AddTagsToResource? Can you then do kubectl get -w -o yaml on the managed resource and see what changes?

Re. RDSInstance, I wonder if you could upgrade to 0.29 to check if the situation has improved. The only issue I know of now is if you use UpperCase in maintenance/backup window, AWS uses lower case.

cest-pas-faux commented 2 years ago

Hello, thanks for your answer, let me explain more of our setup :

What kind of ProviderConfig do you use?

We have a multi-tenant cluster and to avoid any configuration mistakes from our clients, we made our own Kinds available as Composition using CompositeResourceDefinition, there is one for MySQL (Kind : MysqlInstance) and one for PostgreSQL (Kind : PostgresInstance).

You can find below the ones for MySQL (postgres is basically the same, only the values changes), note that I replaced our company name by "company".

MysqlInstance manifest example

```Yaml ## Note : the tags are added by a Kyverno mutation ## only when the operation is CREATE for the object ## same for the subnets ids and vpc id apiVersion: database.company.org/v1alpha1 kind: MysqlInstance metadata: name: sample-app-staging-rdsmysql-mysqlinstance namespace: sample-app-staging spec: compositionRef: name: xmysqlinstances.aws.database.company.org compositionUpdatePolicy: Automatic parameters: dbParameterGroup: forProvider: dbParameterGroupFamily: mysql8.0 description: dbParameterGroup parameters: - applyMethod: immediate parameterName: transaction_isolation parameterValue: READ-COMMITTED region: us-east-1 tags: - key: kubernetes_cluster value: eks01-cluster - key: kubernetes_namespace value: sample-app-staging dbSubnetGroup: forProvider: description: dbSubnetGroup region: us-east-1 subnetIds: - subnet-xxxxxxxxxxxxxxxxx # modified - subnet-yyyyyyyyyyyyyyyyy # modified - subnet-zzzzzzzzzzzzzzzzz # modified tags: - key: kubernetes_cluster value: eks01-cluster - key: kubernetes_namespace value: sample-app-staging rdsInstance: forProvider: allocatedStorage: 10 applyModificationsImmediately: true autoMinorVersionUpgrade: true backupRetentionPeriod: 0 copyTagsToSnapshot: true dbInstanceClass: db.t4g.small dbName: "" dbSubnetGroupNameSelector: matchControllerRef: true deletionProtection: false engine: mysql engineVersion: 8.0.28 masterUsername: root maxAllocatedStorage: 20 multiAZ: false region: us-east-1 skipFinalSnapshotBeforeDeletion: true storageEncrypted: true storageType: gp2 tags: - key: kubernetes_cluster value: eks01-cluster - key: kubernetes_namespace value: sample-app-staging vpcSecurityGroupIDSelector: matchControllerRef: true securityGroup: forProvider: description: securityGroup egress: - fromPort: 3306 ipProtocol: tcp ipRanges: - cidrIp: 10.0.0.0/8 toPort: 3306 groupName: "" ingress: - fromPort: 3306 ipProtocol: tcp ipRanges: - cidrIp: 10.0.0.0/8 toPort: 3306 region: us-east-1 tags: - key: kubernetes_cluster value: eks01-cluster - key: kubernetes_namespace value: sample-app-staging vpcId: vpc-0000000000000000 # modified resourceRef: apiVersion: database.company.org/v1alpha1 kind: XMysqlInstance name: sample-app-staging-rdsmysql-mysqlinstance-5q9dz writeConnectionSecretToRef: name: sample-app-staging-rdsmysql-db ```

Composition

```Yaml apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: xmysqlinstances.aws.database.company.org labels: provider: aws service: rds spec: writeConnectionSecretsToNamespace: crossplane-system compositeTypeRef: apiVersion: database.company.org/v1alpha1 kind: XMysqlInstance resources: - name: db-subnet-group base: apiVersion: database.aws.crossplane.io/v1beta1 kind: DBSubnetGroup spec: forProvider: {} providerConfigRef: name: providerconfig-aws patches: - fromFieldPath: "spec.parameters.dbSubnetGroup.forProvider" toFieldPath: "spec.forProvider" - type: ToCompositeFieldPath fromFieldPath: status.conditions toFieldPath: status.components.dbSubnetGroup - name: security-group base: apiVersion: ec2.aws.crossplane.io/v1beta1 kind: SecurityGroup metadata: labels: type: securityGroup spec: forProvider: {} providerConfigRef: name: providerconfig-aws patches: - fromFieldPath: "spec.parameters.securityGroup.forProvider" toFieldPath: "spec.forProvider" - fromFieldPath: "metadata.name" toFieldPath: "spec.forProvider.groupName" - type: ToCompositeFieldPath fromFieldPath: status.conditions toFieldPath: status.components.securityGroup - name: dbParameterGroupName base: apiVersion: rds.aws.crossplane.io/v1alpha1 kind: DBParameterGroup metadata: labels: type: dbParameterGroup spec: forProvider: {} providerConfigRef: name: providerconfig-aws patches: - fromFieldPath: "spec.parameters.dbParameterGroup.forProvider" toFieldPath: "spec.forProvider" - type: ToCompositeFieldPath fromFieldPath: status.conditions toFieldPath: status.components.dbParameterGroup - name: rdsinstance base: apiVersion: database.aws.crossplane.io/v1beta1 kind: RDSInstance spec: forProvider: writeConnectionSecretToRef: namespace: crossplane-system providerConfigRef: name: providerconfig-aws patches: - fromFieldPath: "spec.parameters.rdsInstance.forProvider" toFieldPath: "spec.forProvider" - fromFieldPath: "metadata.uid" toFieldPath: "spec.writeConnectionSecretToRef.name" transforms: - type: string string: fmt: "%s-mysql" - fromFieldPath: "spec.resourceRefs[2].name" toFieldPath: "spec.forProvider.dbParameterGroupName" - type: ToCompositeFieldPath fromFieldPath: status.conditions toFieldPath: status.components.rdsInstance - type: ToCompositeFieldPath fromFieldPath: "metadata.name" toFieldPath: "status.databaseInstanceName" connectionDetails: - fromConnectionSecretKey: username - fromConnectionSecretKey: password - fromConnectionSecretKey: endpoint - fromConnectionSecretKey: port - fromFieldPath: spec.forProvider.dbName name: dbName ```

CompositeResourceDefinition

```Yaml apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: xmysqlinstances.database.company.org spec: group: database.company.org names: kind: XMysqlInstance plural: xmysqlinstances claimNames: kind: MysqlInstance plural: mysqlinstances shortNames: - mysql connectionSecretKeys: - username - password - endpoint - port - dbName versions: - name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: parameters: type: object properties: # ------------------------------------------------------------- dbSubnetGroup: type: object properties: forProvider: required: - description type: object description: dbSubnetGroup properties: description: description: The description for the DB subnet group. type: string region: description: Region is the region you'd like your DBSubnetGroup to be created in. type: string subnetIdRefs: description: SubnetIDRefs is a set of references that each retrieve the subnetID from the referenced Subnet items: description: A Reference to a named object. properties: name: description: Name of the referenced object. type: string required: - name type: object type: array subnetIdSelector: description: SubnetIDSelector selects a set of references that each retrieve the subnetID from the referenced Subnet properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object subnetIds: description: The EC2 Subnet IDs for the DB subnet group. items: type: string type: array tags: description: A list of tags. For more information, see Tagging Amazon RDS Resources (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Tagging.html) in the Amazon RDS User Guide. items: description: Tag is a metadata assigned to an Amazon RDS resource consisting of a key-value pair. Please also see https://docs.aws.amazon.com/goto/WebAPI/rds-2014-10-31/Tag properties: key: description: 'A key is the required name of the tag. The string value can be from 1 to 128 Unicode characters in length and can''t be prefixed with "aws:" or "rds:". The string can only contain only the set of Unicode letters, digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'', ''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").' type: string value: description: 'A value is the optional value of the tag. The string value can be from 1 to 256 Unicode characters in length and can''t be prefixed with "aws:" or "rds:". The string can only contain only the set of Unicode letters, digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'', ''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").' type: string type: object type: array # ------------------------------------------------------------- securityGroup: type: object properties: forProvider: required: - description - groupName type: object description: SecurityGroupParameters define the desired state of an AWS VPC Security Group. properties: description: description: A description of the security group. type: string egress: description: '[EC2-VPC] One or more outbound rules associated with the security group.' items: description: IPPermission Describes a set of permissions for a security group rule. properties: fromPort: description: The start of port range for the TCP and UDP protocols, or an ICMP/ICMPv6 type number. A value of -1 indicates all ICMP/ICMPv6 types. If you specify all ICMP/ICMPv6 types, you must specify all codes. format: int32 type: integer ipProtocol: description: "The IP protocol name (tcp, udp, icmp, icmpv6) or number (see Protocol Numbers (http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml)). \n [VPC only] Use -1 to specify all protocols. When authorizing security group rules, specifying -1 or a protocol number other than tcp, udp, icmp, or icmpv6 allows traffic on all ports, regardless of any port range you specify. For tcp, udp, and icmp, you must specify a port range. For icmpv6, the port range is optional; if you omit the port range, traffic for all types and codes is allowed." type: string ipRanges: description: The IPv4 ranges. items: description: IPRange describes an IPv4 range. properties: cidrIp: description: The IPv4 CIDR range. You can either specify a CIDR range or a source security group, not both. To specify a single IPv4 address, use the /32 prefix length. type: string description: description: "A description for the security group rule that references this IPv4 address range. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*" type: string required: - cidrIp type: object type: array ipv6Ranges: description: "The IPv6 ranges. \n [VPC only]" items: description: IPv6Range describes an IPv6 range. properties: cidrIPv6: description: The IPv6 CIDR range. You can either specify a CIDR range or a source security group, not both. To specify a single IPv6 address, use the /128 prefix length. type: string description: description: "A description for the security group rule that references this IPv6 address range. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*" type: string required: - cidrIPv6 type: object type: array prefixListIds: description: "PrefixListIDs for an AWS service. With outbound rules, this is the AWS service to access through a VPC endpoint from instances associated with the security group. \n [VPC only]" items: description: PrefixListID describes a prefix list ID. properties: description: description: "A description for the security group rule that references this prefix list ID. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*" type: string prefixListId: description: The ID of the prefix. type: string required: - prefixListId type: object type: array toPort: description: The end of port range for the TCP and UDP protocols, or an ICMP/ICMPv6 code. A value of -1 indicates all ICMP/ICMPv6 codes. If you specify all ICMP/ICMPv6 types, you must specify all codes. format: int32 type: integer userIdGroupPairs: description: UserIDGroupPairs are the source security group and AWS account ID pairs. It contains one or more accounts and security groups to allow flows from security groups of other accounts. items: description: UserIDGroupPair describes a security group and AWS account ID pair. properties: description: description: "A description for the security group rule that references this user ID group pair. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*" type: string groupId: description: The ID of the security group. type: string groupIdRef: description: GroupIDRef reference a security group to retrieve its GroupID properties: name: description: Name of the referenced object. type: string required: - name type: object groupIdSelector: description: GroupIDSelector selects reference to a security group to retrieve its GroupID properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object groupName: description: "The name of the security group. In a request, use this parameter for a security group in EC2-Classic or a default VPC only. For a security group in a nondefault VPC, use the security group ID. \n For a referenced security group in another VPC, this value is not returned if the referenced security group is deleted." type: string userId: description: "The ID of an AWS account. \n For a referenced security group in another VPC, the account ID of the referenced security group is returned in the response. If the referenced security group is deleted, this value is not returned. \n [EC2-Classic] Required when adding or removing rules that reference a security group in another AWS account." type: string vpcId: description: The ID of the VPC for the referenced security group, if applicable. type: string vpcIdRef: description: VPCIDRef reference a VPC to retrieve its vpcId properties: name: description: Name of the referenced object. type: string required: - name type: object vpcIdSelector: description: VPCIDSelector selects reference to a VPC to retrieve its vpcId properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object vpcPeeringConnectionId: description: The ID of the VPC peering connection, if applicable. type: string type: object type: array required: - ipProtocol type: object type: array groupName: description: The name of the security group. type: string ingress: description: One or more inbound rules associated with the security group. items: description: IPPermission Describes a set of permissions for a security group rule. properties: fromPort: description: The start of port range for the TCP and UDP protocols, or an ICMP/ICMPv6 type number. A value of -1 indicates all ICMP/ICMPv6 types. If you specify all ICMP/ICMPv6 types, you must specify all codes. format: int32 type: integer ipProtocol: description: "The IP protocol name (tcp, udp, icmp, icmpv6) or number (see Protocol Numbers (http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml)). \n [VPC only] Use -1 to specify all protocols. When authorizing security group rules, specifying -1 or a protocol number other than tcp, udp, icmp, or icmpv6 allows traffic on all ports, regardless of any port range you specify. For tcp, udp, and icmp, you must specify a port range. For icmpv6, the port range is optional; if you omit the port range, traffic for all types and codes is allowed." type: string ipRanges: description: The IPv4 ranges. items: description: IPRange describes an IPv4 range. properties: cidrIp: description: The IPv4 CIDR range. You can either specify a CIDR range or a source security group, not both. To specify a single IPv4 address, use the /32 prefix length. type: string description: description: "A description for the security group rule that references this IPv4 address range. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*" type: string required: - cidrIp type: object type: array ipv6Ranges: description: "The IPv6 ranges. \n [VPC only]" items: description: IPv6Range describes an IPv6 range. properties: cidrIPv6: description: The IPv6 CIDR range. You can either specify a CIDR range or a source security group, not both. To specify a single IPv6 address, use the /128 prefix length. type: string description: description: "A description for the security group rule that references this IPv6 address range. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*" type: string required: - cidrIPv6 type: object type: array prefixListIds: description: "PrefixListIDs for an AWS service. With outbound rules, this is the AWS service to access through a VPC endpoint from instances associated with the security group. \n [VPC only]" items: description: PrefixListID describes a prefix list ID. properties: description: description: "A description for the security group rule that references this prefix list ID. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*" type: string prefixListId: description: The ID of the prefix. type: string required: - prefixListId type: object type: array toPort: description: The end of port range for the TCP and UDP protocols, or an ICMP/ICMPv6 code. A value of -1 indicates all ICMP/ICMPv6 codes. If you specify all ICMP/ICMPv6 types, you must specify all codes. format: int32 type: integer userIdGroupPairs: description: UserIDGroupPairs are the source security group and AWS account ID pairs. It contains one or more accounts and security groups to allow flows from security groups of other accounts. items: description: UserIDGroupPair describes a security group and AWS account ID pair. properties: description: description: "A description for the security group rule that references this user ID group pair. \n Constraints: Up to 255 characters in length. Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*" type: string groupId: description: The ID of the security group. type: string groupIdRef: description: GroupIDRef reference a security group to retrieve its GroupID properties: name: description: Name of the referenced object. type: string required: - name type: object groupIdSelector: description: GroupIDSelector selects reference to a security group to retrieve its GroupID properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object groupName: description: "The name of the security group. In a request, use this parameter for a security group in EC2-Classic or a default VPC only. For a security group in a nondefault VPC, use the security group ID. \n For a referenced security group in another VPC, this value is not returned if the referenced security group is deleted." type: string userId: description: "The ID of an AWS account. \n For a referenced security group in another VPC, the account ID of the referenced security group is returned in the response. If the referenced security group is deleted, this value is not returned. \n [EC2-Classic] Required when adding or removing rules that reference a security group in another AWS account." type: string vpcId: description: The ID of the VPC for the referenced security group, if applicable. type: string vpcIdRef: description: VPCIDRef reference a VPC to retrieve its vpcId properties: name: description: Name of the referenced object. type: string required: - name type: object vpcIdSelector: description: VPCIDSelector selects reference to a VPC to retrieve its vpcId properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object vpcPeeringConnectionId: description: The ID of the VPC peering connection, if applicable. type: string type: object type: array required: - ipProtocol type: object type: array region: description: Region is the region you'd like your SecurityGroup to be created in. type: string tags: description: Tags represents to current ec2 tags. items: description: Tag defines a tag properties: key: description: Key is the name of the tag. type: string value: description: Value is the value of the tag. type: string required: - key - value type: object type: array vpcId: description: VPCID is the ID of the VPC. type: string vpcIdRef: description: VPCIDRef references a VPC to and retrieves its vpcId properties: name: description: Name of the referenced object. type: string required: - name type: object vpcIdSelector: description: VPCIDSelector selects a reference to a VPC to and retrieves its vpcId properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object # ------------------------------------------------------------- dbParameterGroup: type: object properties: forProvider: required: - dbParameterGroupFamily - description - region type: object description: DBParameterGroupParameters defines the desired state of DBParameterGroup properties: dbParameterGroupFamily: description: "The DB parameter group family name. A DB parameter group can be associated with one and only one DB parameter group family, and can be applied only to a DB instance running a database engine and engine version compatible with that DB parameter group family. \n To list all of the available parameter group families for a DB engine, use the following command: \n aws rds describe-db-engine-versions --query \"DBEngineVersions[].DBParameterGroupFamily\" --engine \n For example, to list all of the available parameter group families for the MySQL DB engine, use the following command: \n aws rds describe-db-engine-versions --query \"DBEngineVersions[].DBParameterGroupFamily\" --engine mysql \n The output contains duplicates. \n The following are the valid DB engine values: \n * aurora (for MySQL 5.6-compatible Aurora) \n * aurora-mysql (for MySQL 5.7-compatible Aurora) \n * aurora-postgresql \n * mariadb \n * mysql \n * oracle-ee \n * oracle-ee-cdb \n * oracle-se2 \n * oracle-se2-cdb \n * postgres \n * sqlserver-ee \n * sqlserver-se \n * sqlserver-ex \n * sqlserver-web" type: string description: description: The description for the DB parameter group. type: string parameters: description: A list of parameters to associate with this DB parameter group items: properties: allowedValues: type: string applyMethod: type: string applyType: type: string dataType: type: string description: type: string isModifiable: type: boolean minimumEngineVersion: type: string parameterName: type: string parameterValue: type: string source: type: string supportedEngineModes: items: type: string type: array type: object type: array region: description: Region is which region the DBParameterGroup will be created. type: string tags: description: Tags to assign to the DB parameter group. items: properties: key: type: string value: type: string type: object type: array # ------------------------------------------------------------- rdsInstance: type: object properties: forProvider: required: - dbInstanceClass - engine type: object description: RDSInstanceParameters define the desired state of an AWS Relational Database Service instance. properties: allocatedStorage: description: 'AllocatedStorage is the amount of storage (in gibibytes) to allocate for the DB instance. Type: Integer Amazon Aurora Not applicable. Aurora cluster volumes automatically grow as the amount of data in your database increases, though you are only charged for the space that you use in an Aurora cluster volume. MySQL Constraints to the amount of storage for each storage type are the following: * General Purpose (SSD) storage (gp2): Must be an integer from 20 to 16384. * Provisioned IOPS storage (io1): Must be an integer from 100 to 16384. * Magnetic storage (standard): Must be an integer from 5 to 3072. MariaDB Constraints to the amount of storage for each storage type are the following: * General Purpose (SSD) storage (gp2): Must be an integer from 20 to 16384. * Provisioned IOPS storage (io1): Must be an integer from 100 to 16384. * Magnetic storage (standard): Must be an integer from 5 to 3072. PostgreSQL Constraints to the amount of storage for each storage type are the following: * General Purpose (SSD) storage (gp2): Must be an integer from 20 to 16384. * Provisioned IOPS storage (io1): Must be an integer from 100 to 16384. * Magnetic storage (standard): Must be an integer from 5 to 3072. Oracle Constraints to the amount of storage for each storage type are the following: * General Purpose (SSD) storage (gp2): Must be an integer from 20 to 16384. * Provisioned IOPS storage (io1): Must be an integer from 100 to 16384. * Magnetic storage (standard): Must be an integer from 10 to 3072. SQL Server Constraints to the amount of storage for each storage type are the following: * General Purpose (SSD) storage (gp2): Enterprise and Standard editions: Must be an integer from 200 to 16384. Web and Express editions: Must be an integer from 20 to 16384. * Provisioned IOPS storage (io1): Enterprise and Standard editions: Must be an integer from 200 to 16384. Web and Express editions: Must be an integer from 100 to 16384. * Magnetic storage (standard): Enterprise and Standard editions: Must be an integer from 200 to 1024. Web and Express editions: Must be an integer from 20 to 1024.' type: integer allowMajorVersionUpgrade: description: 'AllowMajorVersionUpgrade indicates that major version upgrades are allowed. Changing this parameter doesn''t result in an outage and the change is asynchronously applied as soon as possible. Constraints: This parameter must be set to true when specifying a value for the EngineVersion parameter that is a different major version than the DB instance''s current version.' type: boolean applyModificationsImmediately: description: 'ApplyModificationsImmediately specifies whether the modifications in this request and any pending modifications are asynchronously applied as soon as possible, regardless of the PreferredMaintenanceWindow setting for the DB instance. If this parameter is set to false, changes to the DB instance are applied during the next maintenance window. Some parameter changes can cause an outage and are applied on the next call to RebootDBInstance, or the next failure reboot. Review the table of parameters in Modifying a DB Instance and Using the Apply Immediately Parameter (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.Modifying.html) in the Amazon RDS User Guide. to see the impact that setting ApplyImmediately to true or false has for each modified parameter and to determine when the changes are applied. Default: false' type: boolean autoMinorVersionUpgrade: description: 'AutoMinorVersionUpgrade indicates that minor engine upgrades are applied automatically to the DB instance during the maintenance window. Default: true' type: boolean availabilityZone: description: 'AvailabilityZone is the EC2 Availability Zone that the DB instance is created in. For information on AWS Regions and Availability Zones, see Regions and Availability Zones (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html). Default: A random, system-chosen Availability Zone in the endpoint''s AWS Region. Example: us-east-1d Constraint: The AvailabilityZone parameter can''t be specified if the MultiAZ parameter is set to true. The specified Availability Zone must be in the same AWS Region as the current endpoint.' type: string backupRetentionPeriod: description: 'BackupRetentionPeriod is the number of days for which automated backups are retained. Setting this parameter to a positive number enables backups. Setting this parameter to 0 disables automated backups. Amazon Aurora Not applicable. The retention period for automated backups is managed by the DB cluster. For more information, see CreateDBCluster. Default: 1 Constraints: * Must be a value from 0 to 35 * Cannot be set to 0 if the DB instance is a source to Read Replicas' type: integer caCertificateIdentifier: description: CACertificateIdentifier indicates the certificate that needs to be associated with the instance. type: string characterSetName: description: CharacterSetName indicates that the DB instance should be associated with the specified CharacterSet for supported engines, Amazon Aurora Not applicable. The character set is managed by the DB cluster. For more information, see CreateDBCluster. type: string cloudwatchLogsExportConfiguration: description: CloudwatchLogsExportConfiguration is the configuration setting for the log types to be enabled for export to CloudWatch Logs for a specific DB instance. properties: disableLogTypes: description: DisableLogTypes is the list of log types to disable. items: type: string type: array enableLogTypes: description: EnableLogTypes is the list of log types to enable. items: type: string type: array type: object copyTagsToSnapshot: description: CopyTagsToSnapshot should be true to copy all tags from the DB instance to snapshots of the DB instance, and otherwise false. The default is false. type: boolean dbClusterIdentifier: description: 'DBClusterIdentifier is the identifier of the DB cluster that the instance will belong to. For information on creating a DB cluster, see CreateDBCluster. Type: String' type: string dbInstanceClass: description: DBInstanceClass is the compute and memory capacity of the DB instance, for example, db.m4.large. Not all DB instance classes are available in all AWS Regions, or for all database engines. For the full list of DB instance classes, and availability for your engine, see DB Instance Class (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html) in the Amazon RDS User Guide. type: string dbName: description: 'DBName is the meaning of this parameter differs according to the database engine you use. Type: String MySQL The name of the database to create when the DB instance is created. If this parameter is not specified, no database is created in the DB instance. Constraints: * Must contain 1 to 64 letters or numbers. * Cannot be a word reserved by the specified database engine MariaDB The name of the database to create when the DB instance is created. If this parameter is not specified, no database is created in the DB instance. Constraints: * Must contain 1 to 64 letters or numbers. * Cannot be a word reserved by the specified database engine PostgreSQL The name of the database to create when the DB instance is created. If this parameter is not specified, the default "postgres" database is created in the DB instance. Constraints: * Must contain 1 to 63 letters, numbers, or underscores. * Must begin with a letter or an underscore. Subsequent characters can be letters, underscores, or digits (0-9). * Cannot be a word reserved by the specified database engine Oracle The Oracle System ID (SID) of the created DB instance. If you specify null, the default value ORCL is used. You can''t specify the string NULL, or any other reserved word, for DBName. Default: ORCL Constraints: * Cannot be longer than 8 characters SQL Server Not applicable. Must be null. Amazon Aurora The name of the database to create when the primary instance of the DB cluster is created. If this parameter is not specified, no database is created in the DB instance. Constraints: * Must contain 1 to 64 letters or numbers. * Cannot be a word reserved by the specified database engine' type: string dbParameterGroupName: description: 'DBParameterGroupName is the name of the DB parameter group to associate with this DB instance. If this argument is omitted, the default DBParameterGroup for the specified engine is used. Constraints: * Must be 1 to 255 letters, numbers, or hyphens. * First character must be a letter * Cannot end with a hyphen or contain two consecutive hyphens' type: string dbSecurityGroups: description: 'DBSecurityGroups is a list of DB security groups to associate with this DB instance. Default: The default DB security group for the database engine.' items: type: string type: array dbSubnetGroupName: description: DBSubnetGroupName is a DB subnet group to associate with this DB instance. If there is no DB subnet group, then it is a non-VPC DB instance. type: string dbSubnetGroupNameRef: description: DBSubnetGroupNameRef is a reference to a DBSubnetGroup used to set DBSubnetGroupName. properties: name: description: Name of the referenced object. type: string required: - name type: object dbSubnetGroupNameSelector: description: DBSubnetGroupNameSelector selects a reference to a DBSubnetGroup used to set DBSubnetGroupName. properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object deletionProtection: description: DeletionProtection indicates if the DB instance should have deletion protection enabled. The database can't be deleted when this value is set to true. The default is false. For more information, see Deleting a DB Instance (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_DeleteInstance.html). type: boolean domain: description: Domain specifies the Active Directory Domain to create the instance in. type: string domainIAMRoleName: description: DomainIAMRoleName specifies the name of the IAM role to be used when making API calls to the Directory Service. type: string domainIAMRoleNameRef: description: DomainIAMRoleNameRef is a reference to an IAMRole used to set DomainIAMRoleName. properties: name: description: Name of the referenced object. type: string required: - name type: object domainIAMRoleNameSelector: description: DomainIAMRoleNameSelector selects a reference to an IAMRole used to set DomainIAMRoleName. properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object enableCloudwatchLogsExports: description: EnableCloudwatchLogsExports is the list of log types that need to be enabled for exporting to CloudWatch Logs. The values in the list depend on the DB engine being used. For more information, see Publishing Database Logs to Amazon CloudWatch Logs (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html#USER_LogAccess.Procedural.UploadtoCloudWatch) in the Amazon Relational Database Service User Guide. items: type: string type: array enableIAMDatabaseAuthentication: description: 'EnableIAMDatabaseAuthentication should be true to enable mapping of AWS Identity and Access Management (IAM) accounts to database accounts, and otherwise false. You can enable IAM database authentication for the following database engines: Amazon Aurora Not applicable. Mapping AWS IAM accounts to database accounts is managed by the DB cluster. For more information, see CreateDBCluster. MySQL * For MySQL 5.6, minor version 5.6.34 or higher * For MySQL 5.7, minor version 5.7.16 or higher Default: false' type: boolean enablePerformanceInsights: description: EnablePerformanceInsights should be true to enable Performance Insights for the DB instance, and otherwise false. For more information, see Using Amazon Performance Insights (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html) in the Amazon Relational Database Service User Guide. type: boolean engine: description: 'Engine is the name of the database engine to be used for this instance. Not every database engine is available for every AWS Region. Valid Values: * aurora (for MySQL 5.6-compatible Aurora) * aurora-mysql (for MySQL 5.7-compatible Aurora) * aurora-postgresql * mariadb * mysql * oracle-ee * oracle-se2 * oracle-se1 * oracle-se * postgres * sqlserver-ee * sqlserver-se * sqlserver-ex * sqlserver-web Engine is a required field' type: string engineVersion: description: EngineVersion is the version number of the database engine to use. For a list of valid engine versions, call DescribeDBEngineVersions. The following are the database engines and links to information about the major and minor versions that are available with Amazon RDS. Not every database engine is available for every AWS Region. Amazon Aurora Not applicable. The version number of the database engine to be used by the DB instance is managed by the DB cluster. For more information, see CreateDBCluster. MariaDB See MariaDB on Amazon RDS Versions (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html#MariaDB.Concepts.VersionMgmt) in the Amazon RDS User Guide. Microsoft SQL Server See Version and Feature Support on Amazon RDS (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.FeatureSupport) in the Amazon RDS User Guide. MySQL See MySQL on Amazon RDS Versions (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.VersionMgmt) in the Amazon RDS User Guide. Oracle See Oracle Database Engine Release Notes (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.PatchComposition.html) in the Amazon RDS User Guide. PostgreSQL See Supported PostgreSQL Database Versions (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions) in the Amazon RDS User Guide. type: string finalDBSnapshotIdentifier: description: 'The DBSnapshotIdentifier of the new DBSnapshot created when SkipFinalSnapshot is set to false. Specifying this parameter and also setting the SkipFinalShapshot parameter to true results in an error. Constraints: * Must be 1 to 255 letters or numbers. * First character must be a letter * Cannot end with a hyphen or contain two consecutive hyphens * Cannot be specified when deleting a Read Replica.' type: string iops: description: 'IOPS is the amount of Provisioned IOPS (input/output operations per second) to be initially allocated for the DB instance. For information about valid IOPS values, see see Amazon RDS Provisioned IOPS Storage to Improve Performance (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#USER_PIOPS) in the Amazon RDS User Guide. Constraints: Must be a multiple between 1 and 50 of the storage amount for the DB instance. Must also be an integer multiple of 1000. For example, if the size of your DB instance is 500 GiB, then your IOPS value can be 2000, 3000, 4000, or 5000.' type: integer kmsKeyId: description: KMSKeyID for an encrypted DB instance. The KMS key identifier is the Amazon Resource Name (ARN) for the KMS encryption key. If you are creating a DB instance with the same AWS account that owns the KMS encryption key used to encrypt the new DB instance, then you can use the KMS key alias instead of the ARN for the KM encryption key. Amazon Aurora Not applicable. The KMS key identifier is managed by the DB cluster. For more information, see CreateDBCluster. If the StorageEncrypted parameter is true, and you do not specify a value for the KMSKeyID parameter, then Amazon RDS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS Region. type: string licenseModel: description: 'LicenseModel information for this DB instance. Valid values: license-included | bring-your-own-license | general-public-license' type: string masterPasswordSecretRef: description: MasterPasswordSecretRef references the secret that contains the password used in the creation of this RDS instance. If no reference is given, a password will be auto-generated. properties: key: description: The key to select. type: string name: description: Name of the secret. type: string namespace: description: Namespace of the secret. type: string required: - key - name - namespace type: object masterUsername: description: 'MasterUsername is the name for the master user. Amazon Aurora Not applicable. The name for the master user is managed by the DB cluster. For more information, see CreateDBCluster. MariaDB Constraints: * Required for MariaDB. * Must be 1 to 16 letters or numbers. * Cannot be a reserved word for the chosen database engine. Microsoft SQL Server Constraints: * Required for SQL Server. * Must be 1 to 128 letters or numbers. * The first character must be a letter. * Cannot be a reserved word for the chosen database engine. MySQL Constraints: * Required for MySQL. * Must be 1 to 16 letters or numbers. * First character must be a letter. * Cannot be a reserved word for the chosen database engine. Oracle Constraints: * Required for Oracle. * Must be 1 to 30 letters or numbers. * First character must be a letter. * Cannot be a reserved word for the chosen database engine. PostgreSQL Constraints: * Required for PostgreSQL. * Must be 1 to 63 letters or numbers. * First character must be a letter. * Cannot be a reserved word for the chosen database engine.' type: string maxAllocatedStorage: description: "The upper limit to which Amazon RDS can automatically scale the storage of the DB instance. \n For more information about this setting, including limitations that apply to it, see Managing capacity automatically with Amazon RDS storage autoscaling (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.Autoscaling) in the Amazon RDS User Guide." type: integer monitoringInterval: description: 'MonitoringInterval is the interval, in seconds, between points when Enhanced Monitoring metrics are collected for the DB instance. To disable collecting Enhanced Monitoring metrics, specify 0. The default is 0. If MonitoringRoleARN is specified, then you must also set MonitoringInterval to a value other than 0. Valid Values: 0, 1, 5, 10, 15, 30, 60' type: integer monitoringRoleArn: description: MonitoringRoleARN is the ARN for the IAM role that permits RDS to send enhanced monitoring metrics to Amazon CloudWatch Logs. For example, arn:aws:iam:123456789012:role/emaccess. For information on creating a monitoring role, go to Setting Up and Enabling Enhanced Monitoring (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html#USER_Monitoring.OS.Enabling) in the Amazon RDS User Guide. If MonitoringInterval is set to a value other than 0, then you must supply a MonitoringRoleARN value. type: string monitoringRoleArnRef: description: MonitoringRoleARNRef is a reference to an IAMRole used to set MonitoringRoleARN. properties: name: description: Name of the referenced object. type: string required: - name type: object monitoringRoleArnSelector: description: MonitoringRoleARNSelector selects a reference to an IAMRole used to set MonitoringRoleARN. properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object multiAZ: description: MultiAZ specifies if the DB instance is a Multi-AZ deployment. You can't set the AvailabilityZone parameter if the MultiAZ parameter is set to true. type: boolean optionGroupName: description: OptionGroupName indicates that the DB instance should be associated with the specified option group. Permanent options, such as the TDE option for Oracle Advanced Security TDE, can't be removed from an option group, and that option group can't be removed from a DB instance once it is associated with a DB instance type: string performanceInsightsKMSKeyId: description: PerformanceInsightsKMSKeyID is the AWS KMS key identifier for encryption of Performance Insights data. The KMS key ID is the Amazon Resource Name (ARN), KMS key identifier, or the KMS key alias for the KMS encryption key. type: string performanceInsightsRetentionPeriod: description: PerformanceInsightsRetentionPeriod is the amount of time, in days, to retain Performance Insights data. Valid values are 7 or 731 (2 years). type: integer port: description: 'Port number on which the database accepts connections. MySQL Default: 3306 Valid Values: 1150-65535 Type: Integer MariaDB Default: 3306 Valid Values: 1150-65535 Type: Integer PostgreSQL Default: 5432 Valid Values: 1150-65535 Type: Integer Oracle Default: 1521 Valid Values: 1150-65535 SQL Server Default: 1433 Valid Values: 1150-65535 except for 1434, 3389, 47001, 49152, and 49152 through 49156. Amazon Aurora Default: 3306 Valid Values: 1150-65535 Type: Integer' type: integer preferredBackupWindow: description: 'PreferredBackupWindow is the daily time range during which automated backups are created if automated backups are enabled, using the BackupRetentionPeriod parameter. For more information, see The Backup Window (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#USER_WorkingWithAutomatedBackups.BackupWindow) in the Amazon RDS User Guide. Amazon Aurora Not applicable. The daily time range for creating automated backups is managed by the DB cluster. For more information, see CreateDBCluster. The default is a 30-minute window selected at random from an 8-hour block of time for each AWS Region. To see the time blocks available, see Adjusting the Preferred DB Instance Maintenance Window (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html#AdjustingTheMaintenanceWindow) in the Amazon RDS User Guide. Constraints: * Must be in the format hh24:mi-hh24:mi. * Must be in Universal Coordinated Time (UTC). * Must not conflict with the preferred maintenance window. * Must be at least 30 minutes.' type: string preferredMaintenanceWindow: description: 'PreferredMaintenanceWindow is the time range each week during which system maintenance can occur, in Universal Coordinated Time (UTC). For more information, see Amazon RDS Maintenance Window (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html#Concepts.DBMaintenance). Format: ddd:hh24:mi-ddd:hh24:mi The default is a 30-minute window selected at random from an 8-hour block of time for each AWS Region, occurring on a random day of the week. Valid Days: Mon, Tue, Wed, Thu, Fri, Sat, Sun. Constraints: Minimum 30-minute window.' type: string processorFeatures: description: ProcessorFeatures is the number of CPU cores and the number of threads per core for the DB instance class of the DB instance. items: description: ProcessorFeature is a processor feature entry. For more information, see Configuring the Processor of the DB Instance Class (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html#USER_ConfigureProcessor) in the Amazon RDS User Guide. Please also see https://docs.aws.amazon.com/goto/WebAPI/rds-2014-10-31/ProcessorFeature properties: name: description: Name of the processor feature. Valid names are coreCount and threadsPerCore. type: string value: description: Value of a processor feature name. type: string required: - name - value type: object type: array promotionTier: description: 'PromotionTier specifies the order in which an Aurora Replica is promoted to the primary instance after a failure of the existing primary instance. For more information, see Fault Tolerance for an Aurora DB Cluster (http://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Backups.html#Aurora.Managing.FaultTolerance) in the Amazon Aurora User Guide. Default: 1 Valid Values: 0 - 15' type: integer publiclyAccessible: description: 'PubliclyAccessible specifies the accessibility options for the DB instance. A value of true specifies an Internet-facing instance with a publicly resolvable DNS name, which resolves to a public IP address. A value of false specifies an internal instance with a DNS name that resolves to a private IP address. Default: The default behavior varies depending on whether DBSubnetGroupName is specified. If DBSubnetGroupName is not specified, and PubliclyAccessible is not specified, the following applies: * If the default VPC in the target region doesn’t have an Internet gateway attached to it, the DB instance is private. * If the default VPC in the target region has an Internet gateway attached to it, the DB instance is public. If DBSubnetGroupName is specified, and PubliclyAccessible is not specified, the following applies: * If the subnets are part of a VPC that doesn’t have an Internet gateway attached to it, the DB instance is private. * If the subnets are part of a VPC that has an Internet gateway attached to it, the DB instance is public.' type: boolean region: description: Region is the region you'd like your RDSInstance to be created in. type: string restoreFrom: description: RestoreFrom specifies the details of the backup to restore when creating a new RDS instance. (If the RDS instance already exists, this property will be ignored.) properties: s3: description: S3 specifies the details of the S3 backup to restore from. properties: bucketName: description: BucketName is the name of the S3 bucket containing the backup to restore. type: string bucketNameRef: description: BucketNameRef is a reference to a Bucket used to set BucketName. properties: name: description: Name of the referenced object. type: string required: - name type: object bucketNameSelector: description: BucketNameSelector selects a reference to a Bucket used to set BucketName. properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object ingestionRoleARN: description: IngestionRoleARN is the IAM role RDS can assume that will allow it to access the contents of the S3 bucket. type: string ingestionRoleARNRef: description: IngestionRoleARNRef is a reference to a IAM Role used to set IngestionRoleARN. properties: name: description: Name of the referenced object. type: string required: - name type: object ingestionRoleARNSelector: description: IngestionRoleARNSelector selects a reference to a IAM Role used to set IngestionRoleARN. properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object prefix: description: Prefix is the path prefix of the S3 bucket within which the backup to restore is located. type: string sourceEngine: description: SourceEngine is the engine used to create the backup. Must be "mysql". type: string sourceEngineVersion: description: 'SourceEngineVersion is the version of the engine used to create the backup. Example: "5.7.30"' type: string required: - sourceEngine - sourceEngineVersion type: object snapshot: description: Snapshot specifies the details of the database snapshot to restore from. properties: snapshotIdentifier: description: SnapshotIdentifier is the identifier of the database snapshot to restore. type: string required: - snapshotIdentifier type: object source: description: Source is the type of the backup to restore when creating a new RDS instance. Only S3 and Snapshot are supported at present. enum: - S3 - Snapshot type: string required: - source type: object scalingConfiguration: description: ScalingConfiguration is the scaling properties of the DB cluster. You can only modify scaling properties for DB clusters in serverless DB engine mode. properties: autoPause: description: AutoPause specifies whether to allow or disallow automatic pause for an Aurora DB cluster in serverless DB engine mode. A DB cluster can be paused only when it's idle (it has no connections). If a DB cluster is paused for more than seven days, the DB cluster might be backed up with a snapshot. In this case, the DB cluster is restored when there is a request to connect to it. type: boolean maxCapacity: description: MaxCapacity is the maximum capacity for an Aurora DB cluster in serverless DB engine mode. Valid capacity values are 2, 4, 8, 16, 32, 64, 128, and 256. The maximum capacity must be greater than or equal to the minimum capacity. type: integer minCapacity: description: MinCapacity is the minimum capacity for an Aurora DB cluster in serverless DB engine mode. Valid capacity values are 2, 4, 8, 16, 32, 64, 128, and 256. The minimum capacity must be less than or equal to the maximum capacity. type: integer secondsUntilAutoPause: description: SecondsUntilAutoPause is the time, in seconds, before an Aurora DB cluster in serverless mode is paused. type: integer type: object skipFinalSnapshotBeforeDeletion: description: 'Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted. Note that when a DB instance is in a failure state and has a status of ''failed'', ''incompatible-restore'', or ''incompatible-network'', it can only be deleted when the SkipFinalSnapshotBeforeDeletion parameter is set to "true". Specify true when deleting a Read Replica. The FinalDBSnapshotIdentifier parameter must be specified if SkipFinalSnapshotBeforeDeletion is false. Default: false' type: boolean storageEncrypted: description: 'StorageEncrypted specifies whether the DB instance is encrypted. Amazon Aurora Not applicable. The encryption for DB instances is managed by the DB cluster. For more information, see CreateDBCluster. Default: false' type: boolean storageType: description: 'StorageType specifies the storage type to be associated with the DB instance. Valid values: standard | gp2 | io1 If you specify io1, you must also include a value for the IOPS parameter. Default: io1 if the IOPS parameter is specified, otherwise standard' type: string tags: description: Tags. For more information, see Tagging Amazon RDS Resources (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Tagging.html) in the Amazon RDS User Guide. items: description: Tag is a metadata assigned to an Amazon RDS resource consisting of a key-value pair. Please also see https://docs.aws.amazon.com/goto/WebAPI/rds-2014-10-31/Tag properties: key: description: 'A key is the required name of the tag. The string value can be from 1 to 128 Unicode characters in length and can''t be prefixed with "aws:" or "rds:". The string can only contain only the set of Unicode letters, digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'', ''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").' type: string value: description: 'A value is the optional value of the tag. The string value can be from 1 to 256 Unicode characters in length and can''t be prefixed with "aws:" or "rds:". The string can only contain only the set of Unicode letters, digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'', ''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").' type: string type: object type: array timezone: description: Timezone of the DB instance. The time zone parameter is currently supported only by Microsoft SQL Server (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.TimeZone). type: string useDefaultProcessorFeatures: description: A value that specifies that the DB instance class of the DB instance uses its default processor features. type: boolean vpcSecurityGroupIDRefs: description: VPCSecurityGroupIDRefs are references to VPCSecurityGroups used to set the VPCSecurityGroupIDs. items: description: A Reference to a named object. properties: name: description: Name of the referenced object. type: string required: - name type: object type: array vpcSecurityGroupIDSelector: description: VPCSecurityGroupIDSelector selects references to VPCSecurityGroups used to set the VPCSecurityGroupIDs. properties: matchControllerRef: description: MatchControllerRef ensures an object with the same controller reference as the selecting object is selected. type: boolean matchLabels: additionalProperties: type: string description: MatchLabels ensures an object with matching labels is selected. type: object type: object vpcSecurityGroupIds: description: 'VPCSecurityGroupIDs is a list of EC2 VPC security groups to associate with this DB instance. Amazon Aurora Not applicable. The associated list of EC2 VPC security groups is managed by the DB cluster. For more information, see CreateDBCluster. Default: The default EC2 VPC security group for the DB subnet group''s VPC.' items: type: string type: array required: - dbSubnetGroup - securityGroup - dbParameterGroup - rdsInstance required: - parameters status: type: object properties: databaseInstanceName: description: Name of the bucket type: string components: type: object properties: # ------------------------------------------------------------- dbSubnetGroup: description: Conditions of the resource. items: description: A Condition that may apply to a resource. properties: lastTransitionTime: description: LastTransitionTime is the last time this condition transitioned from one status to another. format: date-time type: string message: description: A Message containing details about this condition's last transition from one status to another, if any. type: string reason: description: A Reason for this condition's last transition from one status to another. type: string status: description: Status of this condition; is it currently True, False, or Unknown? type: string type: description: Type of this condition. At most one of each condition type may apply to a resource at any point in time. type: string required: - lastTransitionTime - reason - status - type type: object type: array # ------------------------------------------------------------- securityGroup: description: Conditions of the resource. items: description: A Condition that may apply to a resource. properties: lastTransitionTime: description: LastTransitionTime is the last time this condition transitioned from one status to another. format: date-time type: string message: description: A Message containing details about this condition's last transition from one status to another, if any. type: string reason: description: A Reason for this condition's last transition from one status to another. type: string status: description: Status of this condition; is it currently True, False, or Unknown? type: string type: description: Type of this condition. At most one of each condition type may apply to a resource at any point in time. type: string required: - lastTransitionTime - reason - status - type type: object type: array # ------------------------------------------------------------- dbParameterGroups: description: Conditions of the resource. items: description: A Condition that may apply to a resource. properties: lastTransitionTime: description: LastTransitionTime is the last time this condition transitioned from one status to another. format: date-time type: string message: description: A Message containing details about this condition's last transition from one status to another, if any. type: string reason: description: A Reason for this condition's last transition from one status to another. type: string status: description: Status of this condition; is it currently True, False, or Unknown? type: string type: description: Type of this condition. At most one of each condition type may apply to a resource at any point in time. type: string required: - lastTransitionTime - reason - status - type type: object type: array # ------------------------------------------------------------- rdsInstance: description: Conditions of the resource. items: description: A Condition that may apply to a resource. properties: lastTransitionTime: description: LastTransitionTime is the last time this condition transitioned from one status to another. format: date-time type: string message: description: A Message containing details about this condition's last transition from one status to another, if any. type: string reason: description: A Reason for this condition's last transition from one status to another. type: string status: description: Status of this condition; is it currently True, False, or Unknown? type: string type: description: Type of this condition. At most one of each condition type may apply to a resource at any point in time. type: string required: - lastTransitionTime - reason - status - type type: object type: array ```

While checking our setup I found this after describing an instance, seems like the kind keeps being rewritten and that may be the trigger for all of our issues.

Events:
  Type    Reason                      Age                     From                                                             Message
  ----    ------                      ----                    ----                                                             -------
  Normal  BindCompositeResource       29m (x6808 over 20d)    offered/compositeresourcedefinition.apiextensions.crossplane.io  Successfully bound composite resource
  Normal  ConfigureCompositeResource  6m59s (x6834 over 20d)  offered/compositeresourcedefinition.apiextensions.crossplane.io  Successfully applied composite resource

And all of our objetcs are impacted, the number increase each time I run the command.

 $ k get mysql,pgsql -A -o=custom-columns='RESOURCE_VERSION:.metadata.resourceVersion'
RESOURCE_VERSION
92920052
92920175
92919353
92919299
[...]

Could you also take a look into what kind of resource is tagged with `AddTagsToResource`?

I gathered all events from the last 10 hours and the targets are only the RDS DB object, it doesnt impact any other object (parameter group, security group ...). However, I found out that not all the instances are on the list, and I found why, the instances with an empty rdsInstance.forProvider.dbName are the ones being Tagged over and over. That's something we already made mandatory on our latests templates, people may be using an outdated version and we are going to fix this ASAP.

k get mysql -A -o=custom-columns='UID:.metadata.uid,DBNAME:.spec.parameters.rdsInstance.forProvider.dbName'

```Shell $ k get mysql -A -o=custom-columns='UID:.metadata.uid,DBNAME:.spec.parameters.rdsInstance.forProvider.dbName' UID DBNAME bdf0324d-9900-4181-b5f4-cd689719b917 redacted 6151c630-d994-4b5d-8d5b-0e2eeacd5ca7 redacted e8329191-9c5e-4598-91f9-fdc04917691e redacted c8990313-b451-4316-829d-d7d308abf410 redacted 6d678206-4311-453f-884a-f1be7eb00948 redacted 71fd8bdc-4b21-40bf-9bc1-3b65a7b8a8b1 redacted dde38a53-cb1f-45a4-9322-cf78afb6f0b8 redacted b268c13f-a8b8-47cf-bd84-9a572db81cfc 9185f438-9328-4c37-a095-aba5ca8c7805 807f978a-efe5-43e9-859a-70151b1bd9c4 995a9674-8348-4b34-a1a4-6a07e006ff0f 63917556-a50c-43dc-93a5-b1e725beace7 951137cb-3c5c-4acc-87b6-6e4bd66aa222 6d925516-8510-43ac-bc06-08233cca36e8 5166ef3c-4c30-4240-8e7a-3cedef03caf9 26114338-a911-43d0-afdf-a204cc8dac65 a1fe93b4-967e-4c9a-801c-2b3e8ffbe6ed 738ae87e-f9d2-412f-a9eb-b25b8a6c8756 25847b29-914a-4221-a0de-e5d9050d1fac redacted fbf78898-572c-4b5e-9cc8-28d11833d54b redacted 2595e93e-dfea-4821-930d-7b865d6216ab b43c9e03-c4c1-4947-ac00-ea12b27fdd66 1761b0ba-b78f-4022-b78d-a615fd9d672e redacted 22997ec5-c95d-451c-ade0-2c6c353a2fe4 redacted ae7cc3bf-a06a-489d-b98d-c8181591e30a redacted 545e7dd7-9148-43de-8892-f24c78fbe31f redacted ```

I wonder if you could upgrade to 0.29 to check if the situation has improved.

We need to test the impacts of a live upgrade on another cluster, could be during this week.

UpperCase in maintenance/backup window

We do not specify the values, it uses the defaults

Thanks again for your help

cest-pas-faux commented 2 years ago

Hello @chlunde ,

The situation is worsening, we are being throttled by AWS due to the calls increasing, and we requested a quota extension for this, however it's difficult to explain to AWS why there is nearly 50 calls per second for only 30 databases deployed.

'AddTagsToResource': 435,
'DescribeDBClusterParameterGroups': 17042,
'DescribeDBClusterParameters': 7925,
'DescribeDBClusters': 18020,
'DescribeDBInstances': 49925,
'DescribeDBParameterGroups': 16505,
'DescribeDBParameters': 15496,
'DescribeDBSubnetGroups': 28931,
'ListTagsForResource': 19208,
'ModifyDBInstance': 558,
'ModifyDBParameterGroup': 67

'Processed Events': 174112,
'Timeframe': '- 60 minutes'

We reduced the number of databases without rdsInstance.forProvider.dbName, not sure why it's related, but AddTagsToResource has decreased. We also are planning to upgrade to 0.29.

EDIT : We suspect something, the patches in the compositions that check and posts the status from each component :

        - type: ToCompositeFieldPath
          fromFieldPath: status.conditions
          toFieldPath: status.components.dbSubnetGroup

We are going to test to disable it, and if it does not break anything, deploy it in production and re-assess the calls numbers.

Do we have another way to mitigate those issues ?

cest-pas-faux commented 2 years ago

Update : after some back-and-forth with AWS and a lot of digging from us, we found out that when the provider has an issue to sync an object, it keeps retrying endlessly.

As the calls are retrying, more and more calls are being throttled and it hits the rate limit and is stuck forever. AWS increased temporarily our rate limit and the provider managed to sync, then the calls dropped to a more normal rate.

From AWS :

It is important to highlight that the call rate dropped to 1/3 (from 50 req/s to 16 req/s) after the update, 
this is due to the fact that you are constantly retrying when you get throttled. 
Suggested to review the automation process to avoid those retry storms.
Those limits will be removed in two weeks from now, unless you have a valid business case to keep them.

Regarding our setup, we upgraded to 0.29.0 - no changes, we removed the components status - no changes. We are still wondering how to cap the calls number and -if possible- enable a ExponentialBackOff in order to avoid the situation that may happen again.

preetsindhal commented 1 year ago

anyone was able to fix it? we are having the same problem too many api calls resulting in throttling, specially for cloudfront crossplane version: 1.10 aws-provider: 0.32 eks: 1.22

Thanks

bobh66 commented 1 year ago

The changes in https://github.com/crossplane-contrib/provider-aws/pull/1705 may have solved much of this issue - they are in the latest release - 0.39.0

github-actions[bot] commented 1 year ago

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

crossplane-contrib / provider-aws