search

crossplane-contrib / provider-aws

Crossplane AWS Provider
Apache License 2.0
421 stars 363 forks source link

AWS-Provider update never works smoothly #1511

Closed flaviomoringa closed 11 months ago

flaviomoringa commented 1 year ago

What happened?

Every time we upgrade the aws-provider the provider revision gets stuck with the following warnings:

$> k describe providerrevisions crossplane-provider-aws
...
  Warning  SyncPackage        19m (x4 over 12d)    packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/configurations.kafka.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        16m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/vpcendpointserviceconfigurations.ec2.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        13m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusters.redshift.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        90s (x406 over 12d)  packages/providerrevision.pkg.crossplane.io  (combined from similar events): cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/bucketpolicies.s3.aws.crossplane.io?dryRun=All": context deadline exceeded

And it never gets healthy:

$> k get providers                                                                                                                                                                                                                                                                  
NAME                      INSTALLED   HEALTHY   PACKAGE                           AGE
crossplane-provider-aws   True        False     crossplane/provider-aws:v0.32.0   31d
crossplane-provider-sql   True        True      crossplane/provider-sql:v0.5.0    33d

Se we never get the new aws-provider pod available.

How can we reproduce it?

We are using argoCD to deploy the providers, we just upgraded the version from 0.29.0 to 0.32.0 on our repo:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: crossplane-provider-aws
spec:
  package: crossplane/provider-aws: 0.32.0
  controllerConfigRef:
    name:  aws-config

What environment did it happen in?

This is getting very frustrating since it has happened everytime we upgraded the aws-provider. We have tried on difference clusters, and on different environments (SAP Gardener also), and everytime we have issues. Never a smooth upgrade :-(

There are some tickets opened that refer the same, but none of the information there has helped us.

What can we do?

Thanks

haarchri commented 1 year ago

can you try with crossplane 1.9 ?

flaviomoringa commented 1 year ago

@haarchri I can try yes. Anything there you feel might help? We already upgraded from 1.7 to 1.8 hoping for improvements, but nothing changed. But we will try with 1.9.

haarchri commented 1 year ago

in v1.9.1 the following is fixed: https://github.com/crossplane/crossplane/pull/3309

flaviomoringa commented 1 year ago

@haarchri We are still with the issue of upgrading to 0.32.0 without the provider revision being healthy... Do you think we can upgrade to 1.9.1 like this, or the aws-provider revision should be healthy first?

flaviomoringa commented 1 year ago

I upgraded crossplane to 1.9.1 as @haarchri mentioned above, and after that I Upgraded aws-provider from v0.31.0 to v0.32.0. Maybe it takes more time, but It's been 20 minutes and it's still unealthy and I get this from the provider revisions:

k describe providerrevisions crossplane-provider-aws
...
  Warning  SyncPackage        16m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/identityproviderconfigs.eks.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        13m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/volumes.ec2.aws.crossplane.io": context deadline exceeded
  Normal   ApplyClusterRoles  12m (x5 over 45m)    rbac/providerrevision.pkg.crossplane.io      Applied RBAC ClusterRoles
  Normal   BindClusterRole    12m (x7 over 46m)    rbac/providerrevision.pkg.crossplane.io      Bound system ClusterRole to provider ServiceAccount(s)
  Warning  SyncPackage        12m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "transitgatewayvpcattachments.ec2.aws.crossplane.io": the object has been modified; please apply your changes to the latest version and try again
  Warning  SyncPackage        11m (x2 over 19m)    packages/providerrevision.pkg.crossplane.io  cannot update annotations for package revision: Operation cannot be fulfilled on providerrevisions.pkg.crossplane.io "crossplane-provider-aws-edfa6e79bd65": the object has been modified; please apply your changes to the latest version and try again
  Warning  SyncPackage        8m56s                packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/cachepolicies.cloudfront.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        5m56s (x2 over 25m)  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/hostedzones.route53.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        2m56s                packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/resourceshares.ram.aws.crossplane.io": context deadline exceeded

any more hints?

haarchri commented 1 year ago

@hasheddan any ideas ?

flaviomoringa commented 1 year ago

What I also see is that when I do "k describe providerrevisions crossplane-provider-aws", both old and new versions shows up (0.31.0 and 0.32.0)... I remember this happening before, and I tried deleting the old one, but didn't seemed to help.

flaviomoringa commented 1 year ago

New version:

$> k describe providerrevisions crossplane-provider-aws-245ce7fb587d
Name:         crossplane-provider-aws-245ce7fb587d
Namespace:
Labels:       pkg.crossplane.io/package=crossplane-provider-aws
Annotations:  meta.crossplane.io/description:
                The Amazon Web Services (AWS) Crossplane provider adds support for
                managing AWS resources in Kubernetes.
              meta.crossplane.io/license: Apache-2.0
              meta.crossplane.io/maintainer: Crossplane Maintainers <info@crossplane.io>
              meta.crossplane.io/readme:
                `provider-aws` is the Crossplane infrastructure provider for [Amazon Web
                Services (AWS)](https://aws.amazon.com/).

                If you encounter an issue please reach out on
                [slack.crossplane.io](https://slack.crossplane.io) and create an issue in
                the
                [crossplane-contrib/provider-aws](https://github.com/crossplane-contrib/provider-aws)
                repo.
              meta.crossplane.io/source: github.com/crossplane/provider-aws
API Version:  pkg.crossplane.io/v1
Kind:         ProviderRevision
Metadata:
  Creation Timestamp:  2022-10-04T14:07:01Z
  Finalizers:
    revision.pkg.crossplane.io
  Generation:  1
  Managed Fields:
    API Version:  pkg.crossplane.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.crossplane.io/description:
          f:meta.crossplane.io/license:
          f:meta.crossplane.io/maintainer:
          f:meta.crossplane.io/readme:
          f:meta.crossplane.io/source:
        f:finalizers:
          .:
          v:"revision.pkg.crossplane.io":
        f:labels:
          .:
          f:pkg.crossplane.io/package:
        f:ownerReferences:
          .:
          k:{"uid":"3e7bf1d1-e559-4d15-b79e-ec4635dd0abd"}:
      f:spec:
        .:
        f:controllerConfigRef:
          .:
          f:name:
        f:desiredState:
        f:ignoreCrossplaneConstraints:
        f:image:
        f:packagePullPolicy:
        f:revision:
        f:skipDependencyResolution:
    Manager:      crossplane
    Operation:    Update
    Time:         2022-10-04T14:07:30Z
    API Version:  pkg.crossplane.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:controllerRef:
          .:
          f:name:
    Manager:      crossplane
    Operation:    Update
    Subresource:  status
    Time:         2022-10-04T14:07:30Z
  Owner References:
    API Version:           pkg.crossplane.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Provider
    Name:                  crossplane-provider-aws
    UID:                   3e7bf1d1-e559-4d15-b79e-ec4635dd0abd
  Resource Version:        52126463
  UID:                     5515bbab-f470-4db0-adb6-fe0c43c0ddcd
Spec:
  Controller Config Ref:
    Name:                         aws-config
  Desired State:                  Active
  Ignore Crossplane Constraints:  false
  Image:                          crossplane/provider-aws:v0.32.0
  Package Pull Policy:            IfNotPresent
  Revision:                       2
  Skip Dependency Resolution:     false
Status:
  Conditions:
    Last Transition Time:  2022-10-04T14:07:30Z
    Reason:                UnhealthyPackageRevision
    Status:                False
    Type:                  Healthy
  Controller Ref:
    Name:
Events:
  Type     Reason             Age                  From                                         Message
  ----     ------             ----                 ----                                         -------
  Normal   BindClusterRole    47m (x4 over 47m)    rbac/providerrevision.pkg.crossplane.io      Bound system ClusterRole to provider ServiceAccount(s)
  Normal   ApplyClusterRoles  47m (x4 over 47m)    rbac/providerrevision.pkg.crossplane.io      Applied RBAC ClusterRoles
  Warning  SyncPackage        47m                  packages/providerrevision.pkg.crossplane.io  cannot update annotations for package revision: Operation cannot be fulfilled on providerrevisions.pkg.crossplane.io "crossplane-provider-aws-245ce7fb587d": the object has been modified; please apply your changes to the latest version and try again
  Warning  SyncPackage        45m (x6 over 47m)    packages/providerrevision.pkg.crossplane.io  cannot establish control of object: certificates.acm.aws.crossplane.io is already controlled by ProviderRevision crossplane-provider-aws-edfa6e79bd65 (UID d437e1e1-e4f3-4d77-83eb-7fb1ff913cb2)
  Warning  SyncPackage        39m (x5 over 44m)    packages/providerrevision.pkg.crossplane.io  cannot establish control of object: certificateauthorities.acmpca.aws.crossplane.io is already controlled by ProviderRevision crossplane-provider-aws-edfa6e79bd65 (UID d437e1e1-e4f3-4d77-83eb-7fb1ff913cb2)
  Warning  SyncPackage        35m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/cachepolicies.cloudfront.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        31m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/dbinstances.rds.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        27m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/basepathmappings.apigateway.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        23m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/dbclusterparametergroups.docdb.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        19m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/userpooldomains.cognitoidentityprovider.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        15m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/vpcendpoints.ec2.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        3m22s (x3 over 11m)  packages/providerrevision.pkg.crossplane.io  (combined from similar events): cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/queues.sqs.aws.crossplane.io": context deadline exceeded

Old version still there:

$> k describe providerrevisions crossplane-provider-aws-edfa6e79bd65
Name:         crossplane-provider-aws-edfa6e79bd65
Namespace:
Labels:       pkg.crossplane.io/package=crossplane-provider-aws
Annotations:  company: Crossplane
              description:
                The Amazon Web Services (AWS) Crossplane provider adds support for
                managing AWS resources in Kubernetes.
              descriptionShort:
                The AWS Crossplane provider enables infrastructure management for Amazon
                Web Services.
              friendly-group-name.meta.crossplane.io/acm.aws.crossplane.io: Certificate Manager
              friendly-group-name.meta.crossplane.io/acmpca.aws.crossplane.io: Private CA
              friendly-group-name.meta.crossplane.io/apigatewayv2.aws.crossplane.io: API Gateway
              friendly-group-name.meta.crossplane.io/cache.aws.crossplane.io: ElastiCache
              friendly-group-name.meta.crossplane.io/database.aws.crossplane.io: Databases
              friendly-group-name.meta.crossplane.io/dynamodb.aws.crossplane.io: DynamoDB
              friendly-group-name.meta.crossplane.io/ec2.aws.crossplane.io: Elastic Compute
              friendly-group-name.meta.crossplane.io/ecr.aws.crossplane.io: Elastic Container Registry
              friendly-group-name.meta.crossplane.io/efs.aws.crossplane.io: Elastic Filesystem
              friendly-group-name.meta.crossplane.io/eks.aws.crossplane.io: Elastic Kubernetes
              friendly-group-name.meta.crossplane.io/elasticloadbalancing.aws.crossplane.io: Elastic Load Balancing
              friendly-group-name.meta.crossplane.io/iam.aws.crossplane.io: IAM
              friendly-group-name.meta.crossplane.io/kms.aws.crossplane.io: Key Managment Service
              friendly-group-name.meta.crossplane.io/notification.aws.crossplane.io: SNS
              friendly-group-name.meta.crossplane.io/rds.aws.crossplane.io: RDS
              friendly-group-name.meta.crossplane.io/redshift.aws.crossplane.io: Redshift
              friendly-group-name.meta.crossplane.io/route53.aws.crossplane.io: Route 53
              friendly-group-name.meta.crossplane.io/s3.aws.crossplane.io: S3
              friendly-group-name.meta.crossplane.io/secretsmanager.aws.crossplane.io: Secrets Manager
              friendly-group-name.meta.crossplane.io/sfn.aws.crossplane.io: Step Functions
              friendly-group-name.meta.crossplane.io/sqs.aws.crossplane.io: SQS
              friendly-kind-name.meta.crossplane.io/activity.sfn.aws.crossplane.io: Activity
              friendly-kind-name.meta.crossplane.io/address.ec2.aws.crossplane.io: Address
              friendly-kind-name.meta.crossplane.io/api.apigatewayv2.aws.crossplane.io: API
              friendly-kind-name.meta.crossplane.io/apimapping.apigatewayv2.aws.crossplane.io: API Mapping
              friendly-kind-name.meta.crossplane.io/authorizer.apigatewayv2.aws.crossplane.io: Authorizer
              friendly-kind-name.meta.crossplane.io/backup.dynamodb.aws.crossplane.io: Backup
              friendly-kind-name.meta.crossplane.io/bucket.s3.aws.crossplane.io: Bucket
              friendly-kind-name.meta.crossplane.io/bucketpolicy.s3.aws.crossplane.io: Bucket Policy
              friendly-kind-name.meta.crossplane.io/cachecluster.cache.aws.crossplane.io: Cache Cluster
              friendly-kind-name.meta.crossplane.io/cachesubnetgroup.cache.aws.crossplane.io: Cache Subnet Group
              friendly-kind-name.meta.crossplane.io/certificate.acm.aws.crossplane.io: Certificate
              friendly-kind-name.meta.crossplane.io/certificateauthority.acmpca.aws.crossplane.io: CA
              friendly-kind-name.meta.crossplane.io/certificateauthoritypermission.acmpca.aws.crossplane.io: CA Permission
              friendly-kind-name.meta.crossplane.io/cluster.eks.aws.crossplane.io: EKS Cluster
              friendly-kind-name.meta.crossplane.io/cluster.redshift.aws.crossplane.io: Redshift Cluster
              friendly-kind-name.meta.crossplane.io/dbcluster.rds.aws.crossplane.io: Database Cluster
              friendly-kind-name.meta.crossplane.io/dbparametergroup.rds.aws.crossplane.io: Database Parameter Group
              friendly-kind-name.meta.crossplane.io/dbsubnetgroup.database.aws.crossplane.io: Database Subnet Group
              friendly-kind-name.meta.crossplane.io/deployment.apigatewayv2.aws.crossplane.io: Deployment
              friendly-kind-name.meta.crossplane.io/domainname.apigatewayv2.aws.crossplane.io: Domain Name
              friendly-kind-name.meta.crossplane.io/elb.elasticloadbalancing.aws.crossplane.io: Elastic Load Balancer
              friendly-kind-name.meta.crossplane.io/elbattachment.elasticloadbalancing.aws.crossplane.io: ELB Attachment
              friendly-kind-name.meta.crossplane.io/fargateprofile.eks.aws.crossplane.io: Fargate Profile
              friendly-kind-name.meta.crossplane.io/filesystem.efs.aws.crossplane.io: Filesystem
              friendly-kind-name.meta.crossplane.io/globaltable.dynamodb.aws.crossplane.io: Global Table
              friendly-kind-name.meta.crossplane.io/hostedzone.route53.aws.crossplane.io: Hosted Zone
              friendly-kind-name.meta.crossplane.io/iamaccesskey.iam.aws.crossplane.io: IAM Access Key
              friendly-kind-name.meta.crossplane.io/iamgroup.iam.aws.crossplane.io: IAM Group
              friendly-kind-name.meta.crossplane.io/iamgrouppolicyattachment.iam.aws.crossplane.io: IAM Group Policy Attachment
              friendly-kind-name.meta.crossplane.io/iamgroupusermembership.iam.aws.crossplane.io: IAM Group User Membership
              friendly-kind-name.meta.crossplane.io/iampolicy.iam.aws.crossplane.io: IAM Policy
              friendly-kind-name.meta.crossplane.io/iamrole.iam.aws.crossplane.io: IAM Role
              friendly-kind-name.meta.crossplane.io/iamrolepolicyattachment.iam.aws.crossplane.io: IAM Role Policy Attachment
              friendly-kind-name.meta.crossplane.io/iamuser.iam.aws.crossplane.io: IAM User
              friendly-kind-name.meta.crossplane.io/iamuserpolicyattachment.iam.aws.crossplane.io: IAM User Policy Attachment
              friendly-kind-name.meta.crossplane.io/integration.apigatewayv2.aws.crossplane.io: Integration
              friendly-kind-name.meta.crossplane.io/integrationresponse.apigatewayv2.aws.crossplane.io: Integration Response
              friendly-kind-name.meta.crossplane.io/internetgateway.ec2.aws.crossplane.io: Internet Gateway
              friendly-kind-name.meta.crossplane.io/key.kms.aws.crossplane.io: Key
              friendly-kind-name.meta.crossplane.io/model.apigatewayv2.aws.crossplane.io: Model
              friendly-kind-name.meta.crossplane.io/natgateway.ec2.aws.crossplane.io: NAT Gateway
              friendly-kind-name.meta.crossplane.io/nodegroup.eks.aws.crossplane.io: EKS Node Group
              friendly-kind-name.meta.crossplane.io/queue.sqs.aws.crossplane.io: SQS Queue
              friendly-kind-name.meta.crossplane.io/rdsinstance.database.aws.crossplane.io: RDS Instance
              friendly-kind-name.meta.crossplane.io/replicationgroup.cache.aws.crossplane.io: Replication Group
              friendly-kind-name.meta.crossplane.io/repository.ecr.aws.crossplane.io: Repository
              friendly-kind-name.meta.crossplane.io/repositorypolicy.ecr.aws.crossplane.io: Repository Policy
              friendly-kind-name.meta.crossplane.io/resourcerecordset.route53.aws.crossplane.io: Resource Record Set
              friendly-kind-name.meta.crossplane.io/route.apigatewayv2.aws.crossplane.io: Route
              friendly-kind-name.meta.crossplane.io/routeresponse.apigatewayv2.aws.crossplane.io: Route Response
              friendly-kind-name.meta.crossplane.io/routetable.ec2.aws.crossplane.io: Route Table
              friendly-kind-name.meta.crossplane.io/secret.secretsmanager.aws.crossplane.io: Secret
              friendly-kind-name.meta.crossplane.io/securitygroup.ec2.aws.crossplane.io: Security Group
              friendly-kind-name.meta.crossplane.io/snssubscription.notification.aws.crossplane.io: Subscription
              friendly-kind-name.meta.crossplane.io/snstopic.notification.aws.crossplane.io: Topic
              friendly-kind-name.meta.crossplane.io/stage.apigatewayv2.aws.crossplane.io: Stage
              friendly-kind-name.meta.crossplane.io/statemachine.sfn.aws.crossplane.io: State Machine
              friendly-kind-name.meta.crossplane.io/subnet.ec2.aws.crossplane.io: Subnet
              friendly-kind-name.meta.crossplane.io/table.dynamodb.aws.crossplane.io: Table
              friendly-kind-name.meta.crossplane.io/vpc.ec2.aws.crossplane.io: VPC
              friendly-kind-name.meta.crossplane.io/vpccidrblock.ec2.aws.crossplane.io: VPC CIDR Block
              friendly-kind-name.meta.crossplane.io/vpclink.apigatewayv2.aws.crossplane.io: VPC Link
              friendly-name.meta.crossplane.io: Provider AWS
              iconData:
                CjxzdmcgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB3aWR0aD0iNjUiIGhlaWdodD0iNjUiPjxnIGZpbGw9Im5vbmUiIGZpbGwtcnVsZT0iZXZlbm9kZCI+PHJlY3...
              license: Apache-2.0
              maintainer: Crossplane Maintainers <info@crossplane.io>
              meta.crossplane.io/description:
                The Amazon Web Services (AWS) Crossplane provider adds support for
                managing AWS resources in Kubernetes.
              meta.crossplane.io/iconURI:
                data:image/svg+xml;base64,CjxzdmcgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB3aWR0aD0iNjUiIGhlaWdodD0iNjUiPjxnIGZpbGw9Im5vbmUiIGZpbGwt...
              meta.crossplane.io/license: Apache-2.0
              meta.crossplane.io/maintainer: Crossplane Maintainers <info@crossplane.io>
              meta.crossplane.io/readme:
                `provider-aws` is the Crossplane infrastructure provider for [Amazon Web
                Services (AWS)](https://aws.amazon.com/).

                Available resources and their fields can be found in the [CRD
                Docs](https://doc.crds.dev/github.com/crossplane/provider-aws).

                If you encounter an issue please reach out on
                [slack.crossplane.io](https://slack.crossplane.io) and create an issue in
                the [crossplane/provider-aws](https://github.com/crossplane/provider-aws)
                repo.
              meta.crossplane.io/source: github.com/crossplane/provider-aws
              readme:
                `provider-aws` is the Crossplane infrastructure provider for [Amazon Web
                Services (AWS)](https://aws.amazon.com/).

                Available resources and their fields can be found in the [CRD
                Docs](https://doc.crds.dev/github.com/crossplane/provider-aws).

                If you encounter an issue please reach out on
                [slack.crossplane.io](https://slack.crossplane.io) and create an issue in
                the [crossplane/provider-aws](https://github.com/crossplane/provider-aws)
                repo.
              source: github.com/crossplane/provider-aws
API Version:  pkg.crossplane.io/v1
Kind:         ProviderRevision
Metadata:
  Creation Timestamp:  2022-08-31T19:26:12Z
  Finalizers:
    revision.pkg.crossplane.io
  Generation:  2
  Managed Fields:
    API Version:  pkg.crossplane.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:company:
          f:description:
          f:descriptionShort:
          f:friendly-group-name.meta.crossplane.io/acm.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/acmpca.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/apigatewayv2.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/cache.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/database.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/dynamodb.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/ec2.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/ecr.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/efs.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/eks.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/elasticloadbalancing.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/iam.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/kms.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/notification.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/rds.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/redshift.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/route53.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/s3.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/secretsmanager.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/sfn.aws.crossplane.io:
          f:friendly-group-name.meta.crossplane.io/sqs.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/activity.sfn.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/address.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/api.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/apimapping.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/authorizer.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/backup.dynamodb.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/bucket.s3.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/bucketpolicy.s3.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/cachecluster.cache.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/cachesubnetgroup.cache.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/certificate.acm.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/certificateauthority.acmpca.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/certificateauthoritypermission.acmpca.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/cluster.eks.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/cluster.redshift.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/dbcluster.rds.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/dbparametergroup.rds.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/dbsubnetgroup.database.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/deployment.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/domainname.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/elb.elasticloadbalancing.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/elbattachment.elasticloadbalancing.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/fargateprofile.eks.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/filesystem.efs.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/globaltable.dynamodb.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/hostedzone.route53.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamaccesskey.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamgroup.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamgrouppolicyattachment.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamgroupusermembership.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iampolicy.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamrole.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamrolepolicyattachment.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamuser.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/iamuserpolicyattachment.iam.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/integration.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/integrationresponse.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/internetgateway.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/key.kms.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/model.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/natgateway.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/nodegroup.eks.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/queue.sqs.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/rdsinstance.database.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/replicationgroup.cache.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/repository.ecr.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/repositorypolicy.ecr.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/resourcerecordset.route53.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/route.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/routeresponse.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/routetable.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/secret.secretsmanager.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/securitygroup.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/snssubscription.notification.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/snstopic.notification.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/stage.apigatewayv2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/statemachine.sfn.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/subnet.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/table.dynamodb.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/vpc.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/vpccidrblock.ec2.aws.crossplane.io:
          f:friendly-kind-name.meta.crossplane.io/vpclink.apigatewayv2.aws.crossplane.io:
          f:friendly-name.meta.crossplane.io:
          f:iconData:
          f:license:
          f:maintainer:
          f:meta.crossplane.io/description:
          f:meta.crossplane.io/iconURI:
          f:meta.crossplane.io/license:
          f:meta.crossplane.io/maintainer:
          f:meta.crossplane.io/readme:
          f:meta.crossplane.io/source:
          f:readme:
          f:source:
        f:finalizers:
          .:
          v:"revision.pkg.crossplane.io":
        f:labels:
          .:
          f:pkg.crossplane.io/package:
        f:ownerReferences:
          .:
          k:{"uid":"3e7bf1d1-e559-4d15-b79e-ec4635dd0abd"}:
      f:spec:
        .:
        f:controllerConfigRef:
          .:
          f:name:
        f:desiredState:
        f:ignoreCrossplaneConstraints:
        f:image:
        f:packagePullPolicy:
        f:revision:
        f:skipDependencyResolution:
    Manager:      crossplane
    Operation:    Update
    Time:         2022-08-31T19:26:27Z
    API Version:  pkg.crossplane.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:controllerRef:
          .:
          f:name:
        f:objectRefs:
    Manager:      crossplane
    Operation:    Update
    Subresource:  status
    Time:         2022-09-02T10:08:17Z
  Owner References:
    API Version:           pkg.crossplane.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Provider
    Name:                  crossplane-provider-aws
    UID:                   3e7bf1d1-e559-4d15-b79e-ec4635dd0abd
  Resource Version:        52149855
  UID:                     d437e1e1-e4f3-4d77-83eb-7fb1ff913cb2
Spec:
  Controller Config Ref:
    Name:                         aws-config
  Desired State:                  Inactive
  Ignore Crossplane Constraints:  false
  Image:                          crossplane/provider-aws:v0.31.0
  Package Pull Policy:            IfNotPresent
  Revision:                       1
  Skip Dependency Resolution:     false
Status:
  Conditions:
    Last Transition Time:  2022-10-04T14:50:52Z
    Reason:                HealthyPackageRevision
    Status:                True
    Type:                  Healthy
  Controller Ref:
    Name:  crossplane-provider-aws-edfa6e79bd65
  Object Refs:
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         transitgatewayvpcattachments.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         userpooldomains.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         routes.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         targets.elbv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         authorizers.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         instanceprofiles.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpcendpoints.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         storeconfigs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbclusters.docdb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         tables.dynamodb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         providers.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         requestvalidators.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         clusters.kafka.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         stages.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         jobs.glue.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resourcerecordsets.route53.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         integrations.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         identityproviderconfigs.eks.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         responseheaderspolicies.cloudfront.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         certificates.acm.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         cachepolicies.cloudfront.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpccidrblocks.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbsubnetgroups.database.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         clusters.redshift.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         clusters.dax.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         groupusermemberships.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         userpolicyattachments.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resourceservers.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         policies.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resolverrules.route53resolver.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         distributions.cloudfront.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         accesspoints.efs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         addons.eks.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         cacheclusters.cache.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         apikeys.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         securitygrouprules.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         loadbalancers.elbv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         models.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         securitygroups.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         documentationparts.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         subnetgroups.dax.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         lifecyclepolicies.ecr.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         classifiers.glue.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         statemachines.sfn.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         aliases.kms.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbparametergroups.rds.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         hostedzones.route53.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         functions.lambda.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         integrationresponses.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         taskdefinitions.ecs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         queues.sqs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         keys.kms.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbclusterparametergroups.docdb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         listeners.elbv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         launchtemplates.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         usageplans.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         filesystems.efs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         identitypools.cognitoidentity.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resolverendpoints.route53resolver.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         connections.glue.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         instances.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         brokers.mq.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         globalclusters.rds.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         snstopics.notification.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         grouppolicyattachments.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         openidconnectproviders.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         cloudfrontoriginaccessidentities.cloudfront.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         nodegroups.eks.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         clusters.eks.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpcendpointserviceconfigurations.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         providerconfigusages.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         crawlers.glue.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbclusters.neptune.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         stages.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         snssubscriptions.notification.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         parametergroups.dax.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         activities.sfn.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         repositorypolicies.ecr.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpcpeeringconnections.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         userpoolclients.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         configurations.kafka.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         basepathmappings.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         integrations.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         userpools.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         backups.dynamodb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         privatednsnamespaces.servicediscovery.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         transitgatewayroutetables.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         streams.kinesis.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         domains.cloudsearch.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         workgroups.athena.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         documentationversions.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         domainnames.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resolverruleassociations.route53resolver.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         environments.mwaa.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         methods.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         users.transfer.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         cacheparametergroups.elasticache.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         identityproviders.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         routes.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         topics.sns.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         services.ecs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         servers.transfer.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         subscriptions.sns.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         targetgroups.elbv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         domainnames.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         users.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbinstances.rds.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         authorizers.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         securityconfigurations.glue.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         clusters.ecs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpclinks.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         rdsinstances.database.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         groupusermemberships.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         transitgateways.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         apis.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         policies.iot.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         workspaces.prometheusservice.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         elbattachments.elasticloadbalancing.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         routetables.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         launchtemplateversions.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         apimappings.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         things.iot.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         buckets.s3.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         models.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         groups.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         groups.cognitoidentityprovider.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         routeresponses.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         flowlogs.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         publicdnsnamespaces.servicediscovery.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         permissions.lambda.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         deployments.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbclusterparametergroups.rds.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         transitgatewayroutes.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         accesskeys.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbclusters.rds.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         httpnamespaces.servicediscovery.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpcs.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         replicationgroups.cache.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resourceshares.ram.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         certificateauthoritypermissions.acmpca.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbinstances.docdb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         users.mq.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         addresses.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         vpclinks.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         loggroups.cloudwatchlogs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbsubnetgroups.docdb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         subnets.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         natgateways.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         databases.glue.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         rulegroupsnamespaces.prometheusservice.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         mounttargets.efs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         roles.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         elbs.elasticloadbalancing.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         secrets.secretsmanager.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         internetgateways.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         rolepolicyattachments.iam.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         certificateauthorities.acmpca.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         alertmanagerdefinitions.prometheusservice.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         bucketpolicies.s3.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         restapis.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         usageplankeys.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         integrationresponses.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         globaltables.dynamodb.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         providerconfigs.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         dbinstanceroleassociations.rds.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         resources.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         cachesubnetgroups.cache.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         methodresponses.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         gatewayresponses.apigateway.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         fargateprofiles.eks.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         deployments.apigatewayv2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         volumes.ec2.aws.crossplane.io
    API Version:  apiextensions.k8s.io/v1
    Kind:         CustomResourceDefinition
    Name:         repositories.ecr.aws.crossplane.io
Events:
  Type     Reason             Age                  From                                         Message
  ----     ------             ----                 ----                                         -------
  Warning  SyncPackage        60m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/instances.ec2.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        57m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/publicdnsnamespaces.servicediscovery.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        43m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/responseheaderspolicies.cloudfront.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        37m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/identityproviderconfigs.eks.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        34m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/volumes.ec2.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        33m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "transitgatewayvpcattachments.ec2.aws.crossplane.io": the object has been modified; please apply your changes to the latest version and try again
  Warning  SyncPackage        32m (x2 over 40m)    packages/providerrevision.pkg.crossplane.io  cannot update annotations for package revision: Operation cannot be fulfilled on providerrevisions.pkg.crossplane.io "crossplane-provider-aws-edfa6e79bd65": the object has been modified; please apply your changes to the latest version and try again
  Warning  SyncPackage        29m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/cachepolicies.cloudfront.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        26m (x2 over 46m)    packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/hostedzones.route53.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        23m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/resourceshares.ram.aws.crossplane.io": context deadline exceeded
  Warning  SyncPackage        20m                  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/providerconfigusages.aws.crossplane.io": context deadline exceeded
  Normal   SyncPackage        6m9s (x3 over 54m)   packages/providerrevision.pkg.crossplane.io  Successfully configured package revision
  Normal   ApplyClusterRoles  6m7s (x6 over 66m)   rbac/providerrevision.pkg.crossplane.io      Applied RBAC ClusterRoles
  Warning  SyncPackage        2m53s (x6 over 17m)  packages/providerrevision.pkg.crossplane.io  (combined from similar events): cannot establish control of object: Put "https://172.16.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/providers.aws.crossplane.io": context deadline exceeded
  Normal   BindClusterRole    108s (x9 over 67m)   rbac/providerrevision.pkg.crossplane.io      Bound system ClusterRole to provider ServiceAccount(s)

I see a lot of "Object Refs:" on this old version... Might this be related?

flaviomoringa commented 1 year ago

But.. just checked and in one of our clusters where we upgraded from 0.29.0 to 0.32.0 It seems we also have both versions still there, but we have this:

NAME                      INSTALLED   HEALTHY   PACKAGE                           AGE
crossplane-provider-aws   True        True      crossplane/provider-aws:v0.32.0   32d

And now I'm not sure if even on clusters where 0.32.0 seems installed and healthy, if everything is ok :-(

haarchri commented 1 year ago

Check kubectl get pods -n crossplane-system and describe provider-aws check for Image

flaviomoringa commented 1 year ago

seems ok:

$> k describe pod crossplane-provider-aws-245ce7fb587d-bf996f7bb-pcvhh -n caas-base | grep Image
    Image:          crossplane/provider-aws:v0.32.0
    Image ID:       docker.io/crossplane/provider-aws@sha256:245ce7fb587dd634dbf92b584aa24b1ef58bc1a1fdee023132028e9535efad0b

Regarding health, it seems after 4 or 5 hours the new versions finally becomes healthy.

But the older version stays there, and both continue spitting out "context deadline exceed errors"... It's not reassuring at all...

MisterMX commented 11 months ago

Closing as this is not a problem with the provider itself but with Crossplane which takes too long to install all CRDs and runs in the context timeout. This should have been resolved with https://github.com/crossplane/crossplane/pull/3176.