create error: `subnet` CRD is not working...

[zerunhu]

What happened?

When I create a subnet resource I get an error

kubectl get subnet
NAME                            READY   SYNCED   EXTERNAL-NAME   AGE
pracing-test-private-subnet-1           False                    13m

kubectl describe subnet
Events:
  Type     Reason                           Age                  From                                             Message
  ----     ------                           ----                 ----                                             -------
  Warning  CannotObserveExternalResource    9m28s                managed/ec2.aws.upbound.io/v1beta1, kind=subnet  cannot run refresh: refresh failed:
  Warning  CannotInitializeManagedResource  7m34s                managed/ec2.aws.upbound.io/v1beta1, kind=subnet  Operation cannot be fulfilled on subnets.ec2.aws.upbound.io "pracing-test-private-subnet-1": the object has been modified; please apply your changes to the latest version and try again
  Warning  CannotObserveExternalResource    41s (x4 over 7m35s)  managed/ec2.aws.upbound.io/v1beta1, kind=subnet  cannot run refresh: refresh failed: failed to read schema for aws_subnet.pracing-test-private-subnet-1 in registry.terraform.io/hashicorp/aws: failed to instantiate provider "registry.terraform.io/hashicorp/aws" to obtain schema: timeout while waiting for plugin to start:

How can we reproduce it?

I got a quick start with this document https://marketplace.upbound.io/providers/upbound/provider-aws/v0.21.0 Install the up command in order, then run up uxp install, then install provider and providerconfig, but I didn't create the s3 bucket, I created vpc and subnet directly, then I got the above error

What environment did it happen in?

• Universal Crossplane Version:

  up --version
  v0.15.0

• Provider Version:

  package: xpkg.upbound.io/upbound/provider-aws:v0.21.0

• Kubectl Version:

  kubectl version
  Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:35:40Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration

  aws EKS Fargate cluster

[zerunhu]

I tried a large number of resources with the same error, and I saw the error log in the aws-provider's pod.

W1125 06:11:43.796552       1 reflector.go:347] k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: watch of *v1beta1.CertificateValidation ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W1125 06:11:43.796642       1 reflector.go:347] k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: watch of *v1beta1.Dashboard ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W1125 06:11:43.794631       1 reflector.go:347] k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: watch of *v1beta1.DataCatalog ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W1125 06:11:43.796848       1 reflector.go:347] k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: watch of *v1beta1.Queue ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W1125 06:11:43.796552       1 reflector.go:347] k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: watch of *v1beta1.Repository ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W1125 06:11:43.497855       1 reflector.go:347] k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: watch of *v1beta1.BucketRequestPaymentConfiguration ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding

[turkenh]

Just double-checked on my side and things subnet & vpc working fine with provider-aws:v0.21.0.

@zerunhu I would suspect from the environment you're running the provider not having enough resources:

• either your Kube API server is not powerful enough to handle all CRDs in this provider.
• or your provider pod does not have enough resources.

Not sure how much control you've over these parameters in a Fargate environment, but please consider giving more power somehow.