rvowles commented 2 years ago

I am using 0.30.0 and I tried 0.29.0 as well. I will move back to 0.27 after lodging this ticket. The issue is that even creating a simple bucket, there is an endless loop of create & delete of the bucket:

2022-08-15T04:51:29.927  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.HeadBucket => 404 (404)
2022-08-15T04:51:29.949  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CreateBucket => 200
2022-08-15T04:51:29.960  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketVersioning => 200
2022-08-15T04:51:29.977  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketAccelerateConfiguration => 200
2022-08-15T04:51:30.010  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketCors => 404 (NoSuchCORSConfiguration)
2022-08-15T04:51:30.033  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketLifecycleConfiguration => 404 (NoSuchLifecycleConfiguration)
2022-08-15T04:51:30.046  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketLogging => 200
2022-08-15T04:51:30.053  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketNotificationConfiguration => 200
2022-08-15T04:51:30.074  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketReplication => 404 (ReplicationConfigurationNotFoundError)
2022-08-15T04:51:30.105  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketRequestPayment => 200
2022-08-15T04:51:30.124  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketEncryption => 404 (ServerSideEncryptionConfigurationNotFoundError)
2022-08-15T04:51:30.136  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketTagging => 404 (NoSuchTagSet)
2022-08-15T04:51:30.149  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketWebsite => 404 (NoSuchWebsiteConfiguration)
2022-08-15T04:51:30.163  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetPublicAccessBlock => 404 (NoSuchPublicAccessBlockConfiguration)
2022-08-15T04:51:30.205  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.HeadBucket => 200
2022-08-15T04:51:30.224  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketRequestPayment => 200
2022-08-15T04:51:30.238  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketVersioning => 200
2022-08-15T04:51:30.252  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketAccelerateConfiguration => 200
2022-08-15T04:51:30.274  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketCors => 404 (NoSuchCORSConfiguration)
2022-08-15T04:51:30.297  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketLifecycleConfiguration => 404 (NoSuchLifecycleConfiguration)
2022-08-15T04:51:30.309  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketLogging => 200
2022-08-15T04:51:30.316  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketNotificationConfiguration => 200
2022-08-15T04:51:30.336  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketReplication => 404 (ReplicationConfigurationNotFoundError)
2022-08-15T04:51:30.356  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketRequestPayment => 200
2022-08-15T04:51:30.366  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketEncryption => 404 (ServerSideEncryptionConfigurationNotFoundError)
2022-08-15T04:51:30.378  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetBucketTagging => 404 (NoSuchTagSet)
2022-08-15T04:51:30.391  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.GetBucketWebsite => 404 (NoSuchWebsiteConfiguration)
2022-08-15T04:51:30.407  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetPublicAccessBlock => 404 (NoSuchPublicAccessBlockConfiguration)
2022-08-15T04:51:30.420  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutBucketAcl => 200
2022-08-15T04:51:30.452  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.DeleteBucketOwnershipControls => 204
2022-08-15T04:51:30.478  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.HeadBucket => 404 (404)
2022-08-15T04:51:31.210  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.HeadBucket => 404 (404)
2022-08-15T04:51:33.231  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.HeadBucket => 404 (404)
2022-08-15T04:51:37.254  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.HeadBucket => 404 (404)

then it starts all over again. I tried this on two Mac machines, an Intel and an M1.

=== Install kind:

[source,bash]

kind create cluster --name cplane

=== Install localstack

save a new localstack-values.yml override file (localstack starts up too slowly and keeps crashing):

[source,yml]

livenessProbe: initialDelaySeconds: 0 periodSeconds: 100 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 3

[source,bash]

kubectl create namespace localstack helm repo add localstack-repo https://helm.localstack.cloud helm upgrade --install localstack localstack-repo/localstack --namespace localstack -f ./localstack-values.yml

=== Install crossplane

kubectl create namespace crossplane-system helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

Install the crossplane kubectl extensions (if haven't already):

$ curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh $ sudo mv kubectl-crossplane /usr/local/bin

Install the crossplane AWS provider:

$ kubectl crossplane install provider crossplane/provider-aws:v0.30.0

save this as `localstack-provider.yml` and then `kubectl apply -f` it [source,yaml]

AWS credentials secret

apiVersion: v1 kind: Secret metadata: name: localstack-creds namespace: crossplane-system type: Opaque data:

This is just test/test.

credentials: W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gdGVzdAphd3Nfc2VjcmV0X2FjY2Vzc19rZXkgPSB0ZXN0Cg==

AWS provider that references the secret credentials

apiVersion: aws.crossplane.io/v1beta1 kind: ProviderConfig metadata: name: localstack namespace: localstack spec: endpoint: hostnameImmutable: true url: type: Static static: http://localstack.localstack:4566 credentials: source: Secret secretRef: namespace: crossplane-system name: localstack-creds key: credentials

Now try and create a bucket:

kubectl apply -f this:

[source,yaml]

s3 bucket

apiVersion: s3.aws.crossplane.io/v1beta1 kind: Bucket metadata: name: flux-test-bucket annotations:

This will be the actual bucket name. It must be globally unique

crossplane.io/external-name: flux-test-bucket

spec: forProvider: locationConstraint: ap-southeast-2 acl: private publicAccessBlockConfiguration: blockPublicPolicy: true accelerateConfiguration: status: Enabled versioningConfiguration: status: Enabled tagging: tagSet:

key: app value: core
key: usage value: object-storage objectLockEnabledForBucket: false serverSideEncryptionConfiguration: rules:
applyServerSideEncryptionByDefault: sseAlgorithm: AES256 providerConfigRef: name: localstack

You can simplify it to just the locationConstraint and the acl, but it doesn't make any difference.

it should come up using kubectl get buckets.

NAME READY SYNCED AGE flux-test-bucket True True 37m

Do kubectl port-forward --namespace localstack svc/localstack 4566 on a terminal to create a link to localstack and then aws --endpoint=http://localhost:4566 s3 ls - it might appear and then it will disappear, and never come back.

The error occurring in the logs I did see:

1.6605294868661654e+09 DEBUG events Normal {"object": {"kind":"Bucket","name":"flux-test-bucket","uid":"312a0367-46d3-4fe5-8ca5-828cf322b454","apiVersion":"s3.aws.crossplane.io/v1beta1","resourceVersion":"17669"}, "reason": "UpdatedExternalResource", "message": "Successfully requested update of external resource"} 1.6605294868733933e+09 ERROR controller.managed/bucket.s3.aws.crossplane.io Reconciler error {"reconciler group": "s3.aws.crossplane.io", "reconciler kind": "Bucket", "name": "flux-test-bucket", "namespace": "", "error": "cannot update managed resource status: Operation cannot be fulfilled on buckets.s3.aws.crossplane.io \"flux-test-bucket\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 1.6605294868737316e+09 DEBUG provider-aws Reconciling {"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/flux-test-bucket"} 1.6605294870281725e+09 DEBUG provider-aws External resource is up to date {"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/flux-test-bucket", "uid": "312a0367-46d3-4fe5-8ca5-828cf322b454", "version": "17670", "external-name": "flux-test-bucket", "requeue-after": 1660529547.0281692} 1.66

rvowles commented 2 years ago

I forgot to add the last step:

$ kubectl port-forward --namespace=localstack svc/localstack 4566 $ aws --endpoint=http://localhost:4566/ s3 ls

The bucket is appearing to be synced and ready in kubectl get buckets but it is not in localstack as it keeps getting deleted immediately after creation. I have tried with localstack latest, 1..0.4, 1.0.3 and 0.14.5 - same commands are being issued with the same effect. I'm going to try swapping back to 0.27.1 of this provider next.

rvowles commented 1 year ago

We have given up on crossplane as it is too unreliable

crossplane-contrib / provider-aws

S3 with localstack - endlessly creates and deletes the bucket #1425

[source,bash]

kind create cluster --name cplane

[source,yml]

livenessProbe: initialDelaySeconds: 0 periodSeconds: 100 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 3

[source,bash]

kubectl create namespace localstack helm repo add localstack-repo https://helm.localstack.cloud helm upgrade --install localstack localstack-repo/localstack --namespace localstack -f ./localstack-values.yml

kubectl create namespace crossplane-system helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

save this as `localstack-provider.yml` and then `kubectl apply -f` it [source,yaml]

AWS credentials secret

This is just test/test.

credentials: W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gdGVzdAphd3Nfc2VjcmV0X2FjY2Vzc19rZXkgPSB0ZXN0Cg==

AWS provider that references the secret credentials

[source,yaml]

s3 bucket

This will be the actual bucket name. It must be globally unique

NAME READY SYNCED AGE flux-test-bucket True True 37m

crossplane-contrib / provider-aws

S3 with localstack - endlessly creates and deletes the bucket #1425

[source,bash]

kind create cluster --name cplane

[source,yml]

livenessProbe: initialDelaySeconds: 0 periodSeconds: 100 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 3

[source,bash]

kubectl create namespace localstack helm repo add localstack-repo https://helm.localstack.cloud helm upgrade --install localstack localstack-repo/localstack --namespace localstack -f ./localstack-values.yml

kubectl create namespace crossplane-system helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

save this as localstack-provider.yml and then kubectl apply -f it [source,yaml]

AWS credentials secret

This is just test/test.

credentials: W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gdGVzdAphd3Nfc2VjcmV0X2FjY2Vzc19rZXkgPSB0ZXN0Cg==

AWS provider that references the secret credentials

[source,yaml]

s3 bucket

This will be the actual bucket name. It must be globally unique

NAME READY SYNCED AGE flux-test-bucket True True 37m

save this as `localstack-provider.yml` and then `kubectl apply -f` it [source,yaml]