search

crossplane-contrib / provider-upjet-aws

Official AWS Provider for Crossplane by Upbound.
https://marketplace.upbound.io/providers/upbound/provider-aws
Apache License 2.0
137 stars 112 forks source link

feat(opensearch): domain add more selector and references #1337

Closed haarchri closed 3 weeks ago

haarchri commented 1 month ago

Description of your changes

Fixes #

I have:

How has this code been tested

we want to migrate from community provider - this PR adds missing Selector & References:

https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.48.1/resources/opensearchservice.aws.crossplane.io/Domain/v1alpha1#doc:spec-forProvider-encryptionAtRestOptions-kmsKeyIdRef

https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.48.1/resources/opensearchservice.aws.crossplane.io/Domain/v1alpha1#doc:spec-forProvider-encryptionAtRestOptions-kmsKeyIdSelector

https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.48.1/resources/opensearchservice.aws.crossplane.io/Domain/v1alpha1#doc:spec-forProvider-vpcOptions-securityGroupIdRefs

https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.48.1/resources/opensearchservice.aws.crossplane.io/Domain/v1alpha1#doc:spec-forProvider-vpcOptions-securityGroupIdSelector

https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.48.1/resources/opensearchservice.aws.crossplane.io/Domain/v1alpha1#doc:spec-forProvider-vpcOptions-subnetIdRefs

https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.48.1/resources/opensearchservice.aws.crossplane.io/Domain/v1alpha1#doc:spec-forProvider-vpcOptions-subnetIdSelector

ytsarev commented 1 month ago

/test-examples="examples/opensearch/v1beta1/domain.yaml"

haarchri commented 1 month ago

@ytsarev

 [{0 OpenSearch
    logger.go:42: 08:44:38 | case/0-apply |         Domain "example-domain-name" already exists  []}]'
ytsarev commented 4 weeks ago

@haarchri can you please add ${Rand.RFC1123Subdomain} to the failing example as in https://github.com/crossplane/uptest?tab=readme-ov-file#injecting-dynamic-values-and-datasource so we would avoid clashes during the e2e test of the Domain?

ytsarev commented 4 weeks ago

/test-examples="examples/opensearch/v1beta1/domain.yaml"

haarchri commented 3 weeks ago

2024-06-04T11:13:01.8825601Z logger.go:42: 11:13:01 | case/0-apply | - lastTransitionTime: "2024-06-04T11:08:55Z" 2024-06-04T11:13:01.8827013Z logger.go:42: 11:13:01 | case/0-apply | message: 'async create failed: failed to create the resource: [{0 creating OpenSearch 2024-06-04T11:13:01.8828741Z logger.go:42: 11:13:01 | case/0-apply | Domain: ValidationException: 1 validation error detected: Value ''example-domain-name-op-3j4sno06'' 2024-06-04T11:13:01.8830434Z logger.go:42: 11:13:01 | case/0-apply | at ''domainName'' failed to satisfy constraint: Member must have length less 2024-06-04T11:13:01.8831647Z logger.go:42: 11:13:01 | case/0-apply | than or equal to 28 []}]'

ytsarev commented 3 weeks ago

/test-examples

haarchri commented 3 weeks ago

@ytsarev can we test the following example ? examples/opensearch/v1beta1/domain.yaml

ytsarev commented 3 weeks ago

/test-examples="examples/opensearch/v1beta1/domain.yaml"

haarchri commented 3 weeks ago

mhm - any ideas ? 

    logger.go:42: 14:40:31 | case/0-apply |     - lastTransitionTime: "2024-06-04T14:40:11Z"
    logger.go:42: 14:40:31 | case/0-apply |       message: 'update failed: async update failed: failed to update the resource:
    logger.go:42: 14:40:31 | case/0-apply |         [{0 updating OpenSearch Domain (arn:aws:es:us-west-1:153891904029:domain/op-jz2ysr4s):
    logger.go:42: 14:40:31 | case/0-apply |         ValidationException: A change/update is in progress. Please wait for it to
    logger.go:42: 14:40:31 | case/0-apply |         complete before requesting another change.  []}]'
haarchri commented 3 weeks ago

think its not realted to my change:

 kubectl describe domain.opensearch.aws.upbound.io/example
Name:         example
Namespace:    
Labels:       testing.upbound.io/example-name=example
Annotations:  crossplane.io/external-create-pending: 2024-06-04T15:36:03Z
              crossplane.io/external-create-succeeded: 2024-06-04T15:36:03Z
              crossplane.io/external-name: arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn
              meta.upbound.io/example-id: opensearch/v1beta1/domain
              upjet.upbound.io/test: true
API Version:  opensearch.aws.upbound.io/v1beta1
Kind:         Domain
Metadata:
  Creation Timestamp:  2024-06-04T15:36:00Z
  Finalizers:
    finalizer.managedresource.crossplane.io
  Generation:        3
  Resource Version:  3355
  UID:               e2af1582-b526-40be-8064-0cec18096bf9
Spec:
  Deletion Policy:  Delete
  For Provider:
    Auto Tune Options:
      Desired State:        ENABLED
      Rollback On Disable:  NO_ROLLBACK
    Cluster Config:
      Instance Type:  m4.large.search
    Cognito Options:
      Identity Pool Id:  
      Role Arn:          
      User Pool Id:      
    Domain Endpoint Options:
      Tls Security Policy:  Policy-Min-TLS-1-2-2019-07
    Domain Name:            op-jqg7nlkn
    Ebs Options:
      Ebs Enabled:  true
      Volume Size:  10
      Volume Type:  gp2
    Encrypt At Rest:
      Enabled:       false
    Engine Version:  OpenSearch_1.0
    Node To Node Encryption:
      Enabled:  false
    Off Peak Window Options:
      Enabled:  true
      Off Peak Window:
        Window Start Time:
          Hours:  5
    Region:       us-west-1
    Snapshot Options:
      Automated Snapshot Start Hour:  0
    Software Update Options:
    Tags:
      Crossplane - Kind:            domain.opensearch.aws.upbound.io
      Crossplane - Name:            example
      Crossplane - Providerconfig:  default
  Init Provider:
  Management Policies:
    *
  Provider Config Ref:
    Name:  default
Status:
  At Provider:
    Arn:  arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn
    Auto Tune Options:
      Desired State:        ENABLED
      Rollback On Disable:  NO_ROLLBACK
      Use Off Peak Window:  false
    Cluster Config:
      Cold Storage Options:
        Enabled:                      false
      Dedicated Master Count:         0
      Dedicated Master Enabled:       false
      Dedicated Master Type:          
      Instance Count:                 1
      Instance Type:                  m4.large.search
      Multi Az With Standby Enabled:  false
      Warm Count:                     0
      Warm Enabled:                   false
      Warm Type:                      
      Zone Awareness Enabled:         false
    Cognito Options:
      Enabled:           false
      Identity Pool Id:  
      Role Arn:          
      User Pool Id:      
    Dashboard Endpoint:  search-op-jqg7nlkn-phjcqlnsrmukuvxrg4yaixldn4.us-west-1.es.amazonaws.com/_dashboards
    Domain Endpoint Options:
      Custom Endpoint:                  
      Custom Endpoint Certificate Arn:  
      Custom Endpoint Enabled:          false
      Enforce Https:                    true
      Tls Security Policy:              Policy-Min-TLS-1-2-2019-07
    Domain Id:                          609897127049/op-jqg7nlkn
    Domain Name:                        op-jqg7nlkn
    Ebs Options:
      Ebs Enabled:  true
      Iops:         0
      Throughput:   0
      Volume Size:  10
      Volume Type:  gp2
    Encrypt At Rest:
      Enabled:        false
      Kms Key Id:     
    Endpoint:         search-op-jqg7nlkn-phjcqlnsrmukuvxrg4yaixldn4.us-west-1.es.amazonaws.com
    Engine Version:   OpenSearch_1.0
    Id:               arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn
    Kibana Endpoint:  search-op-jqg7nlkn-phjcqlnsrmukuvxrg4yaixldn4.us-west-1.es.amazonaws.com/_plugin/kibana/
    Node To Node Encryption:
      Enabled:  false
    Off Peak Window Options:
      Enabled:  true
      Off Peak Window:
        Window Start Time:
          Hours:    5
          Minutes:  0
    Snapshot Options:
      Automated Snapshot Start Hour:  0
    Software Update Options:
      Auto Software Update Enabled:  false
    Tags:
      Crossplane - Kind:            domain.opensearch.aws.upbound.io
      Crossplane - Name:            example
      Crossplane - Providerconfig:  default
    Tags All:
      Crossplane - Kind:            domain.opensearch.aws.upbound.io
      Crossplane - Name:            example
      Crossplane - Providerconfig:  default
  Conditions:
    Last Transition Time:  2024-06-04T16:02:36Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-06-04T16:15:47Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
    Last Transition Time:  2024-06-04T15:47:33Z
    Reason:                Success
    Status:                True
    Type:                  LastAsyncOperation
Events:
  Type    Reason                   Age                   From                                                    Message
  ----    ------                   ----                  ----                                                    -------
  Normal  CreatedExternalResource  42m                   managed/opensearch.aws.upbound.io/v1beta1, kind=domain  Successfully requested creation of external resource
  Normal  UpdatedExternalResource  2m47s (x73 over 30m)  managed/opensearch.aws.upbound.io/v1beta1, kind=domain  Successfully requested update of external resource
2024-06-04T16:19:16Z    DEBUG   provider-aws    Diff detected   {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"advanced_security_options.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-06-04T16:19:16Z    DEBUG   provider-aws    Successfully requested update of external resource  {"controller": "managed/opensearch.aws.upbound.io/v1beta1, kind=domain", "request": {"name":"example"}, "uid": "e2af1582-b526-40be-8064-0cec18096bf9", "version": "3355", "external-name": "arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn", "requeue-after": "2024-06-04T16:29:14Z"}
2024-06-04T16:19:16Z    DEBUG   provider-aws    Async update starting...    {"trackerUID": "e2af1582-b526-40be-8064-0cec18096bf9", "resourceName": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "tfID": "arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn"}
2024-06-04T16:19:16Z    DEBUG   provider-aws    Updating the external resource  {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain"}
2024-06-04T16:19:20Z    DEBUG   provider-aws    Async update ended. {"trackerUID": "e2af1582-b526-40be-8064-0cec18096bf9", "resourceName": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "error": null, "tfID": "arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn"}
2024-06-04T16:19:20Z    DEBUG   provider-aws    Reconcile request has been requeued.    {"gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "name": "example", "rateLimiterName": "", "when": "0s"}
2024-06-04T16:19:20Z    DEBUG   provider-aws    Reconciling {"controller": "managed/opensearch.aws.upbound.io/v1beta1, kind=domain", "request": {"name":"example"}}
2024-06-04T16:19:20Z    DEBUG   provider-aws    Connecting to the service provider  {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain"}
2024-06-04T16:19:20Z    DEBUG   provider-aws    Observing the external resource {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain"}
2024-06-04T16:19:21Z    DEBUG   provider-aws    Diff detected   {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"advanced_security_options.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-06-04T16:19:21Z    DEBUG   provider-aws    Successfully requested update of external resource  {"controller": "managed/opensearch.aws.upbound.io/v1beta1, kind=domain", "request": {"name":"example"}, "uid": "e2af1582-b526-40be-8064-0cec18096bf9", "version": "3355", "external-name": "arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn", "requeue-after": "2024-06-04T16:29:45Z"}
2024-06-04T16:19:21Z    DEBUG   provider-aws    Async update starting...    {"trackerUID": "e2af1582-b526-40be-8064-0cec18096bf9", "resourceName": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "tfID": "arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn"}
2024-06-04T16:19:21Z    DEBUG   provider-aws    Updating the external resource  {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain"}
2024-06-04T16:19:22Z    DEBUG   provider-aws    Calling the inner handler for Update event. {"gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "name": "example", "queueLength": 0}
2024-06-04T16:19:22Z    DEBUG   provider-aws    Reconciling {"controller": "managed/opensearch.aws.upbound.io/v1beta1, kind=domain", "request": {"name":"example"}}
2024-06-04T16:19:22Z    DEBUG   provider-aws    Connecting to the service provider  {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain"}
2024-06-04T16:19:22Z    DEBUG   provider-aws    ongoing async operation {"uid": "e2af1582-b526-40be-8064-0cec18096bf9", "name": "example", "gvk": "opensearch.aws.upbound.io/v1beta1, Kind=Domain", "opType": "update"}
2024-06-04T16:19:22Z    DEBUG   provider-aws    Cannot delete external resource {"controller": "managed/opensearch.aws.upbound.io/v1beta1, kind=domain", "request": {"name":"example"}, "uid": "e2af1582-b526-40be-8064-0cec18096bf9", "version": "3846", "external-name": "arn:aws:es:us-west-1:609897127049:domain/op-jqg7nlkn", "deletion-timestamp": "2024-06-04 16:19:22 +0000 UTC", "error": "update operation that started at 2024-06-04 16:19:21.830019794 +0000 UTC m=+1214.870558596 is still running", "errorVerbose": "update operation that started at 2024-06-04 16:19:21.830019794 +0000 UTC m=+1214.870558596 is still running\ngithub.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Delete\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/controller/external_async_tfpluginsdk.go:193\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.2.0.20240510094504-3f697876fa57/pkg/reconciler/managed/reconciler.go:954\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.2.0.20240510094504-3f697876fa57/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_arm64.s:1222"}
ytsarev commented 3 weeks ago

i trust we can merge the selectors and address the MR behaviour separately