crossplane-contrib / x-generation

generate compositions from crossplane provider crds
Apache License 2.0
44 stars 18 forks source link

Patches of nested attribute for some parameters are not being generated #31

Closed jehadnasser closed 4 months ago

jehadnasser commented 4 months ago

What happened?

After migrating RDS dbcluster to Upbound official provider and update the generate.yaml file accordingly, the generated composition had only the object/array name patch for the restoreToPointInTime field and ignored any nested fields underneath restoreToPointInTime. Similar behaviour happened with: s3Import, scalingConfiguration, and serverlessv2ScalingConfiguration. Unlike the normal behaviour with the old provider where there're patches for all nested attribute of similar nested fields. Except for scalingConfiguration, the other three are new parameters, or let's say they got restructured with the Upbound provider.

How can we reproduce it?

whereas we expects patches for the whole new structure

- restoreToPointInTime, type array
      - restoreToTime, type string
      - restoreType, type string
      - sourceClusterIdentifier, type string
      - sourceClusterIdentifierRef, type object
        - name, type string
        - policy, type object
          - resolution, type string
          - resolve, type string
      - sourceClusterIdentifierSelector, type object
        - matchControllerRef, type boolean
        - matchLabels, type object
        - policy, type object
          - resolution, type string
          - resolve, type string
      - useLatestRestorableTime, type boolean
- s3Import, type array
  - bucketName, type string
  - bucketNameRef, type object
    - name, type string
    - policy, type object
      - resolution, type string
      - resolve, type string
  - bucketNameSelector, type object
    - matchControllerRef, type boolean
    - matchLabels, type object
    - policy, type object
      - resolution, type string
      - resolve, type string
  - bucketPrefix, type string
  - ingestionRole, type string
  - sourceEngine, type string
  - sourceEngineVersion, type string
- scalingConfiguration, type array
      - autoPause, type boolean
      - maxCapacity, type number
      - minCapacity, type number
      - secondsUntilAutoPause, type number
      - timeoutAction, type string
- serverlessv2ScalingConfiguration, type array
      - maxCapacity, type number
      - minCapacity, type number

What environment did it happen in?

haarchri commented 4 months ago

the patches are fine for arrays like restoreToPointInTime or scalingConfiguration

we can use the following generator:

group: rds.aws.example.cloud
name: DBCluster
version: v1alpha1
provider:
  baseURL: https://raw.githubusercontent.com/upbound/%s/%s/package/crds/%s
  name: provider-aws
  version: v1.6.0
  crd:
    file: rds.aws.upbound.io_clusters.yaml
    version: v1beta1
ignore: false
usePipeline: true # <-- set pipeline so we can test with crossplane beta render
connectionSecretKeys:
  - username
  - password
  - endpoint
  - port
compositions:
  - name: compositedbcluster.rds.aws.example.cloud
    provider: example
    default: true

lets use the following example:

apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
  namespace: assert-db
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    skipFinalSnapshot: true
    scalingConfiguration:
      - autoPause: true
        maxCapacity: 8
  writeConnectionSecretToRef:
    name: sample-rds-cluster-secret
    namespace: upbound-system

use crossplane beta render:

crossplane beta render examples/rds/db-cluster.yaml package/RDS-Cluster/composition-compositedbcluster.rds.aws.example.cloud.yaml examples/functions.yaml 
---
apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
status:
  conditions:
  - lastTransitionTime: "2024-01-01T00:00:00Z"
    message: 'Unready resources: Cluster'
    reason: Creating
    status: "False"
    type: Ready
---
apiVersion: rds.aws.upbound.io/v1beta1
kind: Cluster
metadata:
  annotations:
    crossplane.io/composition-resource-name: Cluster
  generateName: example-db-cluster-
  labels:
    commonLabelA: commonLabelAValue
    commonLabelB: commonLabelBValue
    crossplane.io/composite: example-db-cluster
  ownerReferences:
  - apiVersion: rds.aws.example.cloud/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: example-db-cluster
    uid: ""
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    scalingConfiguration:
    - autoPause: true
      maxCapacity: 8
    skipFinalSnapshot: true
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    namespace: crossplane-system

so you can see scalingConfiguration is patched correctly

haarchri commented 4 months ago

please use provider-aws 1.7.0 and if its possible v1beta2 for DBCluster - check release notes for cross-reference: https://github.com/crossplane-contrib/provider-upjet-aws/releases/tag/v1.7.0

i highly recommend to use a generator config like:

group: rds.aws.example.cloud
name: DBCluster
version: v1alpha1
provider:
  baseURL: https://raw.githubusercontent.com/upbound/%s/%s/package/crds/%s
  name: provider-aws
  version: v1.7.0 # use this version to get rid of the 1 element arrays
  crd:
    file: rds.aws.upbound.io_clusters.yaml
    version: v1beta2 # bump to v1beta2
ignore: false
usePipeline: true
connectionSecretKeys:
  - username
  - password
  - endpoint
  - port
compositions:
  - name: compositedbcluster.rds.aws.example.cloud
    provider: example
    default: true

lets use the following example:

apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
  namespace: assert-db
  annotations:
    render.crossplane.io/composition-path: package/RDS-Cluster/composition-compositedbcluster.rds.aws.example.cloud.yaml
    render.crossplane.io/function-path: examples/functions.yaml
    render.crossplane.io/environment-path: test/environmentconfig.yaml
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    skipFinalSnapshot: true
    scalingConfiguration:
      autoPause: true
      maxCapacity: 8
  writeConnectionSecretToRef:
    name: sample-rds-cluster-secret
    namespace: upbound-system

use crossplane beta render:

crossplane beta render examples/rds/db-cluster.yaml package/RDS-Cluster/composition-compositedbcluster.rds.aws.example.cloud.yaml examples/functions.yaml 
---
apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
status:
  conditions:
  - lastTransitionTime: "2024-01-01T00:00:00Z"
    message: 'Unready resources: Cluster'
    reason: Creating
    status: "False"
    type: Ready
---
apiVersion: rds.aws.upbound.io/v1beta2
kind: Cluster
metadata:
  annotations:
    crossplane.io/composition-resource-name: Cluster
  generateName: example-db-cluster-
  labels:
    commonLabelA: commonLabelAValue
    commonLabelB: commonLabelBValue
    crossplane.io/composite: example-db-cluster
  ownerReferences:
  - apiVersion: rds.aws.example.cloud/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: example-db-cluster
    uid: ""
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    scalingConfiguration:
      autoPause: true
      maxCapacity: 8
    skipFinalSnapshot: true
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    namespace: crossplane-system