search

crossplane-contrib / function-patch-and-transform

A patch & transform composition function
https://crossplane.io
Apache License 2.0
23 stars 24 forks source link

multistep doesn't seem to be working #132

Closed mlubanski closed 2 months ago

mlubanski commented 2 months ago

I was trying follow https://github.com/crossplane-contrib/function-patch-and-transform/tree/main/example/multistep but in CompositeResource I am getting

│   Warning  ComposeResources   27s (x6 over 52s)  defined/compositeresourcedefinition.apiextensions.crossplane.io  c │
│ annot compose resources: pipeline step "patch-and-transform" returned a fatal result: composed resource "dbcluster" │
│  has no base template, and was not produced by a previous Function in the pipeline

Composition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  labels:
    crossplane.io/xrd: xmysqlclusters.rds.aws.crossplane.sinch.com
  name: mysqlclusters.rds.aws.crossplane.sinch.com-v1alpha1
spec:
  compositeTypeRef:
    apiVersion: rds.aws.crossplane.sinch.com/v1alpha1
    kind: XMySQLCluster
  environment:
    environmentConfigs:
    - ref:
        name: mysqlcluster
      type: Reference
  publishConnectionDetailsWithStoreConfigRef:
    name: default
  mode: Pipeline
  pipeline:
  - step: generate-dbcluster
    functionRef:
      name: function-go-templating
    input:
      apiVersion: gotemplating.fn.crossplane.io/v1beta1
      kind: GoTemplate
      source: Inline
      inline:
        template: |
          {{- $parameters := .observed.composite.resource.spec.parameters -}}
          ---
          apiVersion: rds.aws.crossplane.io/v1alpha1
          kind: DBCluster
          metadata:
            annotations:
              {{ setResourceNameAnnotation "globalcluster" }}
              {{- if $parameters.globalClusterIdentifier }}
              isGlobalClusterSpecified: "yes"
              {{- else }}
              isGlobalClusterSpecified: "no"
              {{- end }}
          spec:
            forProvider:
              region: eu-west-1
              engine: aurora-mysql
              {{- if $parameters.globalClusterIdentifier }}
              globalClusterIdentifier: {{ $parameters.globalClusterIdentifier }}
              autogeneratePassword: false
              {{- else }}
              masterUsername: root
              autogeneratePassword: true
              {{- end }}
              backupRetentionPeriod: 1
              copyTagsToSnapshot: true
              engine: aurora-mysql
              skipFinalSnapshot: true
              storageEncrypted: true
              tags:
              - key: OpsAutomator-Snap-Daily
                value: "true"
              vpcSecurityGroupIDs: [] 
              enableCloudwatchLogsExports:
              - audit
              autoMinorVersionUpgrade: true
            managementPolicies:
            - '*'
            providerConfigRef:
              name: crossplane-provider-aws

  - step: patch-and-transform
    functionRef:
      name: function-patch-and-transform
    input:
      apiVersion: pt.fn.crossplane.io/v1beta1
      environment: null
      kind: Resources
      patchSets: []
      resources:
      - base:
          apiVersion: ec2.aws.crossplane.io/v1beta1
          kind: SecurityGroup
          spec:
            forProvider:
              description: Security group for database
              egress:
              - fromPort: -1
                ipProtocol: "-1"
                ipRanges:
                - cidrIp: 0.0.0.0/0
                toPort: -1
              ingress:
              - fromPort: 3306
                ipProtocol: tcp
                toPort: 3306
                userIdGroupPairs:
                - description: Access from the k8s cluster
            providerConfigRef:
              name: crossplane-provider-aws
        name: securitygroup
        patches:
        - fromFieldPath: metadata.annotations[mysqlcluster.crossplane.sinch.com/securitygroup-external-name]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/standardizedCloudNameVersion2]
          toFieldPath: spec.forProvider.groupName
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsClusterSecurityGroup]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.ingress[0].userIdGroupPairs[0].groupId
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsRegion]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.region
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsVpcId]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.vpcId
          type: FromCompositeFieldPath
        - fromFieldPath: status.atProvider.securityGroupID
          toFieldPath: mysqlcluster_sg
          type: ToEnvironmentFieldPath
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.parameters.tags"
          toFieldPath: "spec.forProvider.tags"
          policy:
            toFieldPath: AppendArray
      - base:
          apiVersion: rds.aws.crossplane.io/v1alpha1
          kind: DBClusterParameterGroup
          spec:
            forProvider:
              description: Parameter group for Aurora Cluster
              parameters:
              - parameterName: server_audit_logs_upload
                parameterValue: "1"
                applyMethod: immediate
              - parameterName: server_audit_logging
                parameterValue: "1"
                applyMethod: immediate
              - parameterName: server_audit_excl_users
                parameterValue: rdsadmin
                applyMethod: immediate
              - parameterName: server_audit_events
                parameterValue: CONNECT,TABLE,QUERY_DCL,QUERY_DDL
                applyMethod: immediate
            providerConfigRef:
              name: crossplane-provider-aws              
        name: parametergroup
        patches:
        - fromFieldPath: spec.parameters.majorEngineVersion
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.dbParameterGroupFamily
          transforms:
          - string:
              fmt: aurora-mysql%s
              type: Format
            type: string
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsRegion]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.region
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/standardizedCloudNameVersion2]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[mysqlcluster.crossplane.sinch.com/dbclusterparametergroup-external-name]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: FromCompositeFieldPath
        - fromFieldPath: spec.parameters.dbClusterParameterGroupParameters
          toFieldPath: spec.forProvider.parameters
          type: FromCompositeFieldPath
          policy:
            toFieldPath: AppendArray
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.parameters.tags"
          toFieldPath: "spec.forProvider.tags"
          policy:
            toFieldPath: AppendArray
      - name: globalcluster
        base:
          apiVersion: rds.aws.crossplane.io/v1alpha1
          kind: GlobalCluster
          spec:
            forProvider:
              sourceDBClusterIdentifierSelector:
                matchControllerRef: true
            managementPolicies:
            - '*'
            providerConfigRef:
              name: crossplane-provider-aws
        patches:
        - combine:
            strategy: string
            string:
              fmt: '%s-global'
            variables:
            - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/standardizedCloudNameVersion2]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: CombineFromComposite        
        - fromFieldPath: metadata.annotations[mysqlcluster.crossplane.sinch.com/mysqlglobalcluster-external-name]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsRegion]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.region
          type: FromCompositeFieldPath
        - fromFieldPath: status.atProvider.endpoint
          toFieldPath: mysqlglobalcluster_endpoint
          type: ToEnvironmentFieldPath
          transforms:
          - map:
              Orphan: "true"
              Delete: "false"
            type: map
          - convert:
              toType: bool
            type: convert
          type: FromCompositeFieldPath  
      - name: dbcluster
        patches:
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/standardizedCloudNameVersion2]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[mysqlcluster.crossplane.sinch.com/mysqlcluster-external-name]
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          type: FromCompositeFieldPath
        - combine:
            strategy: string
            string:
              fmt: '%s-%s-mysqlcluster'
            variables:
            - fromFieldPath: spec.claimRef.namespace
            - fromFieldPath: spec.claimRef.name
          toFieldPath: spec.publishConnectionDetailsTo.name
          type: CombineFromComposite
        - fromFieldPath: spec.parameters.storageType
          toFieldPath: spec.forProvider.storageType
          type: FromCompositeFieldPath
        - fromFieldPath: spec.parameters.preferredMaintenanceWindow
          toFieldPath: spec.forProvider.preferredMaintenanceWindow
          type: FromCompositeFieldPath
        - fromFieldPath: spec.parameters.preferredBackupWindow
          toFieldPath: spec.forProvider.preferredBackupWindow
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsDbSubnetGroupName]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.dbSubnetGroupName
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/awsRegion]
          policy:
            fromFieldPath: Required
          toFieldPath: spec.forProvider.region
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/standardizedCloudNameVersion2]
          toFieldPath: spec.forProvider.dbClusterParameterGroupName
          type: FromCompositeFieldPath
        - fromFieldPath: metadata.annotations[mysqlcluster.crossplane.sinch.com/dbclusterparametergroup-external-name]
          toFieldPath: spec.forProvider.dbClusterParameterGroupName
          type: FromCompositeFieldPath
        - fromFieldPath: status.atProvider.status
          toFieldPath: status.databaseStatus
          type: ToCompositeFieldPath
        - fromFieldPath: metadata.annotations[core.crossplane.sinch.com/rdsKmsKeyId]
          toFieldPath: spec.forProvider.kmsKeyID
          type: FromCompositeFieldPath
        - fromFieldPath: spec.parameters.kmsKeyId
          toFieldPath: spec.forProvider.kmsKeyID
          type: FromCompositeFieldPath
        - fromFieldPath: spec.parameters.masterUserPasswordSecretRef
          toFieldPath: spec.forProvider.masterUserPasswordSecretRef
          type: FromCompositeFieldPath
        - fromFieldPath: spec.parameters.applyModificationsImmediately
          toFieldPath: spec.forProvider.applyImmediately
          type: FromCompositeFieldPath
        # adding SG which allows connect app running in same k8s cluster
        - fromFieldPath: mysqlcluster_sg
          toFieldPath: spec.forProvider.vpcSecurityGroupIDs[0]
          type: FromEnvironmentFieldPath          
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.parameters.tags"
          toFieldPath: "spec.forProvider.tags"
          policy:
            toFieldPath: AppendArray
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.enableIAMDatabaseAuthentication
          toFieldPath: spec.forProvider.enableIAMDatabaseAuthentication
  - step: add-read-replica
    functionRef:
      name: function-cel-filter
    input:
      apiVersion: cel.fn.crossplane.io/v1beta1
      kind: Filters
      filters:
      - name: globalcluster
        expression: observed.composite.resource.spec.parameters.addGloblalCluster == true
  - step: automatically-detect-ready-composed-resources
    functionRef:
      name: function-auto-ready

crossplane 1.16.0 function-patch-and-transform: v0.6.0 function-go-templating: v0.4.1

mlubanski commented 2 months ago

I just figured out that I was using different resource names Everything works as expected, we can close that ticket

P.S. BTW this feature is very powerful and awesome!