kubernetes_manifest can't apply valid yml

davidgiga1993 commented 1 year ago

Terraform Version, Provider Version and Kubernetes Version

terraform -v
Terraform v1.4.6
on windows_amd64
+ provider registry.terraform.io/hashicorp/archive v2.3.0
+ provider registry.terraform.io/hashicorp/aws v5.0.1
+ provider registry.terraform.io/hashicorp/helm v2.9.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.20.0

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: companyrdscomposition
  labels:
    # An optional convention is to include a label of the XRD. This allows
    # easy discovery of compatible Compositions.
    crossplane.io/xrd: xdbinstances.company.com
    # The following label marks this Composition for GCP. This label can
    # be used in 'compositionSelector' in an XR or Claim.
    provider: provider-aws
spec:

  # Each Composition must declare that it is compatible with a particular type
  # of Composite Resource using its 'compositeTypeRef' field. The referenced
  # version must be marked 'referenceable' in the XRD that defines the XR.
  compositeTypeRef:
    apiVersion: dbinstances.company.com/v1alpha1
    kind: XDbInstance

  # Each Composition must specify at least one composed resource template. In
  # this case the Composition tells Crossplane that it should create, update, or
  # delete a CloudSQLInstance whenever someone creates, updates, or deletes an
  # XPostgresSQLInstance.
  resources:

    # It's good practice to provide a unique name for each entry. Note that
    # this identifies the resources entry within the Composition - it's not
    # the name the CloudSQLInstance. The 'name' field will be required in a
    # future version of this API.
    - name: rdsSubnetGroup

      # The 'base' template for the CloudSQLInstance Crossplane will create.
      # You can use the base template to specify fields that never change, or
      # default values for fields that may optionally be patched over. Bases must
      # be a valid Crossplane resource - a Managed Resource, Composite Resource,
      # or a ProviderConfig.
      base:
        apiVersion: database.aws.crossplane.io/v1beta1
        kind: DBSubnetGroup
        metadata:
          name: workload-subnet
        spec:
          forProvider:
            region: eu-central-1
            description: "Subnet for crossplane RDS instances"
            subnetIds:
              # AZ a and b
              - subnet-03acd8cfcba74c16c
              - subnet-0827394c90552c6e0
          providerRef:
            name: provider-aws

      # Each resource can optionally specify a set of 'patches' that copy fields
      # from (or to) the XR.
      patches:
        - type: PatchSet
          patchSetName: metadata
    - name: rds
      base:
        apiVersion: database.aws.crossplane.io/v1beta1
        kind: RDSInstance
        metadata:
          name: rds-example
          namespace: david-test
        spec:
          providerRef:
            name: provider-aws
          forProvider:
            region: eu-central-1
            dbInstanceClass: db.t3.micro
            masterUsername: mainuser
            engine: mysql
            engineVersion: "8.0.32"
            skipFinalSnapshotBeforeDeletion: true
            # Reuse portal for testing
            dbName: RdsExample

            multiAZ: false
            dbSubnetGroupName: "workload-subnet"
            backupRetentionPeriod: 2

            storageType: gp3
            allocatedStorage: 20
            maxAllocatedStorage: 25
            storageEncrypted: true
            kmsKeyId: "alias/aws/rds"

      patches:
        - type: PatchSet
          patchSetName: metadata

        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.storageGB
          toFieldPath: spec.forProvider.allocatedStorage
          transforms:
            - type: math
              math:
                type: ClampMin
                clampMin: 20

        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.storageGB
          toFieldPath: spec.forProvider.maxAllocatedStorage
          transforms:
            - type: math
              math:
                type: ClampMin
                clampMin: 20
            - type: math
              math:
                type: Multiply
                multiply: 2

        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.dbName
          toFieldPath: spec.forProvider.dbName

        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.dbInstanceClass
          toFieldPath: spec.forProvider.dbInstanceClass

        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.engine
          toFieldPath: spec.forProvider.engine

        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.engineVersion
          toFieldPath: spec.forProvider.engineVersion

      connectionDetails:
        - name: endpoint
          fromConnectionSecretKey: endpoint
        - name: password
          fromConnectionSecretKey: password
        - name: port
          fromConnectionSecretKey: port
        - name: username
          fromConnectionSecretKey: username

  # If you find yourself repeating patches a lot you can group them as a named
  # 'patch set' then use a PatchSet type patch to reference them.
  patchSets:
    - name: metadata
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.namespace
          toFieldPath: spec.forProvider.tags[InvoiceToProductName]

resource "kubernetes_manifest" "db-composition" {
  manifest = yamldecode(file("${path.module}/db-composition.yml"))
}

Debug Output

Due to confidentiality of the project I can't provide the full debug output

Steps to Reproduce

terraform apply

Expected Behavior

Terraform should be able to import any valid k8s yml files

Actual Behavior

Terraform type error:

│ Error: Failed to set computed attributes in new resource state
│
│   with module.crossplane-xrd.kubernetes_manifest.db-composition,
│   on crossplane-xrd\xrd.tf line 5, in resource "kubernetes_manifest" "db-composition":
│    5: resource "kubernetes_manifest" "db-composition" {
│
│ AttributeName("transforms"): can't use tftypes.Tuple[tftypes.Object["convert":tftypes.Object["format":tftypes.String, "toType":tftypes.String], "map":tftypes.Map[tftypes.DynamicPseudoType],
│ "match":tftypes.Object["fallbackTo":tftypes.String, "fallbackValue":tftypes.DynamicPseudoType, "patterns":tftypes.Tuple[tftypes.Object["literal":tftypes.String, "regexp":tftypes.String, "result":tftypes.DynamicPseudoType,     
│ "type":tftypes.String]]], "math":tftypes.Object["clampMax":tftypes.Number, "clampMin":tftypes.Number, "multiply":tftypes.Number, "type":tftypes.String], "string":tftypes.Object["convert":tftypes.String, "fmt":tftypes.String,  
│ "regexp":tftypes.Object["group":tftypes.Number, "match":tftypes.String], "trim":tftypes.String, "type":tftypes.String], "type":tftypes.String]] as tftypes.Tuple[tftypes.Object["convert":tftypes.Object["format":tftypes.String, 
│ "toType":tftypes.String], "map":tftypes.Map[tftypes.DynamicPseudoType], "match":tftypes.Object["fallbackTo":tftypes.String, "fallbackValue":tftypes.DynamicPseudoType,
│ "patterns":tftypes.Tuple[tftypes.Object["literal":tftypes.String, "regexp":tftypes.String, "result":tftypes.DynamicPseudoType, "type":tftypes.String]]], "math":tftypes.Object["clampMax":tftypes.Number,
│ "clampMin":tftypes.Number, "multiply":tftypes.Number, "type":tftypes.String], "string":tftypes.Object["convert":tftypes.String, "fmt":tftypes.String, "regexp":tftypes.Object["group":tftypes.Number, "match":tftypes.String],    
│ "trim":tftypes.String, "type":tftypes.String], "type":tftypes.String], tftypes.Object["convert":tftypes.Object["format":tftypes.String, "toType":tftypes.String], "map":tftypes.Map[tftypes.DynamicPseudoType],
│ "match":tftypes.Object["fallbackTo":tftypes.String, "fallbackValue":tftypes.DynamicPseudoType, "patterns":tftypes.Tuple[tftypes.Object["literal":tftypes.String, "regexp":tftypes.String, "result":tftypes.DynamicPseudoType,     
│ "type":tftypes.String]]], "math":tftypes.Object["clampMax":tftypes.Number, "clampMin":tftypes.Number, "multiply":tftypes.Number, "type":tftypes.String], "string":tftypes.Object["convert":tftypes.String, "fmt":tftypes.String,  
│ "regexp":tftypes.Object["group":tftypes.Number, "match":tftypes.String], "trim":tftypes.String, "type":tftypes.String], "type":tftypes.String]]

Important Factoids

Classic kubernetes provider setup. I can import the yml using kubectl without any issues.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

alexsomesan commented 1 year ago

Hi,

the core of the issue here is that the schema for compositions.apiextensions.crossplane.io CRD is incomplete and uses x-kubernetes-preserve-unknown-fields wildcards which allow for ambiguity in the schema. The provider is unable to establish an exact data type for the attributes that are affected by this as required by Terraform and errs on the side of caution because you could otherwise end up with a broken Terraform state later on as those values get updated.

We'll have a to look a bit deeper into this specific CRD to determine if any workarounds are possible. I'll update here as we find out more.

davidgiga1993 commented 1 year ago

Ah I see, thanks a lot for analyzing the issue, would be cool if that would in the future. For now we just use octoploy for the crossplane xrd part.

jcogilvie commented 1 year ago

Hey @alexsomesan, I'm running into possibly a related error.

I am using a CRD which has, in its latest version, set x-kubernetes-preserve-unknown-fields. It was working before this.

The link to the CRD diff is here: https://github.com/argoproj/argo-cd/compare/v2.6.7...v2.8.2#diff-1f5eb01e71d24fec7b77c000aee8ac6cdeffce1403021bd72b1f891809645da0

(I apologize, you'll have to click "files changed" and then ctrl+f "application-crd.yaml" since deeplinking on github doesn't seem to work for this case.)

Notably there is this addition:

                          valuesObject:
                            description: ValuesObject specifies Helm values to be
                              passed to helm template, defined as a map. This takes
                              precedence over Values.
                            type: object
                            x-kubernetes-preserve-unknown-fields: true

Since the change (to sources.helm.valuesObject), I'm getting:

│ Error: Failed to transform List value into Tuple of different length
│ 
│   with module.argo_app.kubernetes_manifest.raw[0],
│   on .terraform/modules/argo_app/main.tf line 172, in resource "kubernetes_manifest" "raw":
│  172: resource "kubernetes_manifest" "raw" {
│ 
│ Error: %!s(<nil>)
│ ...at attribute:
│ spec.sources

The error happens at plan time.

My code is thus:

# locals  
  multiple_sources = [for source in var.services : {
    repo_url        = local.helm_repo_url
    chart           = source.source_chart
    path            = null
    target_revision = source.source_chart_version
    helm = {
      release_name = source.name
      values       = source.helm_values
    }
  }]
  sources_map     = { for source in local.sources : source.helm.release_name => source }

# ...
# then inside kubernetes_manifest:

      sources = [for source, config in local.sources_map : {
        repoURL        = config.repo_url
        path           = config.path
        chart          = config.chart
        targetRevision = config.target_revision
        helm = {
          releaseName  = config.helm.release_name
          values       = config.helm.values
          #valuesObject would live here if I was setting it
        }
      }]

It seems like there is a similar issue with the provider on a different CR, so I'm wondering if these are all related to unknown fields.

Is there a clean way to make the provider ignore the problematic field(s)?

github-actions[bot] commented 12 hours ago

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

hashicorp / terraform-provider-kubernetes