search

crossplane-contrib / provider-kubernetes

Crossplane provider to provision and manage Kubernetes objects on (remote) Kubernetes clusters.
Apache License 2.0
140 stars 81 forks source link

k8s configMap doesn't change #145

Closed NikitaCloudRuntime closed 2 weeks ago

NikitaCloudRuntime commented 11 months ago

What happened?

EKS ConfigMap doesn't not change

How can we reproduce it?

As part of provisioning of a new EKS cluster we want to modify the aws-auth configMap in order to map SSO IAM roles there. The Composite looks like the following

    - base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          deletionPolicy: Orphan
          providerConfigRef: {}
          forProvider:
            manifest:
              apiVersion: v1
              kind: ConfigMap
              metadata:
                name: aws-auth
                namespace: kube-system
              data:
                mapAccounts: ""
                mapRoles: ""
                mapUsers: ""
      name: eks-aws-auth-config
      patches:
        - fromFieldPath: metadata.name
          toFieldPath: spec.providerConfigRef.name
          policy:
            fromFieldPath: Required    

        - fromFieldPath: spec.parameters.awsAuth.mapAccounts
          toFieldPath: spec.forProvider.manifest.data.mapAccounts
        - fromFieldPath: spec.parameters.awsAuth.mapUsers
          toFieldPath: spec.forProvider.manifest.data.mapUsers

        - type: CombineFromComposite
          toFieldPath: spec.forProvider.manifest.data.mapRoles
          combine:
            variables:
            - fromFieldPath: status.ngRoleArn
            - fromFieldPath: spec.parameters.awsAuth.mapRoles
            strategy: string
            string:
              fmt: |
                - rolearn: %s
                  username: system:node:{{EC2PrivateDNSName}}
                  groups:
                  - system:bootstrappers
                  - system:nodes
                %s

Everything works well, we can connect the cluster after provisioning but we noticed that if we try to modify the configMap manually, Crossplane Kubernetes provider doesn't override this change

So, let's assume that we have the following configMap in the new cluster

k get configmap -n kube-system aws-auth -o yaml
apiVersion: v1
data:
  mapAccounts: ""
  mapRoles: |+
    - rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes
    - groups:
      - system:masters
      rolearn: arn:aws:iam::***:role/***
      username: ***

  mapUsers: ""
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","data":{"mapAccounts":"","mapRoles":"-
      rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5\n  username: system:node:{{EC2PrivateDNSName}}\n  groups:\n  -
      system:bootstrappers\n  - system:nodes\n- groups:\n  - system:masters\n  rolearn:
      arn:aws:iam::***:role/***\n  username:
      sso-eks-test-admin\n\n","mapUsers":""},"kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system"}}'
  creationTimestamp: "2023-10-09T09:11:35Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "1029"
  uid: a3b0ed75-ef77-461d-bb87-2da000ff0201

and as soon as we modify it(set up wrong group for the second role)

k edit configmaps -n kube-system aws-auth      
configmap/aws-auth edited

k get configmap -n kube-system aws-auth -o yaml
apiVersion: v1
data:
  mapAccounts: ""
  mapRoles: |+
    - rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes
    - groups:
      - system
      rolearn: arn:aws:iam::***:role/***
      username: ***

  mapUsers: ""
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","data":{"mapAccounts":"","mapRoles":"-
      rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5\n  username: system:node:{{EC2PrivateDNSName}}\n  groups:\n  -
      system:bootstrappers\n  - system:nodes\n- groups:\n  - system:masters\n  rolearn:
      arn:aws:iam::***:role/***\n  username:
      sso-eks-test-admin\n\n","mapUsers":""},"kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system"}}'
  creationTimestamp: "2023-10-09T09:11:35Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "47148"
  uid: a3b0ed75-ef77-461d-bb87-2da000ff0201

the controller sees this change

k get object hellofresh-eks-bcmc4  -o yaml
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
metadata:
  annotations:
    crossplane.io/composition-resource-name: eks-aws-auth-config
    crossplane.io/external-name: hellofresh-eks-bcmc4
  creationTimestamp: "2023-10-09T08:58:43Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: hellofresh-eks-
  generation: 3
  labels:
    crossplane.io/claim-name: ""
    crossplane.io/claim-namespace: ""
    crossplane.io/composite: hellofresh-eks
  name: hellofresh-eks-bcmc4
  ownerReferences:
  - apiVersion: aws.hellofresh.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: eks
    name: hellofresh-eks
    uid: 9d89b96f-dd1b-4887-85a9-79718a30bfb7
  resourceVersion: "12976254"
  uid: 445c8ee7-6b5b-455f-aee0-8d772978bbc4
spec:
  deletionPolicy: Orphan
  forProvider:
    manifest:
      apiVersion: v1
      data:
        mapAccounts: ""
        mapRoles: |+
          - rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5
            username: system:node:{{EC2PrivateDNSName}}
            groups:
            - system:bootstrappers
            - system:nodes
          - groups:
            - system:masters
            rolearn: arn:aws:iam::***:role/***
            username: ***

        mapUsers: ""
      kind: ConfigMap
      metadata:
        name: aws-auth
        namespace: kube-system
  managementPolicy: Default
  providerConfigRef:
    name: hellofresh-eks
  readiness:
    policy: SuccessfulCreate
status:
  atProvider:
    manifest:
      apiVersion: v1
      data:
        mapAccounts: ""
        mapRoles: |+
          - rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5
            username: system:node:{{EC2PrivateDNSName}}
            groups:
            - system:bootstrappers
            - system:nodes
          - groups:
            - system
            rolearn: arn:aws:iam::***:role/***
            username: ***

        mapUsers: ""
      kind: ConfigMap
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","data":{"mapAccounts":"","mapRoles":"-
            rolearn: arn:aws:iam::***:role/hellofresh-eks-cbtc5\n  username:
            system:node:{{EC2PrivateDNSName}}\n  groups:\n  - system:bootstrappers\n  -
            system:nodes\n- groups:\n  - system:masters\n  rolearn: arn:aws:iam::***:role/***\n  username:
            ***\n\n","mapUsers":""},"kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system"}}'
        creationTimestamp: "2023-10-09T09:11:35Z"
        managedFields:
        - apiVersion: v1
          fieldsType: FieldsV1
          fieldsV1:
            f:data: {}
          manager: vpcLambda
          operation: Update
          time: "2023-10-09T09:11:35Z"
        - apiVersion: v1
          fieldsType: FieldsV1
          fieldsV1:
            f:data:
              f:mapAccounts: {}
              f:mapUsers: {}
            f:metadata:
              f:annotations:
                .: {}
                f:kubectl.kubernetes.io/last-applied-configuration: {}
          manager: crossplane-kubernetes-provider
          operation: Update
          time: "2023-10-09T09:11:45Z"
        - apiVersion: v1
          fieldsType: FieldsV1
          fieldsV1:
            f:data:
              f:mapRoles: {}
          manager: kubectl-edit
          operation: Update
          time: "2023-10-09T13:03:50Z"
        name: aws-auth
        namespace: kube-system
        resourceVersion: "47148"
        uid: a3b0ed75-ef77-461d-bb87-2da000ff0201
  conditions:
  - lastTransitionTime: "2023-10-09T09:11:45Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2023-10-09T09:11:45Z"
    reason: Available
    status: "True"
    type: Ready

but doesn't revert it. I assume it's because the last applied configuration doesn't represent this change.

So, either we don't really understand the logic behind this new concept or something goes wrong

What environment did it happen in?

Crossplane Version: v1.13.2 AWS provider versions: v0.41.0 Kubernetes provider version: v0.9.0 Kubernetes Version: 1.25 Kubernetes Distribution: EKS

haarchri commented 11 months ago

this should help with your issue described here: https://github.com/crossplane-contrib/provider-kubernetes/pull/134

We are using last applied configuration since for some k8s resources, API Server fills some fields after you create the resource hence once you get it back you would always see it is different than what you want to apply. see for ref: https://github.com/crossplane-contrib/provider-kubernetes/issues/37#issuecomment-1122145690

NikitaCloudRuntime commented 11 months ago

@haarchri thank you for pointing out to the work ongoing around this issue

Even though it's not a big issue at the moment but we would like to know if you have any estimate when it gets resolved?