search

hashicorp / terraform-provider-helm

Terraform Helm provider
https://www.terraform.io/docs/providers/helm/
Mozilla Public License 2.0
1.01k stars 371 forks source link

Provider generates invalid values with Experimental manifest stores feature turned on #805

Open jw-maynard opened 2 years ago

jw-maynard commented 2 years ago

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: v1.0.11
Provider version: v2.4.1
Kubernetes version: v1.20

Affected Resource(s)

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
locals {
  namespace = "kube-system"
}

resource "helm_release" "aws-load-balancer-controller" {
  timeout          = 600
  name             = "aws-load-balancer-controller"
  repository       = "https://aws.github.io/eks-charts"
  chart            = "aws-load-balancer-controller"
  version          = "1.2.6"
  namespace        = local.namespace
  create_namespace = true
  values = [templatefile("${path.module}/templates/values.yml.tpl", {
    cluster_name         = "kubernetes01"
    region               = "us-east-1"
    vpc_id               = "********"
    service_account_role = "********"
  })]
}

# Values YAML Template

clusterName: ${cluster_name}
serviceAccount:
    annotations:
        eks.amazonaws.com/role-arn: "${service_account_role}"
region: ${region}
vpcId: ${vpc_id}

Debug Output

NOTE: In addition to Terraform debugging, please set HELM_DEBUG=1 to enable debugging info from helm.

Panic Output

Steps to Reproduce

  1. terraform plan
  2. terraform apply

Expected Behavior

Currently in our config there are no changes that should need to be applied so the plan should show no changes

Actual Behavior

Many changes are shown and when applied an error occurs:

Error: Provider produced inconsistent final plan
When expanding the plan for module.kubernetes_cluster.module.aws-load-balancer-controller["service"].helm_release.aws-load-balancer-controller to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/helm" produced an invalid new value for .manifest: was cty.StringVal(...). This is a bug in the provider, which should be reported in the provider's own issue tracker.

The string value (with some data redacted) is below:

"{\"clusterrole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRole\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-role\"},\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"targetgroupbindings\"],\"verbs\":[\"create\",\"delete\",\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"ingressclassparams\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"events\"],\"verbs\":[\"create\",\"patch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"pods\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"networking.k8s.io\"],\"resources\":[\"ingressclasses\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"services\",\"ingresses\"],\"verbs\":[\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"nodes\",\"secrets\",\"namespaces\",\"endpoints\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\",\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"targetgroupbindings/status\",\"pods/status\",\"services/status\",\"ingresses/status\"],\"verbs\":[\"update\",\"patch\"]}]},\"clusterrolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-rolebinding\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"ClusterRole\",\"name\":\"aws-load-balancer-controller-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/deployment.apps/apps/v1/aws-load-balancer-controller\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"template\":{\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"8080\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"In\",\"values\":[\"aws-load-balancer-controller\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"containers\":[{\"args\":[\"--cluster-name=kubernetes01\",\"--ingress-class=alb\",\"--aws-region=us-east-1\",\"--aws-vpc-id=vpc-REDACTED\"],\"command\":[\"/controller\"],\"image\":\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.2.3\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":2,\"httpGet\":{\"path\":\"/healthz\",\"port\":61779,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":30,\"timeoutSeconds\":10},\"name\":\"aws-load-balancer-controller\",\"ports\":[{\"containerPort\":9443,\"name\":\"webhook-server\",\"protocol\":\"TCP\"},{\"containerPort\":8080,\"name\":\"metrics-server\",\"protocol\":\"TCP\"}],\"resources\":{},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true},\"volumeMounts\":[{\"mountPath\":\"/tmp/k8s-webhook-server/serving-certs\",\"name\":\"cert\",\"readOnly\":true}]}],\"priorityClassName\":\"system-cluster-critical\",\"securityContext\":{\"fsGroup\":65534},\"serviceAccountName\":\"aws-load-balancer-controller\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"cert\",\"secret\":{\"defaultMode\":420,\"secretName\":\"aws-load-balancer-tls\"}}]}}}},\"kube-system/role.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"Role\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-role\",\"namespace\":\"kube-system\"},\"rules\":[{\"apiGroups\":[\"\"],\"resources\":[\"configmaps\"],\"verbs\":[\"create\"]},{\"apiGroups\":[\"\"],\"resourceNames\":[\"aws-load-balancer-controller-leader\"],\"resources\":[\"configmaps\"],\"verbs\":[\"get\",\"patch\",\"update\"]}]},\"kube-system/rolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"RoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-rolebinding\",\"namespace\":\"kube-system\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"Role\",\"name\":\"aws-load-balancer-controller-leader-election-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/secret/v1/aws-load-balancer-tls\":{\"kind\":\"Secret\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"aws-load-balancer-tls\",\"namespace\":\"kube-system\",\"creationTimestamp\":null,\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"}},\"data\":{\"ca.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSBiZDg2NWZlOGM4Nzg0MDdmKQ==\",\"tls.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSBmZmNhN2NlNDk1ZjBlYzdlKQ==\",\"tls.key\":\"KHNlbnNpdGl2ZSB2YWx1ZSAxMTNjNTg2NjIxMjk2Zjk5KQ==\"},\"type\":\"kubernetes.io/tls\"},\"kube-system/service/v1/aws-load-balancer-webhook-service\":{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":\"webhook-server\"}],\"selector\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}}},\"kube-system/serviceaccount/v1/aws-load-balancer-controller\":{\"apiVersion\":\"v1\",\"automountServiceAccountToken\":true,\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::REDACTED:role/REDACTED\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}},\"mutatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"MutatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-v1-pod\"}},\"failurePolicy\":\"Fail\",\"name\":\"mpod.elbv2.k8s.aws\",\"namespaceSelector\":{\"matchExpressions\":[{\"key\":\"elbv2.k8s.aws/pod-readiness-gate-inject\",\"operator\":\"In\",\"values\":[\"enabled\"]}]},\"objectSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"NotIn\",\"values\":[\"aws-load-balancer-controller\"]}]},\"rules\":[{\"apiGroups\":[\"\"],\"apiVersions\":[\"v1\"],\"operations\":[\"CREATE\"],\"resources\":[\"pods\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"mtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"}]},\"validatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"ValidatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"vtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-networking-v1beta1-ingress\"}},\"failurePolicy\":\"Fail\",\"matchPolicy\":\"Equivalent\",\"name\":\"vingress.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"networking.k8s.io\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"ingresses\"]}],\"sideEffects\":\"None\"}]}}"), but now cty.StringVal("{\"clusterrole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRole\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-role\"},\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"targetgroupbindings\"],\"verbs\":[\"create\",\"delete\",\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"ingressclassparams\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"events\"],\"verbs\":[\"create\",\"patch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"pods\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"networking.k8s.io\"],\"resources\":[\"ingressclasses\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"services\",\"ingresses\"],\"verbs\":[\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"nodes\",\"secrets\",\"namespaces\",\"endpoints\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\",\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"targetgroupbindings/status\",\"pods/status\",\"services/status\",\"ingresses/status\"],\"verbs\":[\"update\",\"patch\"]}]},\"clusterrolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-rolebinding\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"ClusterRole\",\"name\":\"aws-load-balancer-controller-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/deployment.apps/apps/v1/aws-load-balancer-controller\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"template\":{\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"8080\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"In\",\"values\":[\"aws-load-balancer-controller\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"containers\":[{\"args\":[\"--cluster-name=kubernetes01\",\"--ingress-class=alb\",\"--aws-region=us-east-1\",\"--aws-vpc-id=vpc-REDACTED\"],\"command\":[\"/controller\"],\"image\":\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.2.3\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":2,\"httpGet\":{\"path\":\"/healthz\",\"port\":61779,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":30,\"timeoutSeconds\":10},\"name\":\"aws-load-balancer-controller\",\"ports\":[{\"containerPort\":9443,\"name\":\"webhook-server\",\"protocol\":\"TCP\"},{\"containerPort\":8080,\"name\":\"metrics-server\",\"protocol\":\"TCP\"}],\"resources\":{},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true},\"volumeMounts\":[{\"mountPath\":\"/tmp/k8s-webhook-server/serving-certs\",\"name\":\"cert\",\"readOnly\":true}]}],\"priorityClassName\":\"system-cluster-critical\",\"securityContext\":{\"fsGroup\":65534},\"serviceAccountName\":\"aws-load-balancer-controller\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"cert\",\"secret\":{\"defaultMode\":420,\"secretName\":\"aws-load-balancer-tls\"}}]}}}},\"kube-system/role.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"Role\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-role\",\"namespace\":\"kube-system\"},\"rules\":[{\"apiGroups\":[\"\"],\"resources\":[\"configmaps\"],\"verbs\":[\"create\"]},{\"apiGroups\":[\"\"],\"resourceNames\":[\"aws-load-balancer-controller-leader\"],\"resources\":[\"configmaps\"],\"verbs\":[\"get\",\"patch\",\"update\"]}]},\"kube-system/rolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"RoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-rolebinding\",\"namespace\":\"kube-system\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"Role\",\"name\":\"aws-load-balancer-controller-leader-election-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/secret/v1/aws-load-balancer-tls\":{\"kind\":\"Secret\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"aws-load-balancer-tls\",\"namespace\":\"kube-system\",\"creationTimestamp\":null,\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"}},\"data\":{\"ca.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSA5YzljYjg3YjliMDliOTYzKQ==\",\"tls.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSAzNzhlYTFjODQ3OTRkMWRlKQ==\",\"tls.key\":\"KHNlbnNpdGl2ZSB2YWx1ZSA5Njk1MDMzNjlhYzE4YTJhKQ==\"},\"type\":\"kubernetes.io/tls\"},\"kube-system/service/v1/aws-load-balancer-webhook-service\":{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":\"webhook-server\"}],\"selector\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}}},\"kube-system/serviceaccount/v1/aws-load-balancer-controller\":{\"apiVersion\":\"v1\",\"automountServiceAccountToken\":true,\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::REDACTED:role/REDACTED\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}},\"mutatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"MutatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-v1-pod\"}},\"failurePolicy\":\"Fail\",\"name\":\"mpod.elbv2.k8s.aws\",\"namespaceSelector\":{\"matchExpressions\":[{\"key\":\"elbv2.k8s.aws/pod-readiness-gate-inject\",\"operator\":\"In\",\"values\":[\"enabled\"]}]},\"objectSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"NotIn\",\"values\":[\"aws-load-balancer-controller\"]}]},\"rules\":[{\"apiGroups\":[\"\"],\"apiVersions\":[\"v1\"],\"operations\":[\"CREATE\"],\"resources\":[\"pods\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"mtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"}]},\"validatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"ValidatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"vtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-networking-v1beta1-ingress\"}},\"failurePolicy\":\"Fail\",\"matchPolicy\":\"Equivalent\",\"name\":\"vingress.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"networking.k8s.io\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"ingresses\"]}],\"sideEffects\":\"None\"}]}}"

Important Factoids

References

Community Note

nitrocode commented 2 years ago

I get the same issue with the datadog-agent helm chart.

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.datadog_agent.helm_release.this[0] to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/helm" produced an invalid new value for
│ .manifest: was

...

│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
jw-maynard commented 2 years ago

I did some digging into my issue today and was able to diff the Plan manifests against the Apply manifests to find out what's inconsistent. In my case the aws-load-balancer-controller chart will generate a cert bundle for it's webhook when you run the chart. It looks like the helm is being run during the plan and then again during apply and since it's run twice these auto generated certs are being created twice and therefore have different values which causes the inconstancy.

jw-maynard commented 2 years ago

@BBBmau @jrhouston I don't know if this is feasible, but maybe the plan step could store a complete copy of the Helm values in the plan and then during apply the provider would feed a full set of values into Helm. Also not sure if this strategy would have undesirable side effects in helm.

github-actions[bot] commented 1 year ago

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

brandon-gradient commented 1 year ago

This does seem to be connected to the experimental manifest feature when using helm charts that have some random generation for output values. In addition to the same aws-load-balancer-controller chart generating differing cert bundles, I also ran into this with the grafana/grafana chart and using the basic auth. It would generate a random password on each run of the chart which would differ when the command is run again. It seems like the manifest feature's output would need to be passed on to helm wholesale similar to a terraform apply "plan.output" command

TheKangaroo commented 10 months ago

I ran into this with another chart using the randNumeric function, but I replaced it with the now function and got the same error. At this point I'm pretty sure it's impossible to combine the experiments { manifest = true } feature and any helm function that produces different outputs for two subsequent plans. This is, in my opinion, a major drawback of the provider at the moment. It should at least be mentioned in the documentation of this experimental feature. On the other hand, I don't know how anyone would use the helm provider without the experimental feature enabled, since you can never tell from the plan what will actually change in your cluster. I think the information about this issue is a bit scattered over a lot of issues and mixed with other (already solved) issues with similar symptoms. I hope this issue gets some attention from the maintainers, as it is a showstopper for many use cases.