MalformedPolicyDocument: Policy statement must contain resources.

[IvayloIvanovMM]

Description

I am running the tests/complete example that is provided in https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/tree/main/tests/complete . There are a few deprecated values like:

with module.eks.aws_eks_addon.this["coredns"],
│   on .terraform\modules\eks\main.tf line 392, in resource "aws_eks_addon" "this":
│  392:   resolve_conflicts        = try(each.value.resolve_conflicts, "OVERWRITE")
│
│ The "resolve_conflicts" attribute can't be set to "PRESERVE" on initial resource creation. Use "resolve_conflicts_on_create" and/or "resolve_conflicts_on_update" instead

The actual error that I get and I am scratching my head is :

│ Error: creating IAM Policy (aws-node-termination-handler-20230529162529323600000001): MalformedPolicyDocument: Policy statement must contain resources.
│       status code: 400, request id: 64dc2efe-47c7-4f2f-b51c-2b01a70392c1
│
│   with module.eks_blueprints_addons.module.aws_node_termination_handler.aws_iam_policy.this[0],
│   on .terraform\modules\eks_blueprints_addons.aws_node_termination_handler\main.tf line 237, in resource "aws_iam_policy" "this":
│  237: resource "aws_iam_policy" "this" {
│

• Module version [0.1.0 (latest)]:

• Terraform version: ">= 1.0"

• Provider version(s): aws ">=2.17" kubernetes ">=2.17" helm ">= 2.8"

Steps to reproduce the behavior: terraform init terraform plan terraform apply

Expected behaviour

Deploy architecture without errors

Actual behaviour

Get an error mentioned above

[bryantbiggs]

please provide a reproduction that demonstrates the error

[IvayloIvanovMM]

Sorry, I misclicked and closed the issue by mistake. In regards to the reproduction - I've literally cloned the repo and then followed the steps you've mentioned in the README.md file here https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/tree/main/tests/complete .

[bryantbiggs]

are you saying the complete test case does not work as its written? I just validated this case at the end of last week and I did not encounter any of the errors you have reported here

[IvayloIvanovMM]

are you saying the complete test case does not work as its written? I just validated this case at the end of last week and I did not encounter any of the errors you have reported here

[IvayloIvanovMM]

Yes, I can confirm this does happen on the complete test case. Just happened today again.

After cloning the repo, changing directory to tests/complete , terraform init, terraform plan, terraform apply I create most of the resources but still some of the addons don't work. They take about 20 minutes to create. The ones that I can see are: module.eks.aws_eks_addon.this["aws-guardduty-agent"]: Still creating... [18m1s elapsed] module.eks.aws_eks_addon.this["aws-ebs-csi-driver"]: Still creating... [15m21s elapsed] module.eks.aws_eks_addon.this["vpc-cni"]: Still creating... [15m11s elapsed]

At the end of the 20-ish minutes I get:

╷ │ Warning: Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration │ │ with module.eks.aws_eks_addon.this["aws-ebs-csi-driver"], │ on .terraform\modules\eks\main.tf line 382, in resource "aws_eks_addon" "this": │ 382: resource "aws_eks_addon" "this" { │ │ (and 2 more similar warnings elsewhere) ╵ ╷ │ Warning: Argument is deprecated │ │ with module.eks.aws_eks_addon.this["aws-ebs-csi-driver"], │ on .terraform\modules\eks\main.tf line 392, in resource "aws_eks_addon" "this": │ 392: resolve_conflicts = try(each.value.resolve_conflicts, "OVERWRITE") │ │ The "resolve_conflicts" attribute can't be set to "PRESERVE" on initial resource creation. Use "resolve_conflicts_on_create" and/or "resolve_conflicts_on_update" instead │ │ (and 5 more similar warnings elsewhere) ╵ ╷ │ Warning: Helm release "argo-cd" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.argocd.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.argocd\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "aws-cloudwatch-metrics" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.aws_cloudwatch_metrics.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_cloudwatch_metrics\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "aws-efs-csi-driver" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.aws_efs_csi_driver.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_efs_csi_driver\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "aws-for-fluent-bit" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.aws_for_fluentbit.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_for_fluentbit\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "aws-fsx-csi-driver" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.aws_fsx_csi_driver.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_fsx_csi_driver\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "aws-load-balancer-controller" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.aws_load_balancer_controller.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_load_balancer_controller\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "aws-privateca-issuer" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.aws_privateca_issuer.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_privateca_issuer\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "cert-manager" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.cert_manager.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.cert_manager\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "cluster-autoscaler" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.cluster_autoscaler.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.cluster_autoscaler\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "external-dns" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.external_dns.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.external_dns\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "external-secrets" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.external_secrets.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.external_secrets\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "gatekeeper" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.gatekeeper.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.gatekeeper\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "ingress-nginx" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.ingress_nginx.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.ingress_nginx\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "kube-prometheus-stack" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.kube_prometheus_stack.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.kube_prometheus_stack\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "metrics-server" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.metrics_server.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.metrics_server\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "secrets-store-csi-driver" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.secrets_store_csi_driver.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.secrets_store_csi_driver\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "velero" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.velero.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.velero\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Warning: Helm release "vpa" was created but has a failed status. Use thehelmcommand to investigate the error, correct it, then run Terraform again. │ │ with module.eks_blueprints_addons.module.vpa.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.vpa\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: waiting for EKS Add-On (complete:aws-ebs-csi-driver) create: timeout while waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s) │ │ with module.eks.aws_eks_addon.this["aws-ebs-csi-driver"], │ on .terraform\modules\eks\main.tf line 382, in resource "aws_eks_addon" "this": │ 382: resource "aws_eks_addon" "this" { │ ╵ ╷ │ Error: waiting for EKS Add-On (complete:aws-guardduty-agent) create: timeout while waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s) │ │ with module.eks.aws_eks_addon.this["aws-guardduty-agent"], │ on .terraform\modules\eks\main.tf line 382, in resource "aws_eks_addon" "this": │ 382: resource "aws_eks_addon" "this" { │ ╵ ╷ │ Error: waiting for EKS Add-On (complete:vpc-cni) create: timeout while waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s) │ │ with module.eks.aws_eks_addon.this["vpc-cni"], │ on .terraform\modules\eks\main.tf line 382, in resource "aws_eks_addon" "this": │ 382: resource "aws_eks_addon" "this" { │ ╵ ╷ │ Error: client rate limiter Wait returned an error: context deadline exceeded │ │ with module.eks_blueprints_addons.module.argocd.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.argocd\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.aws_cloudwatch_metrics.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_cloudwatch_metrics\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.aws_efs_csi_driver.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_efs_csi_driver\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.aws_for_fluentbit.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_for_fluentbit\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.aws_fsx_csi_driver.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_fsx_csi_driver\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.aws_load_balancer_controller.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_load_balancer_controller\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.aws_privateca_issuer.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.aws_privateca_issuer\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.cert_manager.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.cert_manager\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.cluster_autoscaler.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.cluster_autoscaler\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.external_dns.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.external_dns\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.external_secrets.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.external_secrets\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: failed pre-install: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.gatekeeper.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.gatekeeper\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: failed pre-install: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.ingress_nginx.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.ingress_nginx\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: failed pre-install: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.kube_prometheus_stack.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.kube_prometheus_stack\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.metrics_server.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.metrics_server\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: failed pre-install: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.secrets_store_csi_driver.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.secrets_store_csi_driver\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { ╵ ╷ │ Error: failed pre-install: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.velero.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.velero\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" { │ ╵ ╷ │ Error: failed pre-install: timed out waiting for the condition │ │ with module.eks_blueprints_addons.module.vpa.helm_release.this[0], │ on .terraform\modules\eks_blueprints_addons.vpa\main.tf line 9, in resource "helm_release" "this": │ 9: resource "helm_release" "this" {

[bryantbiggs]

To be clear:

1. This is a warning based on the v5.0 of the AWS provider that Hashicorp released last week │ The "resolve_conflicts" attribute can't be set to "PRESERVE" on initial resource creation. Use "resolve_conflicts_on_create" and/or "resolve_conflicts_on_update" instead

2. I am not seeing the error you provided in your original message in this output

   │ Error: creating IAM Policy (aws-node-termination-handler-20230529162529323600000001): MalformedPolicyDocument: Policy statement must contain resources.
   │       status code: 400, request id: 64dc2efe-47c7-4f2f-b51c-2b01a70392c1
   │
   │   with module.eks_blueprints_addons.module.aws_node_termination_handler.aws_iam_policy.this[0],
   │   on .terraform\modules\eks_blueprints_addons.aws_node_termination_handler\main.tf line 237, in resource "aws_iam_policy" "this":
   │  237: resource "aws_iam_policy" "this" {
   │

I suspect what you are encountering is an issue with Gatekeeper but that is separate from the issue you posted

[bryantbiggs]

we have made some recent changes (you can see these in the v0.2.0 release notes) and I can confirm that the complete test case is working as intended. I'll close this for now but please feel free to provide additional details and a reproduction if you'd like us to look into your issue further

[armujahid]

I am getting this error with enable_aws_node_termination_handler = true with latest eks_blueprints_addons version 1.12.0

╷
│ Error: creating IAM Policy (aws-node-termination-handler-20231203064834013500000007): MalformedPolicyDocument: Policy statement must contain resources.
│   status code: 400, request id: d1f13162-0b81-4c6e-a914-1dff5c978073
│ 
│   with module.eks_blueprints_addons.module.aws_node_termination_handler.aws_iam_policy.this[0],
│   on .terraform/modules/eks_blueprints_addons.aws_node_termination_handler/main.tf line 242, in resource "aws_iam_policy" "this":
│  242: resource "aws_iam_policy" "this" {