search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
41.75k stars 9.42k forks source link

panic: no expansion has been registered for module.s3_bucket_example_second.data.aws_s3_bucket.existing_logging_bucket #35427

Closed marc-poljak closed 1 week ago

marc-poljak commented 2 weeks ago

Terraform Version

Terraform v1.9.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.57.0

Terraform Configuration Files


locals {
  logging_bucket_name = format("%s-centralized-logs", var.name_prefix)
}

data "aws_s3_bucket" "existing_logging_bucket" {
  bucket = local.logging_bucket_name

  # This will cause the data source to return "not found" if the bucket doesn't exist,
  # allowing us to create it
  count = try(data.aws_s3_bucket.existing_logging_bucket[0].id, null) == null ? 0 : 1
}

resource "aws_s3_bucket" "logging_bucket" {
  # Only create if it doesn't already exist
  count  = try(data.aws_s3_bucket.existing_logging_bucket[0].id, null) == null ? 1 : 0
  bucket = local.logging_bucket_name
  tags   = merge(var.tags, { Name = "Centralized Logging Bucket" })
}

resource "aws_s3_bucket" "bucket" {
  bucket = format("%s-%s-%s", var.name_prefix, var.bucket_name, var.account_id)
  tags   = var.tags
}

resource "aws_s3_bucket_server_side_encryption_configuration" "bucket_encryption" {
  bucket = aws_s3_bucket.bucket.id

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = var.kms_key_id
      sse_algorithm     = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "logging_bucket_encryption" {
  # Use the created bucket if it exists, otherwise use the data source
  bucket = try(aws_s3_bucket.logging_bucket[0].id, data.aws_s3_bucket.existing_logging_bucket[0].id)

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = var.kms_key_id
      sse_algorithm     = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_versioning" "bucket_versioning" {
  count  = var.enable_versioning ? 1 : 0
  bucket = aws_s3_bucket.bucket.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "bucket_public_access_block" {
  bucket                  = aws_s3_bucket.bucket.id
  block_public_acls       = var.block_public_access
  block_public_policy     = var.block_public_access
  ignore_public_acls      = var.block_public_access
  restrict_public_buckets = var.block_public_access
}

resource "aws_s3_bucket_public_access_block" "logging_bucket_public_access_block" {
  # Use the created bucket if it exists, otherwise use the data source
  bucket                  = try(aws_s3_bucket.logging_bucket[0].id, data.aws_s3_bucket.existing_logging_bucket[0].id)
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_logging" "bucket_logging" {
  bucket = aws_s3_bucket.bucket.id

  # Use the created bucket if it exists, otherwise use the data source
  target_bucket = try(aws_s3_bucket.logging_bucket[0].id, data.aws_s3_bucket.existing_logging_bucket[0].id)
  target_prefix = "${aws_s3_bucket.bucket.id}/"
}

resource "aws_s3_bucket_policy" "bucket_policy" {
  count  = var.bucket_policy != null ? 1 : 0
  bucket = aws_s3_bucket.bucket.id
  policy = var.bucket_policy
}

resource "aws_s3_bucket_policy" "logging_bucket_policy" {
  # Use the created bucket if it exists, otherwise use the data source
  bucket = try(aws_s3_bucket.logging_bucket[0].id, data.aws_s3_bucket.existing_logging_bucket[0].id)

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Sid       = "S3ServerAccessLogsPolicy"
        Effect    = "Allow"
        Principal = { Service = "logging.s3.amazonaws.com" }
        Action    = ["s3:PutObject"]
        Resource  = "${try(aws_s3_bucket.logging_bucket[0].arn, data.aws_s3_bucket.existing_logging_bucket[0].arn)}/*"
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = var.account_id
          }
        }
      },
      {
        Sid       = "AllowSSLRequestsOnly"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:*"
        Resource = [
          try(aws_s3_bucket.logging_bucket[0].arn, data.aws_s3_bucket.existing_logging_bucket[0].arn),
          "${try(aws_s3_bucket.logging_bucket[0].arn, data.aws_s3_bucket.existing_logging_bucket[0].arn)}/*"
        ]
        Condition = {
          Bool = {
            "aws:SecureTransport" = "false"
          }
        }
      }
    ]
  })

Debug Output

the log trace file is empty!

Expected Behavior

  1. The module will attempt to find an existing logging bucket with the name -centralized-logs.
  2. If the logging bucket doesn't exist, it will be created.
  3. Subsequent uses of the module will use the existing logging bucket instead of creating a new one.
  4. Each main bucket will have its logs stored in a folder named after its bucket ID within the centralized logging bucket.

Actual Behavior

terraform crash after terraform plan

Steps to Reproduce

terraform init terraform plan

Additional Context

❯ terraform plan                                                                                                                                                                                                                                                              ─╯

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

panic: no expansion has been registered for module.s3_bucket_example_second.data.aws_s3_bucket.existing_logging_bucket
goroutine 500 [running]:
runtime/debug.Stack()
        runtime/debug/stack.go:24 +0x64
github.com/hashicorp/terraform/internal/logging.PanicHandler()
        github.com/hashicorp/terraform/internal/logging/panic.go:84 +0x198
panic({0x1031426e0?, 0x140041d7020?})
        runtime/panic.go:770 +0x124
github.com/hashicorp/terraform/internal/instances.(*Expander).ResourceInstanceKeys(0x14000a70690?, {{}, {0x14001e21640, 0x1, 0x1}, {{}, 0x44, {0x140005d4cd0, 0xd}, {0x1400035d8d8, ...}}})
        github.com/hashicorp/terraform/internal/instances/expander.go:448 +0x1e8
github.com/hashicorp/terraform/internal/terraform.(*evaluationStateData).GetResource(0x14001cf2d80, {{}, 0x44, {0x140005d4cd0, 0xd}, {0x1400035d8d8, 0x17}}, {{0x14000901a20, 0x19}, {0xe, ...}, ...})
        github.com/hashicorp/terraform/internal/terraform/evaluate.go:522 +0x1e0
github.com/hashicorp/terraform/internal/lang.(*Scope).evalContext(0x14001cf2e10, {0x140024d5278, 0x1, 0x1}, {0x0, 0x0})
        github.com/hashicorp/terraform/internal/lang/eval.go:372 +0x13d0
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalContext(...)
        github.com/hashicorp/terraform/internal/lang/eval.go:246
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalExpr(0x14001cf2e10, {0x103840210, 0x140003eb420}, {{0x10383f8a8?, 0x1400000f998?}})
        github.com/hashicorp/terraform/internal/lang/eval.go:171 +0x8c
github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateExpr(0x0?, {0x103840210, 0x140003eb420}, {{0x10383f8a8?, 0x1400000f998?}}, {0x0?, 0x0?})
        github.com/hashicorp/terraform/internal/terraform/eval_context_builtin.go:325 +0x84
github.com/hashicorp/terraform/internal/terraform.evaluateCountExpressionValue({0x103840210, 0x140003eb420}, {0x103860748?, 0x140024c8b40?})
        github.com/hashicorp/terraform/internal/terraform/eval_count.go:71 +0x74
github.com/hashicorp/terraform/internal/terraform.evaluateCountExpression({0x103840210, 0x140003eb420}, {0x103860748?, 0x140024c8b40?}, 0x0)
        github.com/hashicorp/terraform/internal/terraform/eval_count.go:31 +0x38
github.com/hashicorp/terraform/internal/terraform.(*NodeAbstractResource).writeResourceState(0x140016d3500, {0x103860748, 0x140024c8b40}, {{}, {0x14001e21640, 0x1, 0x1}, {{}, 0x44, {0x140005d4c80, ...}, ...}})
        github.com/hashicorp/terraform/internal/terraform/node_resource_abstract.go:419 +0x148
github.com/hashicorp/terraform/internal/terraform.(*nodeExpandPlannableResource).expandResourceInstances(0x14001ff1810, {0x103860748, 0x140015694a0}, {{}, {0x14001e21640, 0x1, 0x1}, {{}, 0x44, {0x140005d4c80, ...}, ...}}, ...)
        github.com/hashicorp/terraform/internal/terraform/node_resource_plan.go:344 +0xfc
github.com/hashicorp/terraform/internal/terraform.(*nodeExpandPlannableResource).dynamicExpand(0x14001ff1810, {0x103860748, 0x140015694a0}, {0x14001e46660, 0x1, 0x1?}, {0x1?})
        github.com/hashicorp/terraform/internal/terraform/node_resource_plan.go:296 +0x390
github.com/hashicorp/terraform/internal/terraform.(*nodeExpandPlannableResource).DynamicExpand(0x14001ff1810, {0x103860748, 0x140015694a0})
        github.com/hashicorp/terraform/internal/terraform/node_resource_plan.go:134 +0x360
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x10377d4e0, 0x14001ff1810})
        github.com/hashicorp/terraform/internal/terraform/graph.go:145 +0x7dc
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0x14000b98960, {0x10377d4e0, 0x14001ff1810}, 0x1400292c440)
        github.com/hashicorp/terraform/internal/dag/walk.go:384 +0x2a8
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update in goroutine 266
        github.com/hashicorp/terraform/internal/dag/walk.go:307 +0xc30

References

no

liamcervante commented 1 week ago

Hi @marc-poljak, thanks for filing this. Unfortunately, what you are attempting to do here isn't possible.

The count attribute you have in your data source is always going to return 0, even if the data source does actually exist. You can't self-reference attributes in this way within Terraform as it has to load the data source before the attributes are available but it requires all the attributes to be available before it can load the data source.

However, Terraform definitely shouldn't be crashing in this case but reporting an error that informs you this kind of referencing isn't possible. I had thought this was fixed in https://github.com/hashicorp/terraform/issues/35038, but it seems an edge case must have been missed so I'll investigate to see exactly why the check we added previously isn't catching this instance of self-reference.

marc-poljak commented 1 week ago

Hi @liamcervante thanks for the answer and the explanation. I hope I could help a little bit with filing the crash report.