DynamoDB Stream ARN not available from TF plan/apply when streams is enabled on an existing table.

CalvinRodo commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

AWS Provider: v3.73.0 Terraform Version: 1.0.10

Affected Resource(s)

aws_dynamodb_table

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_dynamodb_table" "vault" {
  name             = "Vault"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "FormID"
  range_key        = "SubmissionID"
  stream_enabled   = true
  stream_view_type = "NEW_IMAGE"

  attribute {
    name = "FormID"
    type = "S"
  }

  attribute {
    name = "SubmissionID"
    type = "S"
  }

  attribute {
    name = "Retrieved"
    type = "N"
  }

  global_secondary_index {
    name            = "retrieved-index"
    hash_key        = "Retrieved"
    projection_type = "ALL"
  }

  server_side_encryption {
    enabled     = true
    kms_key_arn = var.kms_key_dynamodb_arn
  }

  point_in_time_recovery {
    enabled = var.env == "local" ? false : true
  }

}

output "dynamodb_vault_stream_arn" {
  description = "Vault DynamoDB stream ARN"
  value       = aws_dynamodb_table.vault.stream_arn
}

Debug Output

Panic Output

Expected Behavior

Actual Behavior

The ARN Was not there

Steps to Reproduce

terraform apply

References

650

justinretzolk commented 2 years ago

Hey @CalvinRodo 👋 Thank you for taking the time to raise this! So that we have all of the information necessary to look into this, can you supply (redacted as necessary) debug logs as well?

woeps commented 2 years ago

Hi! I created the linked downstream issue (at Pulumi). I'm no expert in terraform nor go, but after some digging, it looks to me like this code is responsible for the difference in expected and actual behavior: https://github.com/hashicorp/terraform-provider-aws/blob/be1e8a565d57d654116c0b6e9a8f231426eb38f1/internal/service/dynamodb/table.go#L374

It seems to me the linked if statement is EITHER restoring the table XOR setting the other properties.

To achieve the expected behaviour - I believe - the table needs to be restored (like in the first if branch), but after a successfull restore a table update needs to be performed with all the other props. I think the additional update is needed because AWS doesn't support all properties at "restoration time".

I feel my theory confirmed, since a second deployment (which yields a table update) will bring the table into the expected state.

I'm afraid I don't feel comfortable to code this in go and send a PR, but I hope this still could be of help.

piekstra commented 2 years ago

I'm running into this as well. We have an existing table that we are enabling the stream, but our aws_iam_policy_document which I'll show below breaks the aws_iam_policy when trying to run tf apply presumably due to the ARN not existing yet. Running the TF apply again works but that's not a great situation.

data aws_iam_policy_document dynamodb_stream_read_example_policy_document {
  statement {
    sid = "ExamplePolicy"
    actions = [
      "dynamodb:DescribeStream",
      "dynamodb:GetRecords",
      "dynamodb:GetShardIterator",
      "dynamodb:ListStreams"
    ]
    resources = [
      module.example_table.stream_arn,
    ]
  }
}

resource "aws_iam_policy" "dynamodb_stream_read_example_policy" {
  name        = "dynamodb-stream-read-example-policy"
  description = "Policy required for reading the example table dynamo db stream"

  policy      = data.aws_iam_policy_document.dynamodb_stream_read_example_policy_document.json
}

The error we see looks like this (redacted):

Error: error creating IAM policy dynamodb-stream-read-example-policy: MalformedPolicyDocument: Resource  must be in ARN format or "*".
│   status code: 400, request id: baa-baa-faaa-1111-234657abc
│

github-actions[bot] commented 3 months ago

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

github-actions[bot] commented 1 month ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws