Rumbles commented 1 year ago

Terraform Core Version

1.4.4

AWS Provider Version

4.61.0

Affected Resource(s)

Our automated deployments occurred (using a github action runner container) as usual over the weekend, but they failed on the RDS related resources. I was able to confirm this appears to be due to the latest release as I went back to 4.60.0 this morning and was able to apply the same code without any errors.

Expected Behavior

Apply should work without error

Actual Behavior

Plug in panics and crashes

Relevant Error/Panic Output Snippet

[aurora-postgres-v2]: 
╷
│ Error: Request cancelled
│ 
│   with aws_rds_cluster.aurora-cluster,
│   on aurora-postgres.tf line 41, in resource "aws_rds_cluster" "aurora-cluster":
│   41: resource "aws_rds_cluster" "aurora-cluster" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-aws_v4.61.0_x5 plugin:

panic: interface conversion: interface {} is string, not []interface {}

goroutine 491 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/rds.resourceClusterUpdate({0xe8838a0?, 0xc00532f590}, 0xc005ca0280, {0xd384cc0?, 0xc000075800})
    github.com/hashicorp/terraform-provider-aws/internal/service/rds/cluster.go:1282 +0x37b4
github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0x0?, {0xd384cc0?, 0xc000075800?})
    github.com/hashicorp/terraform-provider-aws/internal/provider/intercept.go:94 +0x175
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xe8838a0?, {0xe8838a0?, 0xc005322690?}, 0xd?, {0xd384cc0?, 0xc000075800?})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001979a40, {0xe8838a0, 0xc005322690}, 0xc005a0dee0, 0xc005ca0100, {0xd384cc0, 0xc000075800})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:847 +0x83a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc001a65b48, {0xe8838a0?, 0xc005322570?}, 0xc003b62f50)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:1021 +0xe8d
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0xc0024ce660, 0xc0024ce8d0, {0xc00503ab00, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
    github.com/hashicorp/terraform-plugin-mux@v0.9.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0025c3a40, {0xe8838a0?, 0xc0053059e0?}, 0xc005a362a0)
    github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0xd0899e0?, 0xc0025c3a40}, {0xe8838a0, 0xc0053059e0}, 0xc005a36230, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc004825860, {0xe892fa0, 0xc002392680}, 0xc005a1d200, 0xc005112720, 0x150d5e00, 0x0)
    google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc004825860, {0xe892fa0, 0xc002392680}, 0xc005a1d200, 0x0)
    google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.53.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.61.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

make: *** [../../common/common.mk:294: applyforce] Error 1

While running: 'cd /tmp/github-runner/work_infrastructure-6/infrastructure/infrastructure/resources/rds/live-postgres && make applyforce',
we encountered the following errors: 
ln: failed to create symbolic link '/tmp/vault': File exists
ln: failed to create symbolic link '/tmp/terraform': File exists
╷
│ Error: Request cancelled
│ 
│   with aws_db_instance.live-postgres,
│   on live-postgres.tf line 48, in resource "aws_db_instance" "live-postgres":
│   48: resource "aws_db_instance" "live-postgres" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-aws_v4.61.0_x5 plugin:

panic: interface conversion: interface {} is string, not []interface {}

goroutine 483 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/rds.dbInstancePopulateModify(0xc005e78000, 0xd417cad?)
    github.com/hashicorp/terraform-provider-aws/internal/service/rds/instance.go:2142 +0x2074
github.com/hashicorp/terraform-provider-aws/internal/service/rds.resourceInstanceUpdate({0xe8838a0?, 0xc005e3fc50}, 0xc005e6e000, {0xd384cc0?, 0xc00112e000})
    github.com/hashicorp/terraform-provider-aws/internal/service/rds/instance.go:1965 +0x3593
github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0x0?, {0xd384cc0?, 0xc00112e000?})
    github.com/hashicorp/terraform-provider-aws/internal/provider/intercept.go:94 +0x175
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xe8838a0?, {0xe8838a0?, 0xc0055feff0?}, 0xd?, {0xd384cc0?, 0xc00112e000?})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001a48540, {0xe8838a0, 0xc0055feff0}, 0xc005a1a5b0, 0xc004491e80, {0xd384cc0, 0xc00112e000})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:847 +0x83a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc001b35b48, {0xe8838a0?, 0xc0055feed0?}, 0xc003e1e9b0)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:1021 +0xe8d
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0xc002ebcdb0, 0xc002ebd260, {0xc0050098e0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
    github.com/hashicorp/terraform-plugin-mux@v0.9.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0026ca140, {0xe8838a0?, 0xc0055fe390?}, 0xc0055a69a0)
    github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0xd0899e0?, 0xc0026ca140}, {0xe8838a0, 0xc0055fe390}, 0xc0055a6930, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00513a000, {0xe892fa0, 0xc00016e680}, 0xc005a0c000, 0xc0051404e0, 0x150d5e00, 0x0)
    google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc00513a000, {0xe892fa0, 0xc00016e680}, 0xc005a0c000, 0x0)
    google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.53.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.61.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

make: *** [../../common/common.mk:294: applyforce] Error 1

While running: 'cd /tmp/github-runner/work_infrastructure-6/infrastructure/infrastructure/resources/rds/aurora-postgres-v2 && make applyforce',
we encountered the following errors: 
╷
│ Error: Request cancelled
│ 
│   with aws_rds_cluster.aurora-cluster,
│   on aurora-postgres.tf line 41, in resource "aws_rds_cluster" "aurora-cluster":
│   41: resource "aws_rds_cluster" "aurora-cluster" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-aws_v4.61.0_x5 plugin:

panic: interface conversion: interface {} is string, not []interface {}

goroutine 491 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/rds.resourceClusterUpdate({0xe8838a0?, 0xc00532f590}, 0xc005ca0280, {0xd384cc0?, 0xc000075800})
    github.com/hashicorp/terraform-provider-aws/internal/service/rds/cluster.go:1282 +0x37b4
github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0x0?, {0xd384cc0?, 0xc000075800?})
    github.com/hashicorp/terraform-provider-aws/internal/provider/intercept.go:94 +0x175
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xe8838a0?, {0xe8838a0?, 0xc005322690?}, 0xd?, {0xd384cc0?, 0xc000075800?})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001979a40, {0xe8838a0, 0xc005322690}, 0xc005a0dee0, 0xc005ca0100, {0xd384cc0, 0xc000075800})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:847 +0x83a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc001a65b48, {0xe8838a0?, 0xc005322570?}, 0xc003b62f50)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:1021 +0xe8d
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0xc0024ce660, 0xc0024ce8d0, {0xc00503ab00, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
    github.com/hashicorp/terraform-plugin-mux@v0.9.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0025c3a40, {0xe8838a0?, 0xc0053059e0?}, 0xc005a362a0)
    github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0xd0899e0?, 0xc0025c3a40}, {0xe8838a0, 0xc0053059e0}, 0xc005a36230, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc004825860, {0xe892fa0, 0xc002392680}, 0xc005a1d200, 0xc005112720, 0x150d5e00, 0x0)
    google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc004825860, {0xe892fa0, 0xc002392680}, 0xc005a1d200, 0x0)
    google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.53.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.61.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

There's nothing too crazy in our setup, here is a snippet of one of the resources mentioned in the panic messages:

resource "aws_rds_cluster" "aurora-cluster" {
  cluster_identifier   = "${var.cluster_name}-aurora-serverless-v2-cluster"
  availability_zones   = try(var.aurorav2_availability_zones, slice(data.terraform_remote_state.vpc[0].outputs.rds_availability_zones, 0, 3))
  db_subnet_group_name = aws_db_subnet_group.cluster_db_aurora.id

  engine         = "aurora-postgresql"
  engine_mode    = "provisioned"
  engine_version = var.aurorav2_postgres_version

  database_name             = "main"
  enable_http_endpoint      = false
  vpc_security_group_ids    = [aws_security_group.rds.id]
  master_username           = var.aurora_postgres_db_user
  master_password           = random_password.db_password.result
  storage_encrypted         = true
  copy_tags_to_snapshot     = true
  final_snapshot_identifier = "final-${var.cluster_name}-aurora-cluster-v2"
  skip_final_snapshot       = false
  deletion_protection       = true
  backup_retention_period   = 7

  serverlessv2_scaling_configuration {
    max_capacity = var.aurora_max_capacity
    min_capacity = 1
  }

  apply_immediately = true
}

Steps to Reproduce

I was able to reproduce this locally on my machine this morning and when I moved back to the previous release of the plug in the issue went away so I am fairly sure it is an issue with 4.61.0

Debug Output

Please let me know if you need this, I can reproduce if required

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

ewbankkit commented 1 year ago

@Rumbles Thanks for raising this issue. It has already been noticed in #30383 and #30372. I'm going to close this one as a duplicate so that we can concentrate discussion in the linked issue. Please add any additional comments there.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws

[Bug]: Latest release of plugin crashes during apply of RDS resources #30414

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note