abeluck commented 1 year ago

Terraform Core Version

1.3.8

AWS Provider Version

4.56.0

Affected Resource(s)

aws_gamelift_game_server_group

Expected Behavior

The game lift game server group should be created.

Actual Behavior

It crashed

Relevant Error/Panic Output Snippet

Stack trace from the terraform-provider-aws_v4.56.0_x5 plugin:

panic: runtime error: index out of range [1] with length 1

goroutine 993 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/gamelift.resourceGameServerGroupRead({0xe5376c0, 0xc003ecd860}, 0xc0011a5780, {0xd091680?, 0xc000438400?})
        github.com/hashicorp/terraform-provider-aws/internal/service/gamelift/game_server_group.go:276 +0x1e5e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xe5376c0?, {0xe5376c0?, 0xc003ecd860?}, 0xd?, {0xd091680?, 0xc000438400?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/resource.go:719 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0008a7880, {0xe5376c0, 0xc003ecd860}, 0xc003cbf450, {0xd091680, 0xc000438400})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/resource.go:1015 +0x585
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00012f560, {0xe5376c0?, 0xc003ecd2f0?}, 0xc005f96c80)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/grpc_provider.go:613 +0x4a5
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ReadResource({0xc0032020c0, 0xc003202120, {0xc00322f960, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.9.0/tf5muxserver/mux_server_ReadResource.go:26 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc001fccaa0, {0xe5376c0?, 0xc003ec07e0?}, 0xc005f876e0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:748 +0x4b1
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0xcda6b00?, 0xc001fccaa0}, {0xe5376c0, 0xc003ec07e0}, 0xc0024dee70, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc004523e00, {0xe546640, 0xc003224d00}, 0xc003d845a0, 0xc003e24030, 0x14c30450, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc004523e00, {0xe546640, 0xc003224d00}, 0xc003d845a0, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.53.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.56.0_x5 plugin crashed!

Terraform Configuration Files

resource "aws_gamelift_game_server_group" "this" {
  game_server_group_name = "${module.this.id}"

  instance_definition {
    instance_type = var.instance_type
  }

  instance_definition {
    instance_type = var.instance_type2
  }

  launch_template {
    id = aws_launch_template.this.id
  }
  auto_scaling_policy {
    estimated_instance_warmup = 120
    target_tracking_configuration {
      target_value = 20.0
    }
  }
  vpc_subnets = var.gamelift_subnets
  max_size = 2
  min_size = 1
  role_arn = aws_iam_role.gamelift_service.arn
  tags = module.this.tags
}

Steps to Reproduce

Apply
Observe Crash

Debug Output

2023-03-02T14:08:57.789Z [DEBUG] provider.terraform-provider-aws_v4.56.0_x5:
HTTP Request Sent: http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.3.8
(+https://www.terraform.io) terraform-provider-aws/4.56.0
(+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.208
(go1.19.3; linux; amd64)" tf_provider_addr=registry.terraform.io/hashicorp/aws
tf_req_id=58e2a2bf-443b-355c-3921-cfb4b5755330
http.request.header.authorization="AWS4-HMAC-SHA256
Credential=ASIA************3EN6/20230302/eu-west-1/gamelift/aws4_request,
SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target,
Signature=*****" aws.region=eu-west-1 aws.sdk=aws-sdk-go http.flavor=1.1
http.request_content_length=45 tf_mux_provider=*schema.GRPCProviderServer
tf_resource_type=aws_gamelift_game_server_group
@caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.25/logger.go:90
http.method=POST http.request.body={"GameServerGroupName":"REDACTED-dev-tin"}
http.request.header.x_amz_security_token=*****
http.url=https://gamelift.eu-west-1.amazonaws.com/
net.peer.name=gamelift.eu-west-1.amazonaws.com aws.service=GameLift
http.request.header.content_type=application/x-amz-json-1.1
http.request.header.x_amz_date=20230302T140857Z
http.request.header.x_amz_target=GameLift.DescribeGameServerGroup
tf_rpc=ReadResource @module=aws aws.operation=DescribeGameServerGroup
timestamp=2023-03-02T14:08:57.789Z

2023-03-02T14:08:57.909Z [DEBUG] provider.terraform-provider-aws_v4.56.0_x5:
HTTP Response Received: aws.operation=DescribeGameServerGroup http.duration=119
http.response.header.content_type=application/x-amz-json-1.1
http.response.header.x_amzn_requestid=d3e22663-a392-411d-947e-8c1313b9a2f6
tf_mux_provider=*schema.GRPCProviderServer
tf_provider_addr=registry.terraform.io/hashicorp/aws @module=aws
http.response.header.date="Thu, 02 Mar 2023 14:08:57 GMT" tf_rpc=ReadResource
@caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.25/logger.go:138
aws.sdk=aws-sdk-go
http.response.body="{"GameServerGroup":{"BalancingStrategy":"SPOT_PREFERRED","CreationTime":1.677765400036E9,"GameServerGroupArn":"arn:aws:gamelift:eu-west-1:REDACTED:gameservergroup/REDACTED-dev-tin","GameServerGroupName":"REDACTED-dev-tin","GameServerProtectionPolicy":"NO_PROTECTION","InstanceDefinitions":[{"InstanceType":"m5.large"},{"InstanceType":"c5.large"}],"LastUpdatedTime":1.677765730418E9,"RoleArn":"arn:aws:iam::REDACTED:role/REDACTED-dev-tin-gamelift-service","Status":"ERROR","StatusReason":"The
availability zones of the specified subnets in the launch template and the Auto
Scaling group do not match"}}" http.status_code=200
tf_req_id=58e2a2bf-443b-355c-3921-cfb4b5755330
tf_resource_type=aws_gamelift_game_server_group aws.region=eu-west-1
aws.service=GameLift timestamp=2023-03-02T14:08:57.909Z

2023-03-02T14:08:57.911Z [DEBUG] provider.terraform-provider-aws_v4.56.0_x5: panic: runtime error: index out of range [1] with length 1
2023-03-02T14:08:57.911Z [DEBUG] provider.terraform-provider-aws_v4.56.0_x5: 
...

Full log: log.asc.txt

Panic Output

No response

Important Factoids

No response

References

This is possibly related to https://github.com/hashicorp/terraform-provider-aws/issues/29758

Would you like to implement a fix?

No

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

kurtislamb commented 1 year ago

Just hit this same issue, whats worse is my infra is stuck in a half deployed state and im going to need to make some Open State Surgery just to get Terraform to work as both Apply/Destroy Fail with this error

kurtislamb commented 1 year ago

To add some context;

My issue occurred because in the Launch Template I had provided both a network_interface and a vpc_security_group_ids which resulted in a failure in resource aws_gamelift_game_server_group example which uses the Launch Template

Upon removing the network_Interface my state now shows the resource resource aws_gamelift_game_server_group example but terraform destroy or comment the resource out then terraform apply does not fix it and results in the above mentioned panic. The next logical step to unjam my state was to try removing it directly from the state terraform state rm module.gamelift_fleet_iq.aws_gamelift_game_server_group.example which did allow me to get back to developing.

In the interim I will probably use a local to create the game server group via an api/cli call or potentially create it via a golang application, will update here if I get any success.

kurtislamb commented 1 year ago

As a way to deal with these symptoms I recommend including this block of code in the game_server_group resource.

it does 2 things.

Ensures the Game Server Group is replaced should anything in the Launch Template Change therefore meaning, the Game Server Group does not exist when the Launch Template is changed.
Validates that the Launch Template ID is actually an ID, if the Launch Template fails to create due to an issue, this prevents the Game Server Group from creating.

This combination prevents the scenarios where the Game Server Group Resource causes a panic

Yes I used the term Thermonuclear

  lifecycle {
    replace_triggered_by = [aws_launch_template.gamelift_server_group.id]
    precondition {
      condition     = contains([aws_launch_template.gamelift_server_group.id],"lt-") ? false : true
      error_message = "Launch Template was not created, failing to prevent Game Server Group going thermonuclear"
    }
  }

thomasrost99 commented 5 months ago

As additional context I had run into this error and it was because the version I had provided in the launch template was $latest instead of $Latest Invalid launch template version: either '$Default', '$Latest', or a numeric version are allowed. Just needed to set log level to debug in tf to see the issue unfortunately

abeluck commented 4 months ago

It's been a little over a year and we still have this problem:


Stack trace from the terraform-provider-aws_v5.45.0_x5 plugin:

panic: interface conversion: interface {} is nil, not string

goroutine 1616 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/gamelift.resourceGameServerGroupCreate({0x137d2248?, 0xc002a0cd20}, 0xc003021700, {0x11c26640?, 0xc002b62340})
    github.com/hashicorp/terraform-provider-aws/internal/service/gamelift/game_server_group.go:248 +0xf92
github.com/hashicorp/terraform-provider-aws/internal/provider.New.(*wrappedResource).Create.interceptedHandler[...].func8(0x0?, {0x11c26640?, 0xc002b62340?})
    github.com/hashicorp/terraform-provider-aws/internal/provider/intercept.go:113 +0x283
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0x137d2248?, {0x137d2248?, 0xc002ea21b0?}, 0xd?, {0x11c26640?, 0xc002b62340?})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.33.0/helper/schema/resource.go:773 +0x7a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0016596c0, {0x137d2248, 0xc002ea21b0}, 0xc002b62a90, 0xc003021580, {0x11c26640, 0xc002b62340})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.33.0/helper/schema/resource.go:909 +0xa89
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc002b660d8, {0x137d2248?, 0xc002ea20c0?}, 0xc0002bdea0)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.33.0/helper/schema/grpc_provider.go:1078 +0xdbc
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.(*muxServer).ApplyResourceChange(0x137d2280?, {0x137d2248?, 0xc002689da0?}, 0xc0002bdea0)
    github.com/hashicorp/terraform-plugin-mux@v0.15.0/tf5muxserver/mux_server_ApplyResourceChange.go:36 +0x193
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc000b3d360, {0x137d2248?, 0xc002689560?}, 0xc001fb9dc0)
    github.com/hashicorp/terraform-plugin-go@v0.22.1/tfprotov5/tf5server/server.go:846 +0x3d0
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x119c4220?, 0xc000b3d360}, {0x137d2248, 0xc002689560}, 0xc003021180, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.22.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:518 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001638a00, {0x137d2248, 0xc0026894d0}, {0x13811438, 0xc002c8a1a0}, 0xc002300b40, 0xc002c063f0, 0x1bc77798, 0x0)
    google.golang.org/grpc@v1.62.1/server.go:1386 +0xe23
google.golang.org/grpc.(*Server).handleStream(0xc001638a00, {0x13811438, 0xc002c8a1a0}, 0xc002300b40)
    google.golang.org/grpc@v1.62.1/server.go:1797 +0x100c
google.golang.org/grpc.(*Server).serveStreams.func2.1()
    google.golang.org/grpc@v1.62.1/server.go:1027 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 52
    google.golang.org/grpc@v1.62.1/server.go:1038 +0x135

Error: The terraform-provider-aws_v5.45.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

abeluck commented 4 months ago

This crash isn't just annoying, it can totally corrupt your state file because resources will be created but not recorded in the state, leading to the need for massive state surgery.

(and FWIW @kurtislamb's workaround is applied, but now there is another crash)

hashicorp / terraform-provider-aws

[Bug]: another crash in crash in internal/service/gamelift.resourceGameServerGroupCreate #29759

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note