search

GoogleCloudPlatform / terraformer

CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code
Apache License 2.0
12.45k stars 1.63k forks source link

terraformer import aws: panic: interface conversion: interface {} is nil, not []interface {} #1707

Closed jakern closed 11 months ago

jakern commented 1 year ago

Every time the @github-actions bot complains about this issue being stale I will link to another issue that the bot has closed for inactivity

edit: I've narrowed down to at least ec2 instances and likely ones without a description... I don't have permissions to edit them but will keep pushing where I can.

jakern commented 1 year ago

After a colleague removed an ec2 resource I was able to get further. I am still looking to find what other resources are also causing this issue.

shblue21 commented 1 year ago

Can you show me specific error logs for the bug?

jakern commented 1 year ago

Is this enough information to be helpful?

   http.response.header.cache_control="no-cache, no-store" http.response.header.content_type=text/xml;charset=UTF-8 http.response.header.date="Thu, 20 Jul 2023 20:23:49 GMT" http.response.header.strict_transport_security="max-age=31536000; includeSubDomains" @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.25/logger.go:138 aws.service=EC2 tf_mux_provider="*schema.GRPCProviderServer" timestamp=2023-07-20T16:22:45.913-0400
2023-07-20T16:22:45.914-0400 [DEBUG] plugin.terraform-provider-aws_v4.59.0_x5.exe: HTTP Request Sent: http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" net.peer.name=ec2.us-gov-west-1.amazonaws.com tf_mux_provider="*schema.GRPCProviderServer" tf_resource_type=aws_instance aws.region=us-gov-west-1 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=AKIA************G4RN/20230720/us-gov-west-1/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=*****" http.request_content_length=95 tf_req_id=c058833c-f38b-c9c9-3b6c-c881f7745cf8 http.flavor=1.1 http.method=POST http.request.header.x_amz_date=20230720T202245Z tf_rpc=ReadResource @module=aws aws.operation=DescribeInstanceCreditSpecifications aws.service=EC2 aws.sdk=aws-sdk-go http.request.body="Action=DescribeInstanceCreditSpecifications&InstanceId.1=i-09b597aae49c9c6c5&Version=2016-11-15" http.url=https://ec2.us-gov-west-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/0.12.31 (+https://www.terraform.io) terraform-provider-aws/4.59.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.221 (go1.19.6; windows; amd64)" tf_provider_addr=registry.terraform.io/hashicorp/aws @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.25/logger.go:90 timestamp=2023-07-20T16:22:45.914-0400
2023-07-20T16:22:46.204-0400 [DEBUG] plugin.terraform-provider-aws_v4.59.0_x5.exe: HTTP Response Received: http.duration=289
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <DescribeInstanceCreditSpecificationsResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
  |     <requestId>0161a316-95d5-4dd5-99c4-4fe8130bb710</requestId>
  |     <instanceCreditSpecificationSet>
  |         <item>
  |             <cpuCredits>unlimited</cpuCredits>
  |             <instanceId>i-09b597aae49c9c6c5</instanceId>
  |         </item>
  |     </instanceCreditSpecificationSet>
  | </DescribeInstanceCreditSpecificationsResponse>
   http.response.header.date="Thu, 20 Jul 2023 20:23:49 GMT" tf_rpc=ReadResource @module=aws aws.service=EC2 http.response.header.vary=accept-encoding http.response.header.x_amzn_requestid=0161a316-95d5-4dd5-99c4-4fe8130bb710 aws.sdk=aws-sdk-go http.response.header.content_type=text/xml;charset=UTF-8 http.response.header.server=AmazonEC2 http.status_code=200 tf_mux_provider="*schema.GRPCProviderServer" tf_resource_type=aws_instance aws.region=us-gov-west-1 http.response.header.cache_control="no-cache, no-store" aws.operation=DescribeInstanceCreditSpecifications http.response.header.strict_transport_security="max-age=31536000; includeSubDomains" tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=c058833c-f38b-c9c9-3b6c-c881f7745cf8 @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.25/logger.go:138 timestamp=2023-07-20T16:22:46.203-0400
2023-07-20T16:22:46.204-0400 [TRACE] plugin.terraform-provider-aws_v4.59.0_x5.exe: Called downstream: tf_mux_provider="*schema.GRPCProviderServer" tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=c058833c-f38b-c9c9-3b6c-c881f7745cf8 @module=sdk.helper_schema tf_resource_type=aws_instance tf_rpc=ReadResource @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/resource.go:1016 timestamp=2023-07-20T16:22:46.203-0400
2023-07-20T16:22:46.204-0400 [TRACE] plugin.terraform-provider-aws_v4.59.0_x5.exe: Received downstream response: tf_rpc=ReadResource @module=sdk.proto diagnostic_error_count=0 tf_proto_version=5.3 tf_req_duration_ms=3114 tf_req_id=c058833c-f38b-c9c9-3b6c-c881f7745cf8 @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tf5serverlogging/downstream_request.go:37 diagnostic_warning_count=0 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_instance timestamp=2023-07-20T16:22:46.204-0400
2023-07-20T16:22:46.204-0400 [TRACE] plugin.terraform-provider-aws_v4.59.0_x5.exe: Served request: tf_rpc=ReadResource tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=c058833c-f38b-c9c9-3b6c-c881f7745cf8 tf_resource_type=aws_instance @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:761 @module=sdk.proto tf_proto_version=5.3 timestamp=2023-07-20T16:22:46.204-0400
2023-07-20T16:22:46.310-0400 [DEBUG] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-07-20T16:22:46.367-0400 [INFO]  plugin: plugin process exited: path=.terraform\providers\registry.terraform.io\hashicorp\aws\4.59.0\windows_amd64\terraform-provider-aws_v4.59.0_x5.exe pid=1584
2023-07-20T16:22:46.368-0400 [DEBUG] plugin: plugin exited
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
[github.com/GoogleCloudPlatform/terraformer/providers/aws.(*Ec2Generator).PostConvertHook(0xc0013d8000](http://github.com/GoogleCloudPlatform/terraformer/providers/aws.(*Ec2Generator).PostConvertHook(0xc0013d8000)?)
        /home/runner/work/terraformer/terraformer/providers/aws/ec2.go:99 +0x2e7
[github.com/GoogleCloudPlatform/terraformer/providers/aws.(*AwsFacade).PostConvertHook(0xc0037c52e8](http://github.com/GoogleCloudPlatform/terraformer/providers/aws.(*AwsFacade).PostConvertHook(0xc0037c52e8)?)
        /home/runner/work/terraformer/terraformer/providers/aws/aws_facade.go:75 +0x22
[github.com/GoogleCloudPlatform/terraformer/terraformutils.(*ProvidersMapping).CleanupProviders(0xc0037c5600)](http://github.com/GoogleCloudPlatform/terraformer/terraformutils.(*ProvidersMapping).CleanupProviders(0xc0037c5600))
        /home/runner/work/terraformer/terraformer/terraformutils/providers_mapping.go:175 +0xb4
[github.com/GoogleCloudPlatform/terraformer/cmd.Import({_](http://github.com/GoogleCloudPlatform/terraformer/cmd.Import(%7B_), _}, {{0xc000ef5800, 0x54, 0x80}, {0x117bc818, 0x0, 0x0}, {0xc00014803f, 0x14}, ...}, ...)
        /home/runner/work/terraformer/terraformer/cmd/import.go:110 +0x332
[github.com/GoogleCloudPlatform/terraformer/cmd.importRegionResources({{0xc0004c4260](http://github.com/GoogleCloudPlatform/terraformer/cmd.importRegionResources(%7B%7B0xc0004c4260), 0x1, 0x1}, {0x117bc818, 0x0, 0x0}, {0xc00014803f, 0x14}, {0xa4a117c, 0x9}, ...}, ...)
        /home/runner/work/terraformer/terraformer/cmd/provider_cmd_aws.go:121 +0x292
[github.com/GoogleCloudPlatform/terraformer/cmd.newCmdAwsImporter.func1(0xc000fca000](http://github.com/GoogleCloudPlatform/terraformer/cmd.newCmdAwsImporter.func1(0xc000fca000)?, {0xa493340?, 0x5?, 0x5?})
        /home/runner/work/terraformer/terraformer/cmd/provider_cmd_aws.go:66 +0x13e
[github.com/spf13/cobra.(*Command).execute(0xc000fca000](http://github.com/spf13/cobra.(*Command).execute(0xc000fca000), {0xc000294b40, 0x5, 0x5})
        /home/runner/go/pkg/mod/[github.com/spf13/cobra@v1.1.1/command.go:850](http://github.com/spf13/cobra@v1.1.1/command.go:850) +0x67c
[github.com/spf13/cobra.(*Command).ExecuteC(0xc000ee0dc0)](http://github.com/spf13/cobra.(*Command).ExecuteC(0xc000ee0dc0))
        /home/runner/go/pkg/mod/[github.com/spf13/cobra@v1.1.1/command.go:958](http://github.com/spf13/cobra@v1.1.1/command.go:958) +0x39d
[github.com/spf13/cobra.(*Command).Execute(..](http://github.com/spf13/cobra.(*Command).Execute(..).)
        /home/runner/go/pkg/mod/[github.com/spf13/cobra@v1.1.1/command.go:895](http://github.com/spf13/cobra@v1.1.1/command.go:895)
[github.com/GoogleCloudPlatform/terraformer/cmd.Execute()](http://github.com/GoogleCloudPlatform/terraformer/cmd.Execute())
        /home/runner/work/terraformer/terraformer/cmd/root.go:36 +0x1e
main.main()
        /home/runner/work/terraformer/terraformer/main.go:39 +0x50
github-actions[bot] commented 1 year ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

macbutch commented 12 months ago

I have the same issue:

panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/GoogleCloudPlatform/terraformer/providers/aws.(*Ec2Generator).PostConvertHook(0x14003051258?)
    github.com/GoogleCloudPlatform/terraformer/providers/aws/ec2.go:99 +0x2f8
github.com/GoogleCloudPlatform/terraformer/providers/aws.(*AwsFacade).PostConvertHook(0x0?)
    github.com/GoogleCloudPlatform/terraformer/providers/aws/aws_facade.go:75 +0x28
github.com/GoogleCloudPlatform/terraformer/terraformutils.(*ProvidersMapping).CleanupProviders(0x140030515d8)
    github.com/GoogleCloudPlatform/terraformer/terraformutils/providers_mapping.go:175 +0xa0
github.com/GoogleCloudPlatform/terraformer/cmd.Import({_, _}, {{0x14000b4e520, 0x1, 0x1}, {0x1143196f0, 0x0, 0x0}, {0x16d203306, 0x1d}, ...}, ...)
    github.com/GoogleCloudPlatform/terraformer/cmd/import.go:110 +0x1f4
github.com/GoogleCloudPlatform/terraformer/cmd.importRegionResources({{0x14000b4e4e0, 0x1, 0x1}, {0x1143196f0, 0x0, 0x0}, {0x16d203306, 0x1d}, {0x10a961607, 0x9}, ...}, ...)
    github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:121 +0x200
github.com/GoogleCloudPlatform/terraformer/cmd.newCmdAwsImporter.func1(0x14000e76580?, {0x10a952bea?, 0x8?, 0x8?})
    github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:58 +0x804
github.com/spf13/cobra.(*Command).execute(0x14000e76580, {0x14000261a00, 0x8, 0x8})
    github.com/spf13/cobra@v1.1.1/command.go:850 +0x4c4
github.com/spf13/cobra.(*Command).ExecuteC(0x14000c6b340)
    github.com/spf13/cobra@v1.1.1/command.go:958 +0x34c
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v1.1.1/command.go:895
github.com/GoogleCloudPlatform/terraformer/cmd.Execute()
    github.com/GoogleCloudPlatform/terraformer/cmd/root.go:36 +0x20
main.main()
    github.com/GoogleCloudPlatform/terraformer/main.go:39 +0x54
jakern commented 12 months ago

Thank you for saving this thread @macbutch . I don't know how I missed the stale notification. For what it's worth, I needed to switch to terracognita to finish my work. Might be helpful for you.

macbutch commented 12 months ago

Thanks @jakern. Will definitely check it out too. I think I've fixed this issue - possibly there are better approaches but without context on the codebase I've at least stopped the panic.

jakern commented 11 months ago

@macbutch, can you confirm that you were importing EC2 instances? It will take me a few days before I can get back on the system to try again if I even have the logs still but I'm fairly certain that this error occurred on more resource types than just EC2 instances. I'm pretty sure I also got this error for something in the IAM realm. Will get more examples when I can but it's likely that this fix doesn't cover everything.

macbutch commented 11 months ago

@jakern This fixed my issue which was only with EC2. It's a very old account and some of these instances are practically fossils. I'm guessing they are simply missing some fields that the API consistently returns for newer instances (but it's purely a guess).

I've not hit any similar issues on other services yet but I've got a lot more importing ahead of me: I've done two regions in one of 7 accounts.

I'm not sure if my change is complete. I think it's safe though because it's just adding a check before dereferencing.

nhooey commented 9 months ago

I'm getting the same error:

It would be great if all exceptions were caught and re-emitted with the name of the region, service, and other relevant context information. The logs are really messy and don't look so useful.

panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/GoogleCloudPlatform/terraformer/providers/aws.(*CloudFrontGenerator).PostConvertHook(0x1400647c240)
        github.com/GoogleCloudPlatform/terraformer/providers/aws/cloud_front.go:112 +0x66c
github.com/GoogleCloudPlatform/terraformer/providers/aws.(*AwsFacade).PostConvertHook(0x140002752b8?)
        github.com/GoogleCloudPlatform/terraformer/providers/aws/aws_facade.go:75 +0x28
github.com/GoogleCloudPlatform/terraformer/terraformutils.(*ProvidersMapping).CleanupProviders(0x140002755d8)
        github.com/GoogleCloudPlatform/terraformer/terraformutils/providers_mapping.go:175 +0xa0
github.com/GoogleCloudPlatform/terraformer/cmd.Import({_, _}, {{0x14000e23800, 0x56, 0x80}, {0x1160959d0, 0x0, 0x0}, {0x10c78d6c4, 0x1e}, ...}, ...)
        github.com/GoogleCloudPlatform/terraformer/cmd/import.go:110 +0x1f4
github.com/GoogleCloudPlatform/terraformer/cmd.importRegionResources({{0x140003bc3a0, 0x1, 0x1}, {0x1160959d0, 0x0, 0x0}, {0x10c78d6c4, 0x1e}, {0x10c6e4bdf, 0x9}, ...}, ...)
        github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:121 +0x200
github.com/GoogleCloudPlatform/terraformer/cmd.newCmdAwsImporter.func1(0x14000e9a580?, {0x10c6d61c8?, 0x8?, 0x8?})
        github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:58 +0x804
github.com/spf13/cobra.(*Command).execute(0x14000e9a580, {0x140003dce00, 0x8, 0x8})
        github.com/spf13/cobra@v1.1.1/command.go:850 +0x4c4
github.com/spf13/cobra.(*Command).ExecuteC(0x14000d97340)
        github.com/spf13/cobra@v1.1.1/command.go:958 +0x34c
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.1.1/command.go:895
github.com/GoogleCloudPlatform/terraformer/cmd.Execute()
        github.com/GoogleCloudPlatform/terraformer/cmd/root.go:36 +0x20
main.main()
        github.com/GoogleCloudPlatform/terraformer/main.go:39 +0x54