Terraform 0.13 crash on aws_cloudfront_distribution import

peterpakos commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.0
+ provider registry.terraform.io/hashicorp/aws v3.3.0

Affected Resource(s)

aws_cloudfront_distribution

Terraform Configuration Files

resource "aws_cloudfront_distribution" "default" {
  origin {
    domain_name = aws_s3_bucket.default.bucket_regional_domain_name
    origin_id   = "S3-${aws_s3_bucket.default.id}"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.default.cloudfront_access_identity_path
    }
  }

  enabled             = true
  is_ipv6_enabled     = false
  default_root_object = "index.html"

  aliases = [var.bucket_name]

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "S3-${aws_s3_bucket.default.id}"

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 0

    lambda_function_association {
      event_type   = "origin-request"
      include_body = false
      lambda_arn   = "arn:aws:lambda:us-east-1:225237029829:function:s3-cdn:2"
    }
  }

Debug Output

https://gist.github.com/peterpakos/eeffe7d7b94a0b0928832d4225e6bcdd

Panic Output

https://gist.github.com/peterpakos/05b9cf9d66195ddcc3c24f48e1da1880

Expected Behavior

Terraform import command successful with Cloudfront Distribution successfully imported to the state file.

Actual Behavior

Terraform import command crashed.

Steps to Reproduce

terraform import module.s3-cdn.aws_cloudfront_distribution.default E32EAQRW369ZYR

Important Factoids

Cloudfront Distribution E32EAQRW369ZYR contains 2 Origins:

S3 Origin
Custom Origin

It also has 2 Behaviours configured:

Default (*) for Custom Origin
Path ui/* for S3 Origin.

The latter Behaviour also has Lambda@Edge Function associated with Viewer Request.

bflad commented 4 years ago

Relevant portion of the crash log:

2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: panic: runtime error: invalid memory address or nil pointer dereference
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x38173bb]
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: goroutine 73 [running]:
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenForwardedValues(0x0, 0xc00152a4b0)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:430 +0x2b
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehavior(0xc001825980, 0x5839f60)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:304 +0x22a
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehaviors(0xc001520e40, 0x59fc4f0, 0x15, 0x5839f60)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:185 +0x89
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenDistributionConfig(0xc001a86800, 0xc0000f9200, 0xc001496740, 0x0)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:123 +0xa5b
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsCloudFrontDistributionRead(0xc001a86800, 0x4e5c540, 0xc00033b900, 0x97c6030, 0x5ae3920)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_cloudfront_distribution.go:729 +0x35e
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc00070cb00, 0x6490b00, 0xc001ab4700, 0xc001a86800, 0x4e5c540, 0xc00033b900, 0x0, 0x0, 0x0)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:281 +0x88

Looks like the query_string handling is missing a nil check here:

https://github.com/terraform-providers/terraform-provider-aws/blob/116a671279121171385dca2a9a6cc2bfdd15a612/aws/cloudfront_distribution_configuration_structure.go#L430

anGie44 commented 4 years ago

The fix for this has been merged and will release with version 3.4.0 of the Terraform AWS Provider, likely out tomorrow.

peterpakos commented 4 years ago

Great stuff, thanks!

ghost commented 4 years ago

This has been released in version 3.4.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

peterpakos commented 4 years ago

Unfortunately it still doesn't work with the latest terraform/provider:

Terraform v0.13.1
+ provider registry.terraform.io/hashicorp/aws v3.4.0

crash.log: https://gist.github.com/peterpakos/9b0a15f84a5e1afb9b1c136288218782

✗ terraform import 'module.s3-cdn[0].aws_cloudfront_distribution.default' E32EAQRW369ZYR
module.s3-cdn[0].aws_cloudfront_distribution.default: Importing from ID "E32EAQRW369ZYR"...
module.s3-cdn[0].aws_cloudfront_distribution.default: Import prepared!
  Prepared aws_cloudfront_distribution for import
module.s3-cdn[0].aws_cloudfront_distribution.default: Refreshing state... [id=E32EAQRW369ZYR]

Error: rpc error: code = Unavailable desc = transport is closing

panic: runtime error: invalid memory address or nil pointer dereference
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x445734f]
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: goroutine 88 [running]:
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenForwardedValues(0x0, 0xc0015f3b00)
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:430 +0x2f
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehavior(0xc0007b5580, 0x64a34e0)
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:304 +0x245
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehaviors(0xc001c76a20, 0x666980e, 0x15, 0x64a34e0)
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:185 +0x89
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenDistributionConfig(0xc0014cea00, 0xc000e330e0, 0xc001e30220, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:123 +0xa5b
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsCloudFrontDistributionRead(0xc0014cea00, 0x5aba380, 0xc000908000, 0xa488e20, 0x6751da8)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_cloudfront_distribution.go:729 +0x35e
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0000c26e0, 0x710b300, 0xc0014e9100, 0xc0014cea00, 0x5aba380, 0xc000908000, 0x0, 0x0, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:281 +0x88
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0000c26e0, 0x710b300, 0xc0014e9100, 0xc0019cb1f0, 0x5aba380, 0xc000908000, 0xc00000fe80, 0x0, 0x0, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:555 +0x1c2
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0007a91a0, 0x710b300, 0xc0014e9100, 0xc001e0e0c0, 0xc0007a91a0, 0xc0007a91b0, 0x6751d98)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/helper/plugin/grpc_provider.go:572 +0x42f
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5._Provider_ReadResource_Handler.func1(0x710b300, 0xc0014e9100, 0x6420300, 0xc001e0e0c0, 0xc0014e9100, 0x5c2c540, 0xc001e0e001, 0xc0014bb500)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5/tfplugin5.pb.go:3276 +0x86
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve.func3.1(0x710b3c0, 0xc0015bee10, 0x6420300, 0xc001e0e0c0, 0xc0014bb4e0, 0xc0014bb500, 0xc0010a8ba0, 0x108fe18, 0x62896c0, 0xc0015bee10)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/plugin/serve.go:76 +0x87
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5._Provider_ReadResource_Handler(0x64f6280, 0xc0007a91a0, 0x710b3c0, 0xc0015bee10, 0xc001e0e060, 0xc00108a280, 0x710b3c0, 0xc0015bee10, 0xc001e10000, 0x200)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5/tfplugin5.pb.go:3278 +0x14b
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000162700, 0x712c8a0, 0xc001095080, 0xc0014d6100, 0xc001482510, 0xa44ec50, 0x0, 0x0, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1171 +0x50a
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc000162700, 0x712c8a0, 0xc001095080, 0xc0014d6100, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1494 +0xccd
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000d1e200, 0xc000162700, 0x712c8a0, 0xc001095080, 0xc0014d6100)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:834 +0xa1
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:832 +0x204
2020/09/02 20:33:07 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/09/02 20:33:07 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/09/02 20:33:07 [TRACE] [walkImport] Exiting eval tree: import module.s3-cdn[0].aws_cloudfront_distribution.default result
2020/09/02 20:33:07 [TRACE] vertex "import module.s3-cdn[0].aws_cloudfront_distribution.default result": visit complete
2020/09/02 20:33:07 [TRACE] vertex "module.s3-cdn[0].aws_cloudfront_distribution.default (import id \"E32EAQRW369ZYR\")": dynamic subgraph encountered errors
2020/09/02 20:33:07 [TRACE] vertex "module.s3-cdn[0].aws_cloudfront_distribution.default (import id \"E32EAQRW369ZYR\")": visit complete
2020-09-02T20:33:07.088+0100 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020/09/02 20:33:07 [TRACE] dag/walk: upstream of "module.s3-cdn (close)" errored, so skipping
2020-09-02T20:33:07.088+0100 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.4.0/darwin_amd64/terraform-provider-aws_v3.4.0_x5 pid=87888 error="exit status 2"
2020/09/02 20:33:07 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2020/09/02 20:33:07 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020-09-02T20:33:07.287+0100 [DEBUG] plugin: plugin exited

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

anGie44 commented 4 years ago

ahh thanks for following up on this @peterpakos! Looks like we'll have to add an explicit check for the nil value returned from the API; in this case the *cloudfront.ForwardedValues param seems to be the culprit

dShringi commented 4 years ago

I'm facing the same issue and yes I can confirm it is still not resolved after the update. Please let us know if there is a workaround for now. At the moment I have just removed the code for aws_cloudfront distribution and the state from the state file and it is working fine.

ewbankkit commented 4 years ago

@dShringi Could you please open a new issue for this so that we can track and fix? Thanks.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashicorp / terraform-provider-aws