`terraform plan` fails with `unexpected EOF` when reading remote state

hashibot commented 6 years ago

This issue was originally opened by @BlueDragonX as hashicorp/terraform#15696. It was migrated here as a result of the provider split. The original body of the issue is below.

Terraform Version

Terraform v0.9.11

Terraform Configuration Files

data "terraform_remote_state" "global" {
  backend = "s3"

  config {
    region     = "us-west-2"
    bucket     = "redacted"
    key        = "zo/global.tfstate"
    lock_table = "terraform-state-lock"
  }
}

Debug Output

https://gist.github.com/BlueDragonX/dbce26f86fabb841c26e27b6e66d0e82

Expected Behavior

I expect the terraform plan to run successfully.

Actual Behavior

I get this instead:

Error refreshing state: 2 error(s) occurred:

* data.terraform_remote_state.global: 1 error(s) occurred:

* data.terraform_remote_state.global: data.terraform_remote_state.global: unexpected EOF
* data.terraform_remote_state.net: 1 error(s) occurred:

* data.terraform_remote_state.net: data.terraform_remote_state.net: unexpected EOF

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

Use remote state.
terraform plan

Important Factoids

I've done some digging in the code to see why exactly this is happening. The important bit in the debug log is panic: Unknown: <invalid reflect.Value> which comes from builtin/providers/terraform/flatten.go. Based on the stack trace this implies that one of the output's values was parsed incorrectly. Given that output.Value is an interface and Terraform calls json.Unmarshal to build that, I'm concerned that we're running up against some sort of bug.

Here are the outputs for each of the modules in the state file (mildly redacted):

$ jq '.modules[] | .outputs' global.tfstate
{
  "domain": {
    "sensitive": false,
    "type": "string",
    "value": "redacted."
  },
  "hosted_zone_id": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  }
}
{}
{
  "account": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "accounts": {
    "sensitive": false,
    "type": "map",
    "value": {
      "zd": "redacted",
      "zh": "redacted",
      "zi": "redacted",
      "zo": "redacted",
      "zp": "redacted",
      "zs": "redacted"
    }
  },
  "coreos_ami": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "datacenter": {
    "sensitive": false,
    "type": "string",
    "value": "uw2"
  },
  "ecr_account": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "ecr_domain": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "ecr_region": {
    "sensitive": false,
    "type": "string",
    "value": "us-west-2"
  },
  "env": {
    "sensitive": false,
    "type": "string",
    "value": "zo"
  },
  "prefix": {
    "sensitive": false,
    "type": "string",
    "value": "zo-uw2-"
  },
  "region": {
    "sensitive": false,
    "type": "string",
    "value": "us-west-2"
  },
  "zone_prefix": {
    "sensitive": false,
    "type": "map",
    "value": {
      "us-west-2a": "zo-uw2a-",
      "us-west-2b": "zo-uw2b-",
      "us-west-2c": "zo-uw2c-"
    }
  }
}

I'm not sure why that would be parsed by json.Unmarshal incorrectly but I'm going to experiment to see why.

What I really don't get is why this has worked fine for me previously using the same version but is breaking now. I've deployed this same state in other envirnments and it works.

scream314 commented 6 years ago

~~TF used: Terraform v0.10.0~~

~~After upgrading , I am getting unexpected EOF for all HEREDOCs in aws resources. But if I use HEREDOC with resources provided by (for example) the terraform-provider-libvirt provider, they are accepted. What really funny is, the number of errors are changing if I run plan again.~~

~~In the logs there is first~~

2017/08/04 17:04:31 [DEBUG] plugin: terraform-provider-aws_v0.1.3_x4: panic: runtime error: invalid memory address or nil pointer dereference
2017/08/04 17:04:31 [DEBUG] plugin: terraform-provider-aws_v0.1.3_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x23a327f]

~~then a lot of~~

2017/08/04 17:04:31 [DEBUG] dag/walk: upstream errored, not walking "X.Y.Z.A"

~~Is the connection to AWS interrupted? That would explain why the number of errors is inconsistent (depends on what is left as it fails).~~

~~Ok, I did a run with -parallelism=1, the error is now connection is shut down. With -parallelism=10 I get only unexpected EOFs, with -parallelism=5 a mix of the two.~~

Update: It is something with the network or the AWS API, because I tried terraform plan from another WAN/ISP link and it worked.

Update2: I do not know what happened/changed, but as I tried today I was able to do terraform plan from anywhere. So my issue is resolved.

Update3: The panic happened again, found out it is unrelated to this issue. My problem seems to be fixed in https://github.com/terraform-providers/terraform-provider-aws/commit/3a383997187f718809ffd888a9c585fb725a52b7.

mykhailoponomarov commented 6 years ago

Hi all, I'm experiencing the same error on 0.10, while 0.9.11 works as expected.

catsby commented 6 years ago

Hey @mikhailponomaryov , the previous user reported that https://github.com/terraform-providers/terraform-provider-aws/pull/899 has addressed their issue. That patch should be included in the Terraform AWS Provider since version 0.1.2, do you know what version you're on?

mykhailoponomarov commented 6 years ago

@catsby, Thanks a lot. I have managed to get it worked with the version constraint set to "0.1.4".

scream314 commented 6 years ago

@catsby The error happened with terraform-provider-aws_v0.1.3_x4 and was gone as I go geted the AWS Provider and used that version.

mgresko commented 6 years ago

I see the same issue going from TF 10.7 -> 10.8. I have tried various version of the aws provider all resulting in the same unexpected EOF. Specifically tried: 0.1.4 and 1.2.0. Stuck at the moment.

mgresko commented 6 years ago

017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: panic: Unknown: <invalid reflect.Value>

This happens on a remote state file. If you run plan directly in the project that created the remote state it works fine. It is just the remote state call.

2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: panic: Unknown: <invalid reflect.Value>
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: goroutine 55 [running]:
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/terraform.flatten(0xc0425d4f30, 0xc0423662e0, 0x7, 0x0, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/terraform/flatten.go:48 +0x4f4
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/terraform.remoteStateFlatten(0xc0423457b0, 0xc0423457b0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/terraform/flatten.go:21 +0x11e
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/terraform.dataSourceRemoteStateRead(0xc0423cc150, 0x0, 0x0, 0xc04238f1e0, 0x16)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/terraform/data_source_state.go:114 +0x7cf
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).ReadDataApply(0xc0423177a0, 0xc04214d460, 0x0, 0x0, 0xc04233acb8, 0x1, 0x80000000000)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:259 +0xb7
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).ReadDataApply(0xc0422f1e30, 0xc0421485a0, 0xc04214d460, 0x3ab0d50, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:426 +0xa1
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).ReadDataApply(0xc04214cee0, 0xc042141cd0, 0xc042141d00, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:604 +0x55
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: reflect.Value.call(0xc042316480, 0xc042156648, 0x13, 0x22e278d, 0x4, 0xc042387f20, 0x3, 0x3, 0x0, 0x0, ...)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/reflect/value.go:434 +0x90d
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: reflect.Value.Call(0xc042316480, 0xc042156648, 0x13, 0xc042387f20, 0x3, 0x3, 0x0, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/reflect/value.go:302 +0xab
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: net/rpc.(*service).call(0xc04215fe40, 0xc04215d310, 0xc0422aa5b8, 0xc042158b00, 0xc0422a4660, 0x1b609c0, 0xc042141cd0, 0x16, 0x1b60a00, 0xc042141d00, ...)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/net/rpc/server.go:381 +0x149
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: created by net/rpc.(*Server).ServeCodec
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/net/rpc/server.go:475 +0x372
2017/11/07 16:23:12 [ERROR] root: eval: *terraform.EvalReadDataApply, err: data.terraform_remote_state.management: unexpected EOF
2017/11/07 16:23:12 [ERROR] root: eval: *terraform.EvalSequence, err: data.terraform_remote_state.management: unexpected EOF
2017/11/07 16:23:12 [TRACE] [walkRefresh] Exiting eval tree: data.terraform_remote_state.management
2017/11/07 16:23:12 [TRACE] dag/walk: upstream errored, not walking "provider.terraform (close)"

mgresko commented 6 years ago

FYI running terraform refresh fixed this issue. Sigh.

burdiyan commented 6 years ago

I constantly face this issue when updating resources. After getting unexpected EOF, the resource that was refreshing somehow is removed from the state, and then on the next apply is trying to create again, which completely breaks the workflow, because the resource actually exist.

thiagocaiubi commented 6 years ago

I got unexpected EOF when edited something in Azure portal that Terraform doesn't support.

I tried to add a comma separated list to destination_port_range on azurerm_network_security_rule but it only supports an integer, or a range between o and 65535 or * to indicate any port.

Although the UI accepts comma separated list like 80,443 terraform only implements range support. So I updated the rule directly in the portal just to test my setup and forgot to rollback. So when applied terraform plan I was getting unexpected EOF.

Just deleted the rule and everything started to work again.

burdiyan commented 6 years ago

OMG, I have to take my words back. The EOF issue was because of a panic in our custom provider! I figured it out setting TF_LOG env variable.

RyanJarv commented 6 years ago

For anyone else running into the error panic: Unknown: <invalid reflect.Value> I somehow had a output in the remote tfstate that was set to a value of null. Removing the output there fixed the issue.

cregkly commented 6 years ago

We also get Unexpected EOF from having a output configured in a remote state, the error in debug throws: panic: Unknown: %!s(float64=2)

The bad output configuration:

output "foo" {
  value = 2
}

Fix was to have the integer quoted as a string like this:

output "foo" {
  value = "2"
}

The state file the incorrect configuration looks like this:

...
                "foo": {
                    "sensitive": false,
                    "type": "string",
                    "value":  2,
                },
...

It should be

...
                "foo": {
                    "sensitive": false,
                    "type": "string",
                    "value":  "2",
                },
...

Note the the output posted to the console when applying the state will be shown without quotes in both cases.

Outputs:

foo = 2

hashicorp / terraform-provider-terraform