put with `plan_only: true` returns non zero exit code

[kurtmc]

I have been using this concourse resource in a large number of pipelines, only one of my pipelines seems to have this issue.

I am using the plan_only flag to run terraform plan on pull requests, one of my projects that deploys rundeck jobs fails every time, even though the plan output looks good.

I see the following output at the end of the put:

Plan: 0 to add, 21 to change, 0 to destroy.

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ Terraform Plan ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Failed To Run Terraform Plan!

2021/11/02 20:42:10 Plan Error: Failed to run Terraform command: exit status 1

If I run apply (no plan_only flag) the job succeeds.

I suspect the issue is due to the size of the state file being quite large, I can see in S3 that the state is 1.1MiB.

[ljfranklin]

Added a slightly better error message here: https://github.com/ljfranklin/terraform-resource/commit/0ccd897eee0dae3d0c0660e3db3e8109d50bda85. If you use fly intercept to SSH into the container you can view /tmp/tf-plan.log for more logs. I believe I saw that step print some credentials to the screen at one point, so we redirect output to a file for that step rather than logging directly to the build logs. Hopefully that file contains the underlying error message from Terraform.

[kurtmc]

Thanks @ljfranklin! Here is the content of /tmp/tf-plan.log:

╷
│ Warning: Value for undeclared variable
│ 
│ The root module does not declare a variable named
│ "tools_aws_secret_access_key" but a value was found in file
│ "/tmp/terraform-resource-out668882868/901638748vars-file.tfvars.json". If
│ you meant to use this value, add a "variable" block to the configuration.
│ 
│ To silence these warnings, use TF_VAR_... environment variables to provide
│ certain "global" settings to all configurations in your organization. To
│ reduce the verbosity of these warnings, use the -compact-warnings option.
╵
╷
│ Warning: Value for undeclared variable
│ 
│ The root module does not declare a variable named "concourse_username" but
│ a value was found in file
│ "/tmp/terraform-resource-out668882868/901638748vars-file.tfvars.json". If
│ you meant to use this value, add a "variable" block to the configuration.
│ 
│ To silence these warnings, use TF_VAR_... environment variables to provide
│ certain "global" settings to all configurations in your organization. To
│ reduce the verbosity of these warnings, use the -compact-warnings option.
╵
╷
│ Warning: Values for undeclared variables
│ 
│ In addition to the other similar warnings shown, 7 other variable(s)
│ defined without being declared.
╵
╷
│ Error: Plugin error
│ 
│   with stateful_string.plan_output,
│   on resource_plan_config.tf line 10, in resource "stateful_string" "plan_output":
│   10: resource "stateful_string" "plan_output" {
│ 
│ The plugin returned an unexpected error from
│ plugin.(*GRPCProvider).PlanResourceChange: rpc error: code =
│ ResourceExhausted desc = grpc: received message larger than max (7192174
│ vs. 4194304)
╵
╷
│ Error: Plugin error
│ 
│   with stateful_string.plan_output_json,
│   on resource_plan_config.tf line 13, in resource "stateful_string" "plan_output_json":
│   13: resource "stateful_string" "plan_output_json" {
│ 
│ The plugin returned an unexpected error from
│ plugin.(*GRPCProvider).PlanResourceChange: rpc error: code =
│ ResourceExhausted desc = grpc: received message larger than max (7016630
│ vs. 4194304)
╵

[ljfranklin]

The underlying error:

│ The plugin returned an unexpected error from
│ plugin.(*GRPCProvider).PlanResourceChange: rpc error: code =
│ ResourceExhausted desc = grpc: received message larger than max (7016630
│ vs. 4194304)

Looks like a Terraform limitation. Here's a PR that increases the gRPC MaxRecvMsgSize limit for the GetSchema call: https://github.com/hashicorp/terraform/pull/20906. But the PlanResourceChange call in your error doesn't have that same size limit increase: https://github.com/hashicorp/terraform/blob/2ebc2beda0149a362055cc5e16c42e1549a884ec/plugin/grpc_provider.go#L388-L392. You can try opening an issue/PR to Terraform to see if they'll increase the limit there as well. Alternatively you can workaround the issue by reducing the size of the plan file but I'm not sure how actionable that is.

[jpsmoura]

Hi @ljfranklin I'm getting a similar problem. There seems to be a dependency on a plugin running on TF 0.12.0 that was last updated on Feb 2021 https://github.com/ashald/terraform-provider-stateful https://github.com/ljfranklin/terraform-resource/blob/master/src/terraform-resource/terraform/client.go#L170 Any suggestions?