Closed tgoodsell-tempus closed 1 year ago
Based on the change diff, I suspect for us it's explicitly something changing about how the defaults / env vars are handled with: https://github.com/hashicorp/terraform-provider-google/pull/13645
This seems to be the same bug we have encountered encountered as well.
Looks like the same issue we also facing with: We are running terraform inside Container image: hashicorp/terraform:1.0.8 Issues seems to be within this image, because also we tried to run on MAC and Ubuntu laptops everything works fine. Bellow is an stack trace we are getting:
$ terraform plan -out "planfile"
β·
β Error: Plugin did not respond
β
β with provider["registry.terraform.io/hashicorp/google"],
β on providers.tf line 1, in provider "google":
β 1: provider "google" {
β
β The plugin encountered an error, and failed to respond to the
β plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
β more details.
β΅
Stack trace from the terraform-provider-google_v4.53.0_x5 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x5ecc01]
goroutine 168 [running]:
github.com/hashicorp/terraform-plugin-log/internal/logging.GetProviderRootLogger(...)
github.com/hashicorp/terraform-plugin-log@v0.7.0/internal/logging/provider.go:12
github.com/hashicorp/terraform-plugin-log/tflog.Info({0x0, 0x0}, {0x30d2361, 0x85}, {0x0, 0x0, 0x0})
github.com/hashicorp/terraform-plugin-log@v0.7.0/tflog/provider.go:74 +0x61
github.com/hashicorp/terraform-provider-google/google.getCurrUserEmail(0xc0001e7800, {0xc000e5a100?, 0x356c300?}, 0xc001084dd0)
github.com/hashicorp/terraform-provider-google/google/framework_utils.go:44 +0x25f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).logGoogleIdentities(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:1219 +0x698
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).SetupClient(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:1149 +0x28f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).ConfigureWithData(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:47 +0x23e
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).Configure(0xc0001e7800, {0x3575238, 0xc000c0a300}, {{0xc0011061a8, 0x5}, {{{0x357b990, 0xc000c1fb60}, {0x2b3a1e0, 0xc000c1a720}}, {0x357ce58, ...}}}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_provider.go:821 +0x225
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ConfigureProvider(0xc0004509a0, {0x3575238, 0xc000c0a300}, 0xc001084e40, 0xc001084dc0)
github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:15 +0x10f
github.com/hashicorp/terraform-plugin-framework/internal/proto5server.(*Server).ConfigureProvider(0xc0004509a0, {0x3575238?, 0xc000c0a1b0?}, 0x3?)
github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/proto5server/server_configureprovider.go:36 +0x292
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ConfigureProvider({0xc000974a80, 0xc000974ae0, {0xc000f67840, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ConfigureProvider.go:25 +0x1b3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).Configure(0xc000aef680, {0x3575238?, 0xc000c03890?}, 0xc001084a80)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:556 +0x2ce
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Configure_Handler({0x2f0[88](https://gitlab.com/aerialdev/platform/data-platform/infrastructure/datastream/-/jobs/3767225211#L88)60?, 0xc000aef680}, {0x3575238, 0xc000c03890}, 0xc0010b7f10, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:331 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001012000, {0x357bc20, 0xc001020820}, 0xc000e9d200, 0xc000ffcc60, 0x47d66b8, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc001012000, {0x357bc20, 0xc001020820}, 0xc000e9d200, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.51.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.51.0/server.go:963 +0x28a
Error: The terraform-provider-google_v4.53.0_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
Same error on here as well since today, on apply and plan
Having kinda the same issues/errors here with v4.53.0, everything worked before.. I have some nil pointer reference errors:
Stack trace from the terraform-provider-google_v4.53.0_x5 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x5ecc01]
goroutine 141 [running]:
github.com/hashicorp/terraform-plugin-log/internal/logging.GetProviderRootLogger(...)
github.com/hashicorp/terraform-plugin-log@v0.7.0/internal/logging/provider.go:12
github.com/hashicorp/terraform-plugin-log/tflog.Info({0x0, 0x0}, {0x30d2361, 0x85}, {0x0, 0x0, 0x0})
github.com/hashicorp/terraform-plugin-log@v0.7.0/tflog/provider.go:74 +0x61
github.com/hashicorp/terraform-provider-google/google.getCurrUserEmail(0xc0001f3800, {0xc000a9a280?, 0x3570360?}, 0xc000e18690)
github.com/hashicorp/terraform-provider-google/google/framework_utils.go:44 +0x25f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).logGoogleIdentities(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x2, {0xc00005001a, 0x1d3}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:1219 +0x698
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).SetupClient(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x2, {0xc00005001a, 0x1d3}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:1149 +0x28f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).ConfigureWithData(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x2, {0xc00005001a, 0x1d3}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:47 +0x23e
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).Configure(0xc0001f3800, {0x3575238, 0xc000fd30b0}, {{0xc0001e35d0, 0x6}, {{{0x357b990, 0xc000fe4810}, {0x2b3a1e0, 0xc000fe0b40}}, {0x357ce58, ...}}}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_provider.go:821 +0x225
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ConfigureProvider(0xc000452580, {0x3575238, 0xc000fd30b0}, 0xc000e18700, 0xc000e18680)
github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:15 +0x10f
github.com/hashicorp/terraform-plugin-framework/internal/proto5server.(*Server).ConfigureProvider(0xc000452580, {0x3575238?, 0xc000fd2f60?}, 0x3?)
github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/proto5server/server_configureprovider.go:36 +0x292
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ConfigureProvider({0xc0002bd2f0, 0xc0002bd350, {0xc0009803c0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ConfigureProvider.go:25 +0x1b3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).Configure(0xc0001fcc80, {0x3575238?, 0xc000fd2630?}, 0xc000e18340)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:556 +0x2ce
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Configure_Handler({0x2f08860?, 0xc0001fcc80}, {0x3575238, 0xc000fd2630}, 0xc000def0a0, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:331 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000ab6000, {0x357bc20, 0xc0006b6820}, 0xc000e1efc0, 0xc000b895f0, 0x47d66b8, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc000ab6000, {0x357bc20, 0xc0006b6820}, 0xc000e1efc0, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.51.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.51.0/server.go:963 +0x28a
Error: The terraform-provider-google_v4.53.0_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
β·
β Error: Plugin did not respond
β
β with provider["registry.terraform.io/hashicorp/google"],
β on main.tf line 10, in provider "google":
β 10: provider "google" {
β
β The plugin encountered an error, and failed to respond to the
β plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
β more details.
β΅
I am having the same error.
same error, work around with following for now:
required_providers {
google = {
source = "hashicorp/google"
version = ">=4.47.0, != 4.53.0"
}
}
Same error as well.
Same error:
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
β·
β Error: Plugin did not respond
β
β with provider["registry.terraform.io/hashicorp/google"],
β on main.tf line 1, in provider "google":
β 1: provider "google" {
β
β The plugin encountered an error, and failed to respond to the
β plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
β more details.
β΅
Stack trace from the terraform-provider-google_v4.53.0_x5 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x5ecc01]
goroutine 163 [running]:
github.com/hashicorp/terraform-plugin-log/internal/logging.GetProviderRootLogger(...)
github.com/hashicorp/terraform-plugin-log@v0.7.0/internal/logging/provider.go:12
github.com/hashicorp/terraform-plugin-log/tflog.Info({0x0, 0x0}, {0x30d2361, 0x85}, {0x0, 0x0, 0x0})
github.com/hashicorp/terraform-plugin-log@v0.7.0/tflog/provider.go:74 +0x61
github.com/hashicorp/terraform-provider-google/google.getCurrUserEmail(0xc00011d500, {0xc000ac4100?, 0x356c300?}, 0xc000f10c50)
github.com/hashicorp/terraform-provider-google/google/framework_utils.go:44 +0x25f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).logGoogleIdentities(, {, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:1219 +0x698
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).SetupClient(, {, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:1149 +0x28f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).ConfigureWithData(, {, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_config.go:47 +0x23e
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).Configure(0xc00011d500, {0x3575238, 0xc0001dc000}, {{0xc00121e060, 0x5}, {{{0x357b990, 0xc00128b920}, {0x2b3a1e0, 0xc0001fb3b0}}, {0x357ce58, ...}}}, ...)
github.com/hashicorp/terraform-provider-google/google/framework_provider.go:821 +0x225
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ConfigureProvider(0xc0004b0160, {0x3575238, 0xc0001dc000}, 0xc000f10cc0, 0xc000f10c40)
github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:15 +0x10f
github.com/hashicorp/terraform-plugin-framework/internal/proto5server.(*Server).ConfigureProvider(0xc0004b0160, {0x3575238?, 0xc001091f20?}, 0x3?)
github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/proto5server/server_configureprovider.go:36 +0x292
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ConfigureProvider({0xc000983650, 0xc0009836b0, {0xc000faa460, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ConfigureProvider.go:25 +0x1b3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).Configure(0xc000508d20, {0x3575238?, 0xc001090690?}, 0xc000f108c0)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:556 +0x2ce
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Configure_Handler({0x2f08860?, 0xc000508d20}, {0x3575238, 0xc001090690}, 0xc000db6000, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:331 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000001680, {0x357bc20, 0xc000bac340}, 0xc000ec4900, 0xc000ff9290, 0x47d66b8, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc000001680, {0x357bc20, 0xc000bac340}, 0xc000ec4900, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.51.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.51.0/server.go:963 +0x28a
Error: The terraform-provider-google_v4.53.0_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
same error, work around with following for now:
required_providers { google = { source = "hashicorp/google" version = ">=4.47.0, != 4.53.0" } }
Thanks, that fixed it for me!
Avoiding 4.53.0 worked for us too as a workaround.
To help us investigate, could someone who is experiencing this issue please provide us with which environment variables you are setting?
To help us investigate, could someone who is experiencing this issue please provide us with which environment variables you are setting?
In my case, I am just running it on a GCE binded with GCP ServiceAccount Didn't actually using any environment variables
atlantis-on-gce:/# gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* atlantis@${our_project}.iam.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
atlantis-on-gce:/# printenv | grep GOOGLE
(Empty)
We could reproduce the issue locally on a developer pc (macos), without setting any specific evn-vars before starting terraform init/apply.
@roaks3 We experience the issue during init even when our provider config does not rely heavily on env vars at all:
provider google {
project = "any-of-our-projects"
region = "any"
token = "some access token from vault"
}
These are run in concourse so there are plenty other env vars but we dont rely on those for terraform.
To help us investigate, could someone who is experiencing this issue please provide us with which environment variables you are setting?
I get this issue when running with the following environment variables set and no provider configuration in the terraform files (ie just provider "google" {}
):
GOOGLE_OAUTH_ACCESS_TOKEN
GOOGLE_PROJECT
GOOGLE_REGION
I'm sorry about all of this, we're working hard to get to the bottom of it. Has anyone been able to run with a TRACE logging? I think i have a fix for the crash, however, the crash happens when trying to log an error, and I'm concerned the error is being covered by this crash. I'd feel more comfortable with the fix if I could see the underlying error that gets us to this point, then we might have a better chance of reproducing it. Thanks!
Also encountering the issue, every test that ran last night experienced the bug. Like Hadrien I also just have provider "google" {}
.
I'm sorry about all of this, we're working hard to get to the bottom of it. Has anyone been able to run with a TRACE logging? I think i have a fix for the crash, however, the crash happens when trying to log an error, and I'm concerned the error is being covered by this crash. I'd feel more comfortable with the fix if I could see the underlying error that gets us to this point, then we might have a better chance of reproducing it. Thanks!
@megan07
Here is a TRACE log of the issue: https://gist.github.com/tgoodsell-tempus/03dd21b4c89d4d5e270f89aa8f7fe9a0
This was produced by only setting GOOGLE_OAUTH_ACCESS_TOKEN
and then using the following config:
data "google_project" "project" {
project_id = "example-project"
}
resource "google_storage_bucket" "example" {
name = "snd-psc-host-testing"
location = "US"
force_destroy = true
project = data.google_project.project.project_id
public_access_prevention = "enforced"
}
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
}
}
Trace log obtained with the following snippet:
provider "google" {}
resource "google_storage_bucket" "example" {
name = local.bucket_name
location = var.location
}
We're currently creating a revert of the change we believe triggered the issue, will test the revert resolved the issue with a reproduction we've identified, and will push a 4.53.1
release from there. I'd estimate it's a low number of hours (2-4) until that's out.
Sorry about the delay on our end so far- across a handful of folks on the team we'd been unable to reproduce this for some time. We've got one now (supplying an invalid access token) but it's not one I'm entirely happy with because it's likely not representative of the trigger for anyone who's reported an issue in this thread.
Edit: Sounding this out loud, maybe it has to do with scopes on the token- I'll try that out.
4.53.1
is going out now, it should be available within the hour.
Edit: Releases for both providers are complete and 4.53.1
appears in the Terraform Registry (https://registry.terraform.io/providers/hashicorp/google/4.53.1), individual terraform init
runs may be hitting a cache in the meantime
I've got a reproduction that's more representative of a typical user configuration than my earlier one (supplying a bad access token). We have logic during provider initialization that logs the user's identity into local debug output (using a different identity than expected is a common source of customer issue!) that's expected to silently log an error if it encounters an issue and proceed, failing later in the configuration or apply. That's largely because that logic requires an additional scope ("userinfo-email"
) that isn't required by the rest of the provider. Using an access token that does not include that scope (such as one produced by the steps below) triggers the issue.
During a large code change to the provider initialization logic, the logging code was inadvertently changed. I'm speculating a bit to the exact cause here (other folks on the team drove the investigation), but I believe that we both accessed global state out of order (causing this crash) and threw an error when we encountered an error when checking the Terraform runner's identity, rather than moving on. That meant that the initial solution in https://github.com/GoogleCloudPlatform/magic-modules/pull/7285, which resolves the crash, still failed at provider initialization time.
Configuration to produce an invalid access token:
data "google_service_account_access_token" "default" {
target_service_account = "my-service-account@my-project.iam.gserviceaccount.com"
scopes = ["cloud-platform"]
lifetime = "3600s"
}
output "token" {
sensitive = true
value = nonsensitive(data.google_service_account_access_token.default.access_token)
}
Use that with Terraform:
GOOGLE_OAUTH_ACCESS_TOKEN={{access token}} terraform apply
With the following (simple) configuration:
provider "google" {
version = "= 4.53.0"
}
resource "google_compute_address" "default" {
name = "my-address"
}
Closing this now that the release is out + we've received positive confirmation that it worked for a customer internally. If this doesn't resolve the issue, please comment here and we'll reopen it!
I'm going to lock this issue because it has been closed for 30 days β³. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Community Note
modular-magician
user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned tohashibot
, a community member has claimed the issue already.Terraform Version
1.3.1
Affected Resource(s)
Terraform Configuration Files
We set our credentials using the ENV VARs available.
Debug Output
https://gist.github.com/tgoodsell-tempus/bb4ce2a8a3d20752e00a170f1aae4751
Panic Output
Expected Behavior
Provider starts as expected
Actual Behavior
Crashing on init/setup
Steps to Reproduce
Important Factoids
google
provider block directlyReferences