Plugin crushes after upgrading to 1.3.1

gkocur commented 4 years ago

Terraform Version and Provider Version

v0.13.3

Provider Version

v1.3.1

Terraform Configuration Files

provider "helm" {
  kubernetes {
    load_config_file = false
    token = data.google_client_config.current.access_token
    host = module.k8s.cluster_endpoint
    client_certificate     = module.k8s.client_certificate
    client_key             = module.k8s.cluster_endpoint
    cluster_ca_certificate = module.k8s.cluster_ca_certificate
  }
  version = "1.3.1"
}

Expected Behavior

terraform plan and terraform apply should work as expected

Actual Behavior

Plugin crushes with panic: runtime error: invalid memory address or nil pointer dereference

Steps to Reproduce

terraform plan

Important Factoids

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

As in the title - after upgrading to v1.3.1 running plan or apply makes the plugin crush. Works fine with v1.3.0

sennerholm commented 4 years ago

Hi! When I tried to upgrade the provider in my setup with already created resources it seems to work. Are you starting with a clean (empty state?) Sincerely Mikael

jrhouston commented 4 years ago

Hi @gkocur can you add a gist with the full debug log of your run?

evgmoskalenko commented 4 years ago

Hi,

I faced the same problem when my terraform CI automatically updated the terraform provider version. But on my agent installed helm version: 3.2.4. In the Changelog release helm should be 3.3.4. I will try to update on my CI/CD agent.

Error: rpc error: code = Unavailable desc = transport is closing

Error: rpc error: code = Canceled desc = context canceled

panic: runtime error: invalid memory address or nil pointer dereference
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0xa8 pc=0x17c6522]
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: 
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: goroutine 39 [running]:
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: k8s.io/cli-runtime/pkg/genericclioptions.(*ConfigFlags).ToRawKubeConfigLoader(0x0, 0x5fcb23d81f2c9ebf, 0x1)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go:136 +0x22
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/cli.(*EnvSettings).Namespace(0xc0005c2aa0, 0xc000445bf0, 0x20b3ae7)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/cli/environment.go:131 +0x2f
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/cli.(*EnvSettings).EnvVars(0xc0005c2aa0, 0x160)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/cli/environment.go:116 +0x553
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/plugin.SetupPluginEnv(0xc0005c2aa0, 0xc000b8992e, 0x2, 0xc00014e400, 0x30)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/plugin/plugin.go:260 +0x40
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/getter.(*pluginGetter).Get(0xc000a5a6c0, 0xc000612340, 0x40, 0xc000b14ff0, 0x4, 0x6, 0xc000a5a6c0, 0x0, 0x0)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/getter/plugingetter.go:73 +0x290
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/downloader.(*ChartDownloader).DownloadTo(0xc0006cf398, 0xc00014e200, 0x31, 0xc0003dcfa0, 0x5, 0xc0005cd200, 0x21, 0x1, 0x0, 0x1d036c0, ...)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/downloader/chart_downloader.go:97 +0x13b
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart(0xc000154ea0, 0xc00014e200, 0x31, 0xc0005c2aa0, 0x0, 0x0, 0xc0003dcfa0, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/action/install.go:669 +0x4bb
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-provider-helm/helm.getChart(0x24a59e0, 0xc000af9880, 0xc000445620, 0xc00014e200, 0x31, 0xc000154ea0, 0x0, 0x0, 0x0, 0x0, ...)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/helm/resource_release.go:799 +0xdc
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-provider-helm/helm.resourceDiff(0xc000af9880, 0x1e887a0, 0xc000445620, 0xc000b11b00, 0xc000af9880)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/helm/resource_release.go:663 +0x25c
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.schemaMap.Diff(0xc000282a50, 0xc000793cc0, 0xc000a56b10, 0x2187160, 0x1e887a0, 0xc000445620, 0x0, 0x24f9940, 0xffffffffffffffff, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/schema.go:518 +0xac2
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).simpleDiff(0xc000154900, 0xc000793cc0, 0xc000a56b10, 0x1e887a0, 0xc000445620, 0xc000a56b01, 0x0, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:356 +0x85
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).SimpleDiff(0xc00017ea80, 0xc0007d9a28, 0xc000793cc0, 0xc000a56b10, 0xc0006493e0, 0xc000a56b10, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:321 +0x99
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).PlanResourceChange(0xc0000d4890, 0x24f8340, 0xc000649380, 0xc000775420, 0xc0000d4890, 0x0, 0xc0007c8b30)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:633 +0x793
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x1ffb4e0, 0xc0000d4890, 0x24f8340, 0xc000649380, 0xc000790240, 0x0, 0x24f8340, 0xc000649380, 0xc000a06000, 0x7fb1)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3287 +0x217
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004df080, 0x2520fa0, 0xc0001a6180, 0xc0002dcc00, 0xc000173e00, 0x35809a8, 0x0, 0x0, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1024 +0x4f4
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: google.golang.org/grpc.(*Server).handleStream(0xc0004df080, 0x2520fa0, 0xc0001a6180, 0xc0002dcc00, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1313 +0xd97
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004ae750, 0xc0004df080, 0x2520fa0, 0xc0001a6180, 0xc0002dcc00)
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:722 +0xbb
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:720 +0xa1
2020/10/01 12:52:25 [ERROR] module.prometheus: eval: *terraform.EvalDiff, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:52:25 [ERROR] module.prometheus: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:52:25 [TRACE] [walkPlan] Exiting eval tree: module.prometheus.helm_release.prometheus[0]
2020/10/01 12:52:25 [TRACE] vertex "module.prometheus.helm_release.prometheus[0]": visit complete
2020/10/01 12:52:25 [TRACE] vertex "module.prometheus.helm_release.prometheus": dynamic subgraph encountered errors
2020/10/01 12:52:25 [TRACE] vertex "module.prometheus.helm_release.prometheus": visit complete
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "module.prometheus.output.status_prometheus" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "helm_release.alb_ingress_controller" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.alb_ingress_controller_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "helm_release.k8s_spot_termination_handler" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.k8s_spot_termination_handler_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "helm_release.external_dns" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.external_dns_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.prometheus_operator_status" errored, so skipping
2020-10-01T12:52:25.255Z [DEBUG] plugin: plugin process exited: path=/godata/pipelines/***/***/***/.terraform/plugins/linux_amd64/terraform-provider-helm_v1.3.1_x4 pid=86063 error="exit status 2"
2020/10/01 12:52:25 [TRACE] Re-validating config for "module.argocd.helm_release.argocd[0]"
2020/10/01 12:52:25 [TRACE] GRPCProvider: ValidateResourceTypeConfig
2020/10/01 12:52:25 [ERROR] module.argocd: eval: *terraform.EvalDiff, err: rpc error: code = Canceled desc = context canceled
2020/10/01 12:52:25 [ERROR] module.argocd: eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2020/10/01 12:52:25 [TRACE] [walkPlan] Exiting eval tree: module.argocd.helm_release.argocd[0]
2020/10/01 12:52:25 [TRACE] vertex "module.argocd.helm_release.argocd[0]": visit complete
2020/10/01 12:52:25 [TRACE] vertex "module.argocd.helm_release.argocd": dynamic subgraph encountered errors
2020/10/01 12:52:25 [TRACE] vertex "module.argocd.helm_release.argocd": visit complete
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "module.argocd.output.status_argocd" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.argocd_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "provider.helm (close)" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/10/01 12:52:25 [INFO] backend/local: plan operation completed
2020-10-01T12:52:25.364Z [DEBUG] plugin: plugin exited

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

aareet commented 4 years ago

Thanks @evgmoskalenko, @gkocur do you have the same issue after upgrading your helm version?

jrhouston commented 4 years ago

@evgmoskalenko @gkocur Are either of you using a helm plugin for your chart repository? I've managed to reproduce this crash with the google cloud storage plugin.

evgmoskalenko commented 4 years ago

I use only these plugins:

gkocur commented 4 years ago

Sorry for late reply. Yes, I use gcs plugin with helm. Here is gist with full crush.log: https://gist.github.com/gkocur/7d9b2e982ec068b73b171c03ada7b3c8

jrhouston commented 4 years ago

OK I figured out what's going on here.

When we upgraded to the latest helm package we brought in this change. This code relies upon the private config field of EnvSettings being set. When we initialize the provider we just initialize an EnvSettings struct by hand instead of using the New() function in the helm/cli package, which is what sets the config field. This code only gets hit when helm has to setup the environment for a plugin, because it calls EnvVars() on the EnvSettings struct, which in turn calls the above code.

In the commit I linked above you can see initializations of the EnvSettings struct were changed to use the New() function so we we should do the same. PR incoming. 🚀

bergbrains commented 4 years ago

@evgmoskalenko @gkocur Are either of you using a helm plugin for your chart repository? I've managed to reproduce this crash with the google cloud storage plugin.

I hit this over the weekend, and I'm using the S3 Helm plugin.

ghost commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-helm