Multiple helm_release resources concurrency issue (fatal error: concurrent map read and map write)

krzysztof-miemiec commented 4 years ago

TL;DR: I created my own wrapper module (see below) for Helm provider's helm_release resource. I use that wrapper module multiple times in my main.tf to roll out complete stack for my cluster (see below). I experience random crashes during terraform plan and apply:

Error: rpc error: code = Unavailable desc = transport is closing
Error: rpc error: code = Unavailable desc = transport is closing
Error: rpc error: code = Unavailable desc = transport is closing

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version and Provider Version

Terraform v0.12.24 and v0.12.25 (always hangs)
terragrunt version v0.23.10 and v0.23.18
Provider Version
terraform-provider-helm_v1.2.1_x4

Affected Resource(s)

helm_release

Terraform Configuration Files

kubernetes-services-copy.zip

Debug Output

Stack Trace

fatal error: concurrent map read and map write

goroutine 31 [running]:
runtime.throw(0x2c94295, 0x21)
    /opt/goenv/versions/1.13.7/src/runtime/panic.go:774 +0x72 fp=0xc000aa7020 sp=0xc000aa6ff0 pc=0x102e112
runtime.mapaccess2_faststr(0x2957ac0, 0xc0002aa360, 0xc000432700, 0x1b, 0x1, 0xc000432700)
    /opt/goenv/versions/1.13.7/src/runtime/map_faststr.go:116 +0x48f fp=0xc000aa7090 sp=0xc000aa7020 pc=0x1012bdf
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*DiffFieldReader).ReadField(0xc0002aa300, 0xc0006d2720, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5555555555555555, ...)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/field_reader_diff.go:51 +0x10d fp=0xc000aa71d0 sp=0xc000aa7090 pc=0x197623d
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*MultiLevelFieldReader).ReadFieldMerge(0xc000b05940, 0xc0006d2720, 0x3, 0x3, 0x2c5b89f, 0x3, 0x0, 0x0, 0x0, 0x0, ...)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/field_reader_multi.go:45 +0x1d8 fp=0xc000aa72e0 sp=0xc000aa71d0 pc=0x19796b8
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*ResourceData).get(0xc000116540, 0xc0006d2720, 0x3, 0x3, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource_data.go:537 +0x2f8 fp=0xc000aa73c8 sp=0xc000aa72e0 pc=0x1986128
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*ResourceData).getRaw(0xc000116540, 0xc0004326e0, 0x1b, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource_data.go:128 +0x75 fp=0xc000aa7430 sp=0xc000aa73c8 pc=0x19837c5
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*ResourceData).GetOk(0xc000116540, 0xc0004326e0, 0x1b, 0x2c6c3d0, 0xe, 0xc0004326e0)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource_data.go:94 +0x5f fp=0xc000aa74e0 sp=0xc000aa7430 pc=0x198351f
github.com/terraform-providers/terraform-provider-helm/helm.k8sGetOk(0xc000116540, 0x2c6c3d0, 0xe, 0x24, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/helm/provider.go:268 +0x93 fp=0xc000aa7588 sp=0xc000aa74e0 pc=0x27709b3
github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).toRawKubeConfigLoader(0xc0006d2060, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:86 +0xd38 fp=0xc000aa7718 sp=0xc000aa7588 pc=0x277bc68
github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).ToRawKubeConfigLoader(0xc0006d2060, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:68 +0xab fp=0xc000aa7780 sp=0xc000aa7718 pc=0x277aecb
github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).ToRESTConfig(0xc0006d2060, 0x0, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:31 +0x2b fp=0xc000aa77b0 sp=0xc000aa7780 pc=0x277ab1b
k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).ToRESTConfig(...)
    github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/kubectl/pkg/cmd/util/factory_client_access.go:63
k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).KubernetesClientSet(0xc0006d2090, 0x0, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/kubectl/pkg/cmd/util/factory_client_access.go:79 +0x38 fp=0xc000aa77e0 sp=0xc000aa77b0 pc=0x25167e8
helm.sh/helm/v3/pkg/kube.(*Client).IsReachable(0xc0006d20c0, 0x1983656, 0x28d1360)
    github.com/terraform-providers/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/kube/client.go:91 +0x37 fp=0xc000aa7840 sp=0xc000aa77e0 pc=0x2518be7
helm.sh/helm/v3/pkg/action.(*Get).Run(0xc000aa78d0, 0xc000744960, 0x12, 0x28d1360, 0x28d1360, 0x3012010)
    github.com/terraform-providers/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/action/get.go:41 +0x3b fp=0xc000aa7888 sp=0xc000aa7840 pc=0x271acab
github.com/terraform-providers/terraform-provider-helm/helm.getRelease(0xc000a192c0, 0xc000744960, 0x12, 0x28d1360, 0xc0006ced50, 0x1)
    github.com/terraform-providers/terraform-provider-helm/helm/resource_release.go:894 +0x64 fp=0xc000aa7900 sp=0xc000aa7888 pc=0x2778fe4
github.com/terraform-providers/terraform-provider-helm/helm.resourceReleaseExists(0xc0002aca10, 0x2a55f40, 0xc0002aa2a0, 0xc0002aca10, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/helm/resource_release.go:730 +0x10b fp=0xc000aa7958 sp=0xc000aa7900 pc=0x277748b
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000596c60, 0xc000b685a0, 0x2a55f40, 0xc0002aa2a0, 0xc000146960, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:445 +0x24d fp=0xc000aa79c8 sp=0xc000aa7958 pc=0x19812dd
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc00000ebc8, 0x3098960, 0xc000573020, 0xc00013c720, 0xc00000ebc8, 0xc000573020, 0xc0006b5b30)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:525 +0x3d8 fp=0xc000aa7ad0 sp=0xc000aa79c8 pc=0x19a0b28
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x2bc4c00, 0xc00000ebc8, 0x3098960, 0xc000573020, 0xc00013c6c0, 0x0, 0x3098960, 0xc000573020, 0xc000684580, 0x54a)
    github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217 fp=0xc000aa7b40 sp=0xc000aa7ad0 pc=0x18b2787
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00053fb00, 0x30c10e0, 0xc000532780, 0xc00013a200, 0xc0003063c0, 0x4108710, 0x0, 0x0, 0x0)
    github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1024 +0x4f4 fp=0xc000aa7e18 sp=0xc000aa7b40 pc=0x147ad24
google.golang.org/grpc.(*Server).handleStream(0xc00053fb00, 0x30c10e0, 0xc000532780, 0xc00013a200, 0x0)
    github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1313 +0xd97 fp=0xc000aa7f48 sp=0xc000aa7e18 pc=0x147ea47
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004ba890, 0xc00053fb00, 0x30c10e0, 0xc000532780, 0xc00013a200)
    github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:722 +0xbb fp=0xc000aa7fb8 sp=0xc000aa7f48 pc=0x148be9b
runtime.goexit()
    /opt/goenv/versions/1.13.7/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000aa7fc0 sp=0xc000aa7fb8 pc=0x105b761
created by google.golang.org/grpc.(*Server).serveStreams.func1
    github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:720 +0xa1

Full Stack Trace

https://gist.github.com/krzysztof-miemiec/2d44d187a75a2f106ccd9c71fbf67883

Expected Behavior

Helm properly plans and applies changes to multiple releases.

Actual Behavior

Random crashes happen during plan phase (I provided stack trace for that) OR sometimes everything gets stuck during deployment (no logs 😞).

Steps to Reproduce

Have Kubernetes cluster up and running, be able to freely access it via configured context
Download zip file attached
terraform plan

Important Factoids

I never debugged or contributed to any Go tool, so these are mostly guesses based on my knowledge of Terraform:

It happens randomly, I guess it occurs more often when you define more helm_release resources (it probably won't occur when multiple instances of helm provider are not running at the same time 🤔)
I noticed that it takes ages to refresh state of Helm resources, especially ones that are using CRDs (not the ones that define them). Using -parallelism=1 flag helps a bit with performance and with this flag I didn't encounter crashes or infinite wait on plan phase.
Infinite wait happens sometimes during terraform apply even with wait = false.
If I understand stack trace correctly, it is a concurrency issue with Helm provider or Terraform itself

References

https://github.com/terraform-providers/terraform-provider-helm/issues/486 - same user-facing error message, but I guess the cause is different
https://github.com/hashicorp/terraform/issues/24589 - similar root cause, issue is closed, PR's merged, but it's not released yet

ishioni commented 4 years ago

@krzysztof-miemiec looks like we hit the same bug at the same time ;) https://github.com/terraform-providers/terraform-provider-helm/issues/493

stevie- commented 4 years ago

I also encounter this error.

It seems that helm provider can not talk to k8s api. When i set my local kubeconfig to working k8s context, the terraform plan works. When I want to rely on the kubernetes provider only and set KUBE_LOAD_CONFIG_FILE=false then I see the issue above.

stevie- commented 4 years ago

I can solve this issue when I provide the kubernetes configuration block to the helm provider https://www.terraform.io/docs/providers/helm/index.html#kubernetes

Example with tf module EKS https://registry.terraform.io/modules/terraform-aws-modules/eks/aws:

provider "helm" {
  kubernetes {
    load_config_file       = false
    host                   = element(concat(data.aws_eks_cluster.cluster[*].endpoint, list("")), 0)
    cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.cluster[*].certificate_authority.0.data, list("")), 0))
    token                  = element(concat(data.aws_eks_cluster_auth.cluster[*].token, list("")), 0)
  }
}

krzysztof-miemiec commented 4 years ago

@stevie- did your helm provider also crash randomly or was it constant? I believe that Unavailable desc = transport is closing is a generic error saying "something bad happened in provider" and our root causes may be different. I'll adjust the name of this issue t be more descriptive 🤔

stevie- commented 4 years ago

This error is/was constant.

krzysztof-miemiec commented 4 years ago

I guess we are/were having different errors then 😉 I think I "fixed" mine by setting wait parameter back to false, but I'm not 100% sure if thats the cause. I'll add more info for maintainers to this issue when I encounter trouble.

ErikLundJensen commented 4 years ago

The error is not constant. It fails 50-60% of the time. The success rate might be related to the performance of the systems.

dak1n1 commented 4 years ago

Today Terraform version 0.12.25 was released, which fixed a concurrency bug. https://github.com/hashicorp/terraform/blob/v0.12.25/CHANGELOG.md There's a chance that could have fixed this issue, but we still need to test using the reproducer.

krzysztof-miemiec commented 4 years ago

I just checked after updating to the latest Terraform build available - no luck. Out of 3 tries - it failed 3 times. From logs with trace level, I see that helm successfully fetches all necessary information, but in the end it looks like it falls into some kind of deadlock:

2020-05-16T18:02:24.540+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [DEBUG] Using custom current context: "{{redacted}}-beta"
2020-05-16T18:02:24.540+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [DEBUG] Using overidden context: api.Context{LocationOfOrigin:"", Cluster:"", AuthInfo:"", Namespace:"", Extensions:map[string]runtime.Object(nil)}
2020-05-16T18:02:24.540+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [INFO] Successfully initialized config
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [DEBUG] got release &{basic-resources 0xc000bbd000 0xc0001eb400 map[] ---
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: # Source: basic-resources/templates/cert-issuers.yaml
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: apiVersion: cert-manager.io/v1alpha2
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: kind: ClusterIssuer
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: metadata:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   name: letsencrypt-staging
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: spec:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   acme:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     email: devops@{{redacted}}
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     server: https://acme-staging-v02.api.letsencrypt.org/directory
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     privateKeySecretRef:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:       name: letsencrypt-staging-issuer-key
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     solvers:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     - dns01:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:         route53:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:           region: eu-central-1
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: ---
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: # Source: basic-resources/templates/cert-issuers.yaml
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: apiVersion: cert-manager.io/v1alpha2
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: kind: ClusterIssuer
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: metadata:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   name: letsencrypt
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: spec:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   acme:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     email: devops@{{redacted}}
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     server: https://acme-v02.api.letsencrypt.org/directory
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     privateKeySecretRef:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:       name: letsencrypt-issuer-key
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     solvers:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:     - dns01:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:         route53:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:           region: eu-central-1
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: ---
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: # Source: basic-resources/templates/priority-classes.yaml
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: apiVersion: scheduling.k8s.io/v1
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: kind: PriorityClass
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: metadata:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   name: required
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: value: 100
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: globalDefault: false
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: description: Pods with this priority are critical for cluster itself (i.e. autoscaler, dns, monitoring).
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: ---
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: # Source: basic-resources/templates/priority-classes.yaml
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: apiVersion: scheduling.k8s.io/v1
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: kind: PriorityClass
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: metadata:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   name: high
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: value: 50
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: globalDefault: false
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: description: Pods with this priority are important for multiple services (i.e. database, queue).
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: ---
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: # Source: basic-resources/templates/priority-classes.yaml
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: apiVersion: scheduling.k8s.io/v1
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: kind: PriorityClass
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: metadata:
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:   name: standard
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: value: 10
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: globalDefault: true
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: description: Pods with this priority can just die and that is not an issue for us.
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  [] 2 default}
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [DEBUG] Using custom current context: "{{redacted}}-beta"
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [DEBUG] Using overidden context: api.Context{LocationOfOrigin:"", Cluster:"", AuthInfo:"", Namespace:"", Extensions:map[string]runtime.Object(nil)}
2020-05-16T18:02:24.603+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: 2020/05/16 18:02:24 [INFO] Successfully initialized config
2020/05/16 18:02:28 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:02:28 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:33 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:02:33 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:38 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:02:38 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:43 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:43 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:02:48 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:48 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:02:53 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:53 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:02:58 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:02:58 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:03 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:03 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:08 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:08 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:13 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:13 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:18 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:18 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:23 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:23 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:28 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:28 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:33 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:33 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:38 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:38 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:43 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:43 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:48 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:48 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:53 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:53 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:03:58 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:03:58 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:03 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:03 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:08 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:08 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:13 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:13 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:19 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:19 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:24 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:24 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:29 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:29 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:34 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:34 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:39 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:39 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:44 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:44 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:49 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:49 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:54 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:04:54 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:59 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:04:59 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:04 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:04 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:09 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:09 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:14 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:14 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:19 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:19 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:24 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:24 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:29 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:29 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:34 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:34 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:39 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:39 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:44 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:44 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:49 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:49 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:54 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:54 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:05:59 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:05:59 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:04 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:04 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:09 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:09 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:14 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:14 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:19 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:19 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:24 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:24 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:29 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:29 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:34 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:34 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:39 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:39 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:44 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:44 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:49 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:49 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:54 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:54 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:06:59 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:06:59 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:04 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:04 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:09 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:09 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:14 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:14 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:19 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:19 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:24 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:24 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:29 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:29 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:34 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:34 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:39 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:39 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:44 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:44 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:49 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:49 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:54 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:07:54 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:59 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:07:59 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:04 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:04 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:09 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:09 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:14 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:14 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:19 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:19 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:24 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:24 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:29 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:29 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:34 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:34 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:39 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:39 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:44 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:44 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:49 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:49 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:54 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:54 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:08:59 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:08:59 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:09:04 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:09:04 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020-05-16T18:09:07.339+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: plugin received interrupt signal, ignoring: count=1 timestamp=2020-05-16T18:09:07.339+0200
2020-05-16T18:09:07.511+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: plugin received interrupt signal, ignoring: count=2 timestamp=2020-05-16T18:09:07.510+0200
2020-05-16T18:09:08.566+0200 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: plugin received interrupt signal, ignoring: count=3 timestamp=2020-05-16T18:09:08.566+0200
2020/05/16 18:09:09 [TRACE] dag/walk: vertex "root" is waiting for "provider.helm (close)"
2020/05/16 18:09:09 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "helm_release.basic_resources"
2020/05/16 18:09:12 [ERROR] : eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/05/16 18:09:12 [ERROR] : eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/05/16 18:09:12 [TRACE] [walkRefresh] Exiting eval tree: helm_release.basic_resources
2020/05/16 18:09:12 [TRACE] vertex "helm_release.basic_resources": visit complete
2020/05/16 18:09:12 [TRACE] vertex "helm_release.basic_resources": dynamic subgraph encountered errors
2020/05/16 18:09:12 [TRACE] vertex "helm_release.basic_resources": visit complete
2020/05/16 18:09:12 [TRACE] dag/walk: upstream of "provider.helm (close)" errored, so skipping
2020/05/16 18:09:12 [TRACE] dag/walk: upstream of "root" errored, so skipping

What's more, now -parallelism=1 doesn't work and Helm provider seems to hang before returning.

krzysztof-miemiec commented 4 years ago

I think that I dealt with concurrent map read/write by tinkering with provider's code (I'm creating clientcmd.ClientConfig directly in GetHelmConfiguration instead of delaying it until ToRawKubeConfigLoader() is called). I can make a PR, however I keep getting stuck while Kubernetes client is being loaded (the weird part for me is that it doesn't look like something Helm-related - it gets stuck on klog.Warningf, os.Stdout.write to be precise). See more details here: https://github.com/terraform-providers/terraform-provider-helm/issues/498#issuecomment-630120960

danylohetmantsev commented 4 years ago

i've tested helm-provider again with native kubernetes 1.16.9

so basically it works 15 out of 15 times with token specified in kubernetes spec.

other ways (like exec command or direct kubeconfig with or without exec path - causing the problem described problem).

krzysztof-miemiec commented 4 years ago

From user's perspective: I switched back to helm provider v1.0.0 and it works fine (at least with force=false and wait=false flags).

Also: Not knowing Golang much, I made these changes to overcome concurrent map read/write: https://github.com/krzysztof-miemiec/terraform-provider-helm/pull/1/files. Basically I don't touch metadata outside helm initialization function - currently Helm provider reads stuff from metadata on first (delayed) call to Kubernetes client config.

This change only fixed one issue for me, the actual hang happens somewhere deep in K8s sources. I guess that's caused by klogs stderr buffer being full, but I've got no idea how to patch that.

danylohetmantsev commented 4 years ago

@krzysztof-miemiec 1.0.0 also works for me, it's appeared right after update to 1.1.0 and upper.

im using 1.0.0 in production atm (without any additional flags)

Sytten commented 4 years ago

I believe the issue is a bad concurrency access to a Map like some people pointed above. I also downgraded to v1.0.0 for now and the problem doesn't appear. Can someone from the maintenance team address this? In the debug I can see:

module.service_xxxxxxx.module.api.module.configuration.helm_release.secret: Refreshing state... [id=xxxxxxx-api-config]
2020-06-01T13:32:41.361-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: fatal error: concurrent map read and map write
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 83 [running]:
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: runtime.throw(0x2c94295, 0x21)
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/runtime/panic.go:774 +0x72 fp=0xc00073f020 sp=0xc00073eff0 pc=0x102e112
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: runtime.mapaccess2_faststr(0x2957ac0, 0xc00028b050, 0xc00068c120, 0x15, 0x1, 0xc00068c120)
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/runtime/map_faststr.go:116 +0x48f fp=0xc00073f090 sp=0xc00073f020 pc=0x1012bdf
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*DiffFieldReader).ReadField(0xc00028aff0, 0xc000826de0, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5555555555555555, ...)
2020-06-01T13:32:41.364-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/field_reader_diff.go:51 +0x10d fp=0xc00073f1d0 sp=0xc00073f090 pc=0x197623d
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*MultiLevelFieldReader).ReadFieldMerge(0xc000ad5260, 0xc000826de0, 0x3, 0x3, 0x2c5b89f, 0x3, 0x0, 0x0, 0x0, 0x0, ...)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/field_reader_multi.go:45 +0x1d8 fp=0xc00073f2e0 sp=0xc00073f1d0 pc=0x19796b8
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*ResourceData).get(0xc0002c24d0, 0xc000826de0, 0x3, 0x3, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource_data.go:537 +0x2f8 fp=0xc00073f3c8 sp=0xc00073f2e0 pc=0x1986128
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*ResourceData).getRaw(0xc0002c24d0, 0xc00068c0a0, 0x15, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource_data.go:128 +0x75 fp=0xc00073f430 sp=0xc00073f3c8 pc=0x19837c5
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*ResourceData).GetOk(0xc0002c24d0, 0xc00068c0a0, 0x15, 0x2c6299d, 0x8, 0xc00068c0a0)
rm-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource_data.go:94 +0x5f fp=0xc00073f4e0 sp=0xc00073f430 pc=0x198351f
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.k8sGetOk(0xc0002c24d0, 0x2c6299d, 0x8, 0x28d1360, 0x3012fc0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/provider.go:268 +0x93 fp=0xc00073f588 sp=0xc00073f4e0 pc=0x27709b3
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).toRawKubeConfigLoader(0xc0007a1ad0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:135 +0x327 fp=0xc00073f718 sp=0xc00073f588 pc=0x277b257
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).ToRawKubeConfigLoader(0xc0007a1ad0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:68 +0xab fp=0xc00073f780 sp=0xc00073f718 pc=0x277aecb
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).ToRESTConfig(0xc0007a1ad0, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:31 +0x2b fp=0xc00073f7b0 sp=0xc00073f780 pc=0x277ab1b
m-provider-helm_v1.2.1_x4: k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).ToRESTConfig(...)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/kubectl/pkg/cmd/util/factory_client_access.go:63
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).KubernetesClientSet(0xc0007a1b00, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/kubectl/pkg/cmd/util/factory_client_access.go:79 +0x38 fp=0xc00073f7e0 sp=0xc00073f7b0 pc=0x25167e8
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: helm.sh/helm/v3/pkg/kube.(*Client).IsReachable(0xc0007a1b30, 0x1983656, 0x28d1360)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/kube/client.go:91 +0x37 fp=0xc00073f840 sp=0xc00073f7e0 pc=0x2518be7
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: helm.sh/helm/v3/pkg/action.(*Get).Run(0xc00073f8d0, 0xc00058e280, 0x17, 0x28d1360, 0x28d1360, 0x3012010)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/action/get.go:41 +0x3b fp=0xc00073f888 sp=0xc00073f840 pc=0x271acab
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.getRelease(0xc0009c87c0, 0xc00058e280, 0x17, 0x28d1360, 0xc0008130c0, 0x1)
terraform-providers/terraform-provider-helm/helm/resource_release.go:894 +0x64 fp=0xc00073f900 sp=0xc00073f888 pc=0x2778fe4
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.resourceReleaseExists(0xc0001824d0, 0x2a55f40, 0xc00028af90, 0xc0001824d0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/resource_release.go:730 +0x10b fp=0xc00073f958 sp=0xc00073f900 pc=0x277748b
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0005a6d80, 0xc000596640, 0x2a55f40, 0xc00028af90, 0xc0000caeb0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:445 +0x24d fp=0xc00073f9c8 sp=0xc00073f958 pc=0x19812dd
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0000cabc8, 0x3098960, 0xc00065e180, 0xc0000cc7e0, 0xc0000cabc8, 0xc00065e180, 0xc000987b30)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:525 +0x3d8 fp=0xc00073fad0 sp=0xc00073f9c8 pc=0x19a0b28
x40b)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217 fp=0xc00073fb40 sp=0xc00073fad0 pc=0x18b2787
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000674000, 0xc000686930, 0x4108710, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1024 +0x4f4 fp=0xc00073fe18 sp=0xc00073fb40 pc=0x147ad24
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).handleStream(0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000674000, 0x0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1313 +0xd97 fp=0xc00073ff48 sp=0xc00073fe18 pc=0x147ea47
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000682b40, 0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000674000)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:722 +0xbb fp=0xc00073ffb8 sp=0xc00073ff48 pc=0x148be9b
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: runtime.goexit()
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00073ffc0 sp=0xc00073ffb8 pc=0x105b761
20-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:720 +0xa1
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 1 [chan receive]:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.Serve(0xc000a2ded0)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/server.go:354 +0xc17
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/plugin.Serve(0xc000755f40)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/plugin/serve.go:65 +0x1d5
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: main.main()
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/main.go:9 +0x48
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 5 [syscall]:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: os/signal.signal_recv(0x0)

2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: os/signal.loop()
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/os/signal/signal_unix.go:23 +0x22
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by os/signal.init.0
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/os/signal/signal_unix.go:29 +0x41
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 34 [select]:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: go.opencensus.io/stats/view.(*worker).start(0xc000214190)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by go.opencensus.io/stats/view.init.0
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 35 [chan receive]:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: k8s.io/klog.(*loggingT).flushDaemon(0x412d680)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/klog/klog.go:1010 +0x8b
t.0
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/klog/klog.go:411 +0xd6
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 11 [select]:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0xc000686750, 0x2d4f628, 0xc0002f4fb8, 0x25eee79)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:121 +0xb3
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc000214000)
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:411 +0x7a
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_server.go:80 +0x32b
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 21 [chan receive]:
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.Serve.func2(0xc0000cc420, 0x30d4480, 0xc0000a0300)
ent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/server.go:342 +0x94
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by github.com/hashicorp/go-plugin.Serve
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/server.go:339 +0xb8e
2020-06-01T13:32:41.365-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 22 [IO wait]:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.runtime_pollWait(0x4c71b68, 0x72, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/runtime/netpoll.go:184 +0x55
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.(*pollDesc).wait(0xc00052f418, 0x72, 0x0, 0x0, 0x2c5e7f5)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/internal/poll/fd_poll_runtime.go:87 +0x45
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.(*pollDesc).waitRead(...)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/internal/poll/fd_poll_runtime.go:92
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.(*FD).Accept(0xc00052f400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/internal/poll/fd_unix.go:384 +0x1f8
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: net.(*netFD).accept(0xc00052f400, 0xc00000e008, 0x0, 0x0)
_x4:    /opt/goenv/versions/1.13.7/src/net/fd_unix.go:238 +0x42
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: net.(*UnixListener).accept(0xc0005942a0, 0xc00000e008, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/net/unixsock_posix.go:162 +0x32
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: net.(*UnixListener).Accept(0xc0005942a0, 0xc000734dd8, 0x18, 0xc00053e900, 0x147898b)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/net/unixsock.go:260 +0x47
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).Serve(0xc000553380, 0x307faa0, 0xc000590880, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:597 +0x22e
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.(*GRPCServer).Serve(0xc0004ae280, 0x307faa0, 0xc000590880)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_server.go:131 +0x8b
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by github.com/hashicorp/go-plugin.Serve
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/server.go:353 +0xbfd
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 40 [select]:
 plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0002140a0, 0x1, 0x0, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x122
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000582240, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e3
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0004f2300)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xcb
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by google.golang.org/grpc/internal/transport.newHTTP2Server
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x105d
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 41 [select]:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004f2300)
.2.1_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x26b
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by google.golang.org/grpc/internal/transport.newHTTP2Server
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x1082
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 42 [IO wait]:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.runtime_pollWait(0x4c71a98, 0x72, 0xffffffffffffffff)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/runtime/netpoll.go:184 +0x55
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.(*pollDesc).wait(0xc0006b0318, 0x72, 0xf00, 0xf32, 0xffffffffffffffff)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/internal/poll/fd_poll_runtime.go:87 +0x45
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.(*pollDesc).waitRead(...)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/internal/poll/fd_poll_runtime.go:92
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: internal/poll.(*FD).Read(0xc0006b0300, 0xc00032f000, 0xf32, 0xf32, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/internal/poll/fd_unix.go:169 +0x22b
d(0xc0006b0300, 0xc00032f000, 0xf32, 0xf32, 0x203000, 0x0, 0xf2d)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/net/fd_unix.go:202 +0x4f
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: net.(*conn).Read(0xc00000e008, 0xc00032f000, 0xf32, 0xf32, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/net/net.go:184 +0x68
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: crypto/tls.(*atLeastReader).Read(0xc000a11500, 0xc00032f000, 0xf32, 0xf32, 0x27, 0x30458c0, 0xc000745968)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/crypto/tls/conn.go:780 +0x60
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: bytes.(*Buffer).ReadFrom(0xc0005a9ad8, 0x3045600, 0xc000a11500, 0x1009c95, 0x29c7620, 0x2bf9f20)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/bytes/buffer.go:204 +0xb4
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: crypto/tls.(*Conn).readFromUntil(0xc0005a9880, 0x4c7eb98, 0xc00000e008, 0x5, 0xc00000e008, 0x11)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/crypto/tls/conn.go:802 +0xec
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: crypto/tls.(*Conn).readRecordOrCCS(0xc0005a9880, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/crypto/tls/conn.go:609 +0x124
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: crypto/tls.(*Conn).readRecord(...)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/crypto/tls/conn.go:577
x4: crypto/tls.(*Conn).Read(0xc0005a9880, 0xc00096e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/crypto/tls/conn.go:1255 +0x161
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: bufio.(*Reader).Read(0xc0000ccba0, 0xc00033a2d8, 0x9, 0x9, 0x1084e0a, 0x8000000000000000, 0x12d6ce9b)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/bufio/bufio.go:226 +0x26a
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: io.ReadAtLeast(0x3045360, 0xc0000ccba0, 0xc00033a2d8, 0x9, 0x9, 0x9, 0x13a0a05, 0xc000214108, 0xc000745e30)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/io/io.go:310 +0x87
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: io.ReadFull(...)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/io/io.go:329
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: golang.org/x/net/http2.readFrameHeader(0xc00033a2d8, 0x9, 0x9, 0x3045360, 0xc0000ccba0, 0x0, 0x0, 0x7070e0910100402, 0x9d3db3d8)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/golang.org/x/net/http2/frame.go:237 +0x87
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: golang.org/x/net/http2.(*Framer).ReadFrame(0xc00033a2a0, 0xc0005e4020, 0x412d0a0, 0x0, 0x0)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
tp2Server).HandleStreams(0xc0004f2300, 0xc00046a540, 0x2d4de38)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0xa1
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).serveStreams(0xc000553380, 0x30c10e0, 0xc0004f2300)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:718 +0xf3
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000553380, 0x30c10e0, 0xc0004f2300)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:679 +0x3f
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by google.golang.org/grpc.(*Server).handleRawConn
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:678 +0x562
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 25 [select]:
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0000e43c0, 0xc000a76710, 0x5, 0x5, 0x0, 0x0, 0x2d4f968)
-helm/vendor/google.golang.org/grpc/internal/transport/transport.go:174 +0xc7
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0000e43c0, 0xc000a76710, 0x5, 0x5, 0xc0007339d8, 0xc0000547c0, 0xc0007336b8)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/transport.go:168 +0x21a
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000529d70, 0xc000a76710, 0x5, 0x5, 0x4772ddf2848fe697, 0xc0007336e0, 0x13e265c)
2020-06-01T13:32:41.366-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x55
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: io.ReadAtLeast(0x3047fa0, 0xc000529d70, 0xc000a76710, 0x5, 0x5, 0x5, 0x2c, 0x2c, 0x86)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/io/io.go:310 +0x87
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: io.ReadFull(...)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/goenv/versions/1.13.7/src/io/io.go:329
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000336500, 0xc000a76710, 0x5, 0x5, 0x0, 0xc0007337d0, 0x101739e)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xcd
 plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*parser).recvMsg(0xc000a76700, 0x400000, 0x203000, 0x203000, 0x203000, 0x4c76890, 0x2, 0x10)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.recvAndDecompress(0xc000a76700, 0xc000336500, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0xc000528de0, 0x30, ...)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.recv(0xc000a76700, 0x476d160, 0x4154f60, 0xc000336500, 0x0, 0x0, 0x2b3b3a0, 0xc0000e45a0, 0x400000, 0x0, ...)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9b
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*serverStream).RecvMsg(0xc000998000, 0x2b3b3a0, 0xc0000e45a0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/stream.go:1474 +0x170
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamServer).Recv(0xc000997190, 0x2d4cd10, 0xc000300180, 0xc000686750)
-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/internal/plugin/grpc_broker.pb.go:184 +0x62
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.(*gRPCBrokerServer).StartStream(0xc000686750, 0x30bba40, 0xc000997190, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:84 +0x11d
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin/internal/plugin._GRPCBroker_StartStream_Handler(0x2a201e0, 0xc000686750, 0x30ad580, 0xc000998000, 0x4154f60, 0xc000336500)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/internal/plugin/grpc_broker.pb.go:165 +0xad
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).processStreamingRPC(0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000336500, 0xc0006867e0, 0x40fbc40, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1237 +0xb42
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).handleStream(0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000336500, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1317 +0xd30
unc1.1(0xc000682b40, 0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000336500)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:722 +0xbb
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:720 +0xa1
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 26 [select]:
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/go-plugin.(*gRPCBrokerServer).StartStream.func1(0xc000300180, 0xc000686750, 0x30bba40, 0xc000997190)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:70 +0x145
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by github.com/hashicorp/go-plugin.(*gRPCBrokerServer).StartStream
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:68 +0xf5
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: goroutine 29 [runnable]:
02c24d0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/provider.go:118 +0x812
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.k8sGetOk(0xc0002c24d0, 0x2c6577a, 0xa, 0x28d1360, 0x3012fd0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/provider.go:277 +0x19d
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).toRawKubeConfigLoader(0xc0007c08d0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:138 +0x39b
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).ToRawKubeConfigLoader(0xc0007c08d0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:68 +0xab
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.(*KubeConfig).ToRESTConfig(0xc0007c08d0, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/structure_kubeconfig.go:31 +0x2b
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).ToRESTConfig(...)
0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/kubectl/pkg/cmd/util/factory_client_access.go:63
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).KubernetesClientSet(0xc0007c0960, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/k8s.io/kubectl/pkg/cmd/util/factory_client_access.go:79 +0x38
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: helm.sh/helm/v3/pkg/kube.(*Client).IsReachable(0xc0007c0990, 0x1983656, 0x28d1360)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/kube/client.go:91 +0x37
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: helm.sh/helm/v3/pkg/action.(*Get).Run(0xc0001918d0, 0xc0006a9900, 0x18, 0x28d1360, 0x28d1360, 0x3012010)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/action/get.go:41 +0x3b
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/terraform-providers/terraform-provider-helm/helm.getRelease(0xc000777800, 0xc0006a9900, 0x18, 0x28d1360, 0xc000805af0, 0x1)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/resource_release.go:894 +0x64
56540, 0x2a55f40, 0xc00028af90, 0xc000456540, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/helm/resource_release.go:730 +0x10b
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0005a6d80, 0xc0000e51d0, 0x2a55f40, 0xc00028af90, 0xc000168ef0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:445 +0x24d
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0000cabc8, 0x3098960, 0xc000632e70, 0xc0008706c0, 0xc0000cabc8, 0xc000632e70, 0xc000735b30)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:525 +0x3d8
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x2bc4c00, 0xc0000cabc8, 0x3098960, 0xc000632e70, 0xc000870660, 0x0, 0x3098960, 0xc000632e70, 0xc0000fc900, 0x410)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217
30c10e0, 0xc0004f2300, 0xc000336400, 0xc000686930, 0x4108710, 0x0, 0x0, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1024 +0x4f4
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).handleStream(0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000336400, 0x0)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1313 +0xd97
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000682b40, 0xc000553380, 0x30c10e0, 0xc0004f2300, 0xc000336400)
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:722 +0xbb
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-06-01T13:32:41.367-0400 [DEBUG] plugin.terraform-provider-helm_v1.2.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:720 +0xa1

ajfriesen commented 4 years ago

I have encountered a similar problem. I have created a module that will deploy helm releases in a rancher namespace. (helm provider and rancher 2 provider)

At first, I did try to run this with 80-90 modules and did encounter the problem as well. So I did go down to 10 modules.

Deploying >= 10 modules will encounter in constant errors. (Did not test less than 10) Some of the helm releases where reported as deployed by the helm provider but in the "pending" state. When I checked in Kubernetes I could not see any resources regarding the specific helm release (Pods, deployments, etc), also helm ls did not show anything.

The namespaces were created successfully since this is done by the rancher2 provider.

When running with -parallelism=1 everything worked fine. -parallelism=2 does not work either.

Also running 2 parallel terraform apply -parallelism=1 with 2 states with 10 modules each does work fine. I think it is not an issue by Kubernetes itself. I think it is related to terraform or the helm provider.

Tested with terraform version

0.12.16
0.12.26 helm provider
1.0.0
1.2.2

Any idea of how I could provide more useful information to debug this?

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-helm