search

rancher / terraform-provider-rke

Terraform provider plugin for deploy kubernetes cluster by RKE(Rancher Kubernetes Engine)
Mozilla Public License 2.0
340 stars 151 forks source link

Upgrade to v1.1.5 from v1.1.4 is failing #264

Closed abhi1693 closed 3 years ago

abhi1693 commented 3 years ago

I'm only able to use v1.1.5 on a fresh install. But that is not possible for our production workload.

Here is my terraform code

resource "rke_cluster" "cluster" {
  depends_on = [var.vm_depends_on]

  dynamic "nodes" {
    for_each = [for node in var.cluster_nodes:{
      name       = node["name"]
      ip         = node["ip"]
      dns_record = node["dns_record"]
    }]
    content {
      address           = nodes.value.ip
      hostname_override = nodes.value.name
      user              = "ubuntu"
      role              = ["controlplane", "etcd", "worker"]
      ssh_key           = var.ssh_key
    }
  }

  addons = file("${path.module}/templates/addons.yaml")

  authentication {
    strategy = "x509"
  }
  cluster_name = "rancher-management"

  monitoring {
    provider = "metrics-server"
  }

  network {
    plugin = "canal"
    mtu    = 0
  }

  dns {
    provider = "coredns"
  }

  services {
    etcd {
      backup_config {
        enabled        = true
        interval_hours = 12
        retention      = 6
        safe_timestamp = false
      }
      creation   = "12h"
      extra_args = {
        election-timeout   = 5000
        heartbeat-interval = 500
      }
      gid        = 49716
      retention  = "72h"
      snapshot   = true
      uid        = 49716
    }

    kube_api {
      always_pull_images      = false
      audit_log {
        enabled = true
      }
      event_rate_limit {
        enabled = true
      }
      pod_security_policy     = true
      secrets_encryption_config {
        enabled = true
      }
      service_node_port_range = "30000-32767"
    }
    kube_controller {
      extra_args = {
        address                     = "127.0.0.1"
        feature-gates               = "RotateKubeletServerCertificate=true"
        profiling                   = false
        terminated-pod-gc-threshold = 1000
      }
    }
    kubelet {
      extra_args                   = {
        anonymous-auth                    = false
        event-qps                         = 0
        feature-gates                     = "RotateKubeletServerCertificate=true"
        make-iptables-util-chains         = true
        protect-kernel-defaults           = true
        read-only-port                    = 0
        streaming-connection-idle-timeout = "1800s"
      }
      fail_swap_on                 = false
      generate_serving_certificate = true
    }
    scheduler {
      extra_args = {
        address   = "127.0.0.1"
        profiling = false
      }
    }
  }
  ssh_agent_auth = false
  update_only    = var.rke_update_only

  upgrade_strategy {
    drain                        = false
    max_unavailable_controlplane = 1
    max_unavailable_worker       = "10%"
  }
}

Note: The addons.yaml is based on the hardening guide

Terraform version: 0.13.5

module.rke_cluster.rke_cluster.cluster: Refreshing state... [id=abc48260-dc05-4bfe-b77e-4ccfecd32dde]
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: 
Error: rpc error: code = Unavailable desc = transport is closing

panic: runtime error: invalid memory address or nil pointer dereference
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19aa5f1]
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: 
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: goroutine 23 [running]:
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/rancher/terraform-provider-rke/rke.flattenRKEClusterNodeUpgradeStrategy(0xc000ae2330, 0x1e30abf, 0x10, 0x1aa5c20)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/src/github.com/rancher/terraform-provider-rke/rke/structure_rke_cluster_node.go:33 +0x41
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/rancher/terraform-provider-rke/rke.flattenRKECluster(0xc000460f50, 0xc000396400, 0x0, 0x0)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/src/github.com/rancher/terraform-provider-rke/rke/structure_rke_cluster.go:181 +0x1220
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/rancher/terraform-provider-rke/rke.resourceRKEClusterRead(0xc000460f50, 0x1bb5b00, 0xc000604780, 0xc000460f50, 0x0)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/src/github.com/rancher/terraform-provider-rke/rke/resource_rke_cluster.go:120 +0x14b
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000adc750, 0xc000ebe0a0, 0x1bb5b00, 0xc000604780, 0xc0005aef10, 0x0, 0x0)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/helper/schema/resource.go:460 +0x119
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0005ae2c8, 0x2227820, 0xc000ac8f00, 0xc000130480, 0xc0005ae2c8, 0xc000ac8f00, 0xc000e8fb78)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/helper/plugin/grpc_provider.go:525 +0x3d8
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x1d8c940, 0xc0005ae2c8, 0x2227820, 0xc000ac8f00, 0xc000130420, 0x0, 0x2227820, 0xc000ac8f00, 0xc000e1e000, 0x45d47)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000642a80, 0x2241540, 0xc000643e00, 0xc0004f2100, 0xc000215440, 0x3119490, 0x0, 0x0, 0x0)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1024 +0x501
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: google.golang.org/grpc.(*Server).handleStream(0xc000642a80, 0x2241540, 0xc000643e00, 0xc0004f2100, 0x0)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1313 +0xd3d
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0005c3a30, 0xc000642a80, 0x2241540, 0xc000643e00, 0xc0004f2100)
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:722 +0xa1
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-11-11T18:25:42.392-0800 [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:720 +0xa1
2020-11-11T18:25:42.396-0800 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/rancher/rke/1.1.5/linux_amd64/terraform-provider-rke_v1.1.5 pid=103184 error="exit status 2"
2020/11/11 18:25:42 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/11/11 18:25:42 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/11/11 18:25:42 [TRACE] [walkRefresh] Exiting eval tree: module.rke_cluster.rke_cluster.cluster
2020-11-11T18:25:42.396-0800 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020/11/11 18:25:42 [TRACE] vertex "module.rke_cluster.rke_cluster.cluster": visit complete
abhi1693 commented 3 years ago

I was finally able to upgrade, but I had to remove upgrade_strategy in v1.1.4 and add again in v1.1.5.

xom4ek commented 3 years ago

Same issue with v1.1.5 provider:

panic: runtime error: invalid memory address or nil pointer dereference
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19aa5f1]
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: 
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: goroutine 35 [running]:
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/rancher/terraform-provider-rke/rke.flattenRKEClusterNodeUpgradeStrategy(0xc00096b980, 0x1e30abf, 0x10, 0x1aa5c20)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/src/github.com/rancher/terraform-provider-rke/rke/structure_rke_cluster_node.go:33 +0x41
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/rancher/terraform-provider-rke/rke.flattenRKECluster(0xc00019e620, 0xc00076e400, 0x0, 0x0)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/src/github.com/rancher/terraform-provider-rke/rke/structure_rke_cluster.go:181 +0x1220
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/rancher/terraform-provider-rke/rke.resourceRKEClusterRead(0xc00019e620, 0x1bb5b00, 0xc0002969c0, 0xc00019e620, 0x0)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/src/github.com/rancher/terraform-provider-rke/rke/resource_rke_cluster.go:120 +0x14b
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000166870, 0xc00021a780, 0x1bb5b00, 0xc0002969c0, 0xc0004c93d0, 0x0, 0x0)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/helper/schema/resource.go:460 +0x119
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0004c8358, 0x2227820, 0xc00066c540, 0xc000200540, 0xc0004c8358, 0xc00066c540, 0xc001091b78)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/helper/plugin/grpc_provider.go:525 +0x3d8
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x1d8c940, 0xc0004c8358, 0x2227820, 0xc00066c540, 0xc0002004e0, 0x0, 0x2227820, 0xc00066c540, 0xc000846000, 0x3a41b)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003d6480, 0x2241540, 0xc000d0a300, 0xc000186500, 0xc000255ef0, 0x3119490, 0x0, 0x0, 0x0)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1024 +0x501
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: google.golang.org/grpc.(*Server).handleStream(0xc0003d6480, 0x2241540, 0xc000d0a300, 0xc000186500, 0x0)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1313 +0xd3d
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0000b44b0, 0xc0003d6480, 0x2241540, 0xc000d0a300, 0xc000186500)
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:722 +0xa1
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-11-12T11:05:04.725Z [DEBUG] plugin.terraform-provider-rke_v1.1.5:  /go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:720 +0xa1
2020-11-12T11:05:04.729Z [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/rancher/rke/1.1.5/linux_amd64/terraform-provider-rke_v1.1.5 pid=145 error="exit status 2"
2020/11/12 11:05:04 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Canceled desc = context canceled
2020-11-12T11:05:04.729Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Canceled desc = context canceled"
2020/11/12 11:05:04 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2020/11/12 11:05:04 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/11/12 11:05:04 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing

Try to delete upgrade strategy with v1.1.4 and add after update provider to v1.1.5 - everything fine.

rawmind0 commented 3 years ago

Issue reproduced. PR #265 should address the issue.