kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.96k stars 4.65k forks source link

Can't apply when upgrade cluster from kops 1.17.1 to kops 1.18.0 #9673

Closed chuonglh closed 4 years ago

chuonglh commented 4 years ago

Hi, When we performed the kops update cluster and then got that issue.

1. What kops version are you running? The command kops version, will display this information. 1.18.0 3. What cloud provider are you using? Openstack 4. What commands did you run? What is the simplest way to reproduce this issue? kops update cluster 5. What happened after the commands executed?

I0804 10:05:00.902859   47987 bootstrapchannelbuilder.go:85] hash 8ac327c94864dffa67d95b6633d26e9e60f9b65d
I0804 10:05:00.903897   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903923   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903926   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903930   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903933   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903936   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903942   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903945   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903948   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903952   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903955   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903963   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.903968   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903973   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903978   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903982   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903985   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903988   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903991   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903995   47987 task.go:102] testing task "Secret"
I0804 10:05:00.903998   47987 task.go:102] testing task "Secret"
I0804 10:05:00.904010   47987 task.go:102] testing task "MirrorSecrets"
I0804 10:05:00.904018   47987 task.go:102] testing task "MirrorKeystore"
I0804 10:05:00.912420   47987 task.go:102] testing task "ManagedFile"
I0804 10:05:00.912703   47987 build_flags.go:49] ignoring non-field: 
I0804 10:05:00.912716   47987 build_flags.go:49] ignoring non-field: 
I0804 10:05:00.912749   47987 proxy.go:30] proxies is == nil, returning empty list
I0804 10:05:00.913843   47987 task.go:102] testing task "ManagedFile"
I0804 10:05:00.913916   47987 task.go:102] testing task "ManagedFile"
I0804 10:05:00.914038   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.914045   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.914052   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.914568   47987 build_flags.go:49] ignoring non-field: 
I0804 10:05:00.914585   47987 build_flags.go:49] ignoring non-field: 
I0804 10:05:00.914616   47987 proxy.go:30] proxies is == nil, returning empty list
I0804 10:05:00.915033   47987 task.go:102] testing task "ManagedFile"
I0804 10:05:00.915046   47987 task.go:102] testing task "ManagedFile"
I0804 10:05:00.915050   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.915054   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.915058   47987 task.go:102] testing task "Keypair"
I0804 10:05:00.915066   47987 task.go:75] EnsureTask ignoring identical 
I0804 10:05:00.915085   47987 task.go:102] testing task "Volume"
I0804 10:05:00.915740   47987 task.go:102] testing task "Volume"
I0804 10:05:00.915766   47987 task.go:102] testing task "Volume"
I0804 10:05:00.915771   47987 task.go:102] testing task "Volume"
I0804 10:05:00.915776   47987 task.go:102] testing task "Volume"
I0804 10:05:00.915780   47987 task.go:102] testing task "Volume"
I0804 10:05:00.916256   47987 task.go:102] testing task "Network"
I0804 10:05:00.916617   47987 task.go:102] testing task "Subnet"
I0804 10:05:00.916635   47987 task.go:102] testing task "RouterInterface"
I0804 10:05:00.916642   47987 task.go:102] testing task "Subnet"
I0804 10:05:00.916646   47987 task.go:102] testing task "RouterInterface"
I0804 10:05:00.916651   47987 task.go:102] testing task "Router"
I0804 10:05:00.917034   47987 task.go:102] testing task "SSHKey"
I0804 10:05:00.917091   47987 cloud.go:351] authenticating to keystone
I0804 10:05:01.294011   47987 cloud.go:470] Openstack disabled loadbalancer support
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x37327d6]

goroutine 1 [running]:
github.com/gophercloud/gophercloud/openstack/loadbalancer/v2/apiversions.listURL(0x0, 0x0, 0x0)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/github.com/gophercloud/gophercloud@v0.9.0/openstack/loadbalancer/v2/apiversions/urls.go:11 +0x26
github.com/gophercloud/gophercloud/openstack/loadbalancer/v2/apiversions.List(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/github.com/gophercloud/gophercloud@v0.9.0/openstack/loadbalancer/v2/apiversions/requests.go:10 +0x5b
k8s.io/cloud-provider-openstack/pkg/util/openstack.getOctaviaVersion(0x0, 0x120, 0x42d0860, 0xcfbf1ab5d8859a01, 0xc00095f0e0)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/k8s.io/cloud-provider-openstack@v1.17.0/pkg/util/openstack/loadbalancer.go:55 +0x5a
k8s.io/cloud-provider-openstack/pkg/util/openstack.IsOctaviaFeatureSupported(0x0, 0x1, 0x4ea0220)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/k8s.io/cloud-provider-openstack@v1.17.0/pkg/util/openstack/loadbalancer.go:78 +0x2f
k8s.io/kops/pkg/model/openstackmodel.(*OpenstackModelContext).UseVIPACL(0xc000010268, 0xc000a48000)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/pkg/model/openstackmodel/context.go:42 +0x171
k8s.io/kops/pkg/model/openstackmodel.(*FirewallModelBuilder).Build(0xc000c171c0, 0xc0010609c0, 0x0, 0x0)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/pkg/model/openstackmodel/firewall.go:484 +0x8f4
k8s.io/kops/upup/pkg/fi/cloudup.(*Loader).BuildTasks(0xc00049caf0, 0x4e68120, 0xc0006736c0, 0xc000c171d0, 0x1, 0x1, 0xc00042b3f0, 0xc000cd0bb0, 0xc0009e9da0, 0x0, ...)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/upup/pkg/fi/cloudup/loader.go:177 +0x448
k8s.io/kops/upup/pkg/fi/cloudup.(*ApplyClusterCmd).Run(0xc000a70000, 0x4e192e0, 0xc0000d6010, 0x0, 0x0)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/upup/pkg/fi/cloudup/apply_cluster.go:809 +0x1d5d
main.RunUpdateCluster(0x4e192e0, 0xc0000d6010, 0xc0009ca780, 0xc000902be0, 0xd, 0x4dbf520, 0xc0000d2008, 0xc0000e4990, 0x5, 0x4761d4d, ...)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/cmd/kops/update_cluster.go:274 +0x9ba
main.NewCmdUpdateCluster.func1(0xc00091fb80, 0xc0009cb260, 0x0, 0x2)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/cmd/kops/update_cluster.go:107 +0x109
github.com/spf13/cobra.(*Command).execute(0xc00091fb80, 0xc0009cb220, 0x2, 0x2, 0xc00091fb80, 0xc0009cb220)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x6ba8cc0, 0x6bf2b48, 0x0, 0x0)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.Execute()
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/cmd/kops/root.go:96 +0x8f
main.main()
    /private/tmp/kops-20200803-23108-1vcnagw/kops-1.18.0/src/k8s.io/kops/cmd/kops/main.go:25 +0x20
johngmyers commented 4 years ago

/area provider/openstack

olemarkus commented 4 years ago

Could you paste your cluster spec?

chuonglh commented 4 years ago

Yes, here is our cluster spec.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null
  generation: 9
  name: hcm.k8s.local
spec:
  api:
    loadBalancer:
      type: ' '
  authorization:
    rbac: {}
  channel: stable
  cloudConfig:
    openstack:
      blockStorage:
        ignore-volume-az: true
        override-volume-az: nova
      monitor:
        delay: 1m
        maxRetries: 3
        timeout: 30s
      router:
        externalNetwork: public
  cloudProvider: openstack
  configBase: swift://kops-openstack/hcm.k8s.local
  etcdClusters:
  - cpuRequest: 300m
    etcdMembers:
    - instanceGroup: master-nova-1
      name: "1"
      volumeSize: 2
      volumeType: __DEFAULT__
    - instanceGroup: master-nova-2
      name: "2"
      volumeSize: 2
      volumeType: __DEFAULT__
    - instanceGroup: master-nova-3
      name: "3"
      volumeSize: 2
      volumeType: __DEFAULT__
    memoryRequest: 200Mi
    name: main
    version: 3.4.3
  - cpuRequest: 300m
    etcdMembers:
    - instanceGroup: master-nova-1
      name: "1"
      volumeSize: 2
      volumeType: __DEFAULT__
    - instanceGroup: master-nova-2
      name: "2"
      volumeSize: 2
      volumeType: __DEFAULT__
    - instanceGroup: master-nova-3
      name: "3"
      volumeSize: 2
      volumeType: __DEFAULT__
    memoryRequest: 200Mi
    name: events
    version: 3.4.3
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    admissionControl:
    - NamespaceLifecycle
    - LimitRanger
    - ServiceAccount
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - MutatingAdmissionWebhook
    - ValidatingAdmissionWebhook
    - NodeRestriction
    - ResourceQuota
    - Priority
    auditLogMaxAge: 10
    auditLogMaxBackups: 1
    auditLogMaxSize: 100
    auditLogPath: /var/log/kube-apiserver-audit.log
  kubeControllerManager:
    attachDetachReconcileSyncPeriod: 3m0s
  kubeDNS:
    provider: CoreDNS
  kubeProxy:
    metricsBindAddress: 0.0.0.0
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
    featureGates:
      PodPriority: "true"
    imagePullProgressDeadline: 5m0s
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.18.6
  masterInternalName: api.internal.hcm.k8s.local
  masterPublicName: api.hcm.k8s.local
  networkCIDR: 172.16.0.0/16
  networking:
    calico:
      ipipMode: Always
      majorVersion: v3
      mtu: 1390
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.16.32.0/19
    name: nova
    type: Private
    zone: nova
  - cidr: 172.16.0.0/22
    name: utility-nova
    type: Utility
    zone: nova
  topology:
    dns:
      type: Public
    masters: private
    nodes: private
zetaab commented 4 years ago

the row which is causing the issue is https://github.com/kubernetes/kops/blob/release-1.18/pkg/model/openstackmodel/firewall.go#L484

What is this loadbalancer type?

  api:
    loadBalancer:
      type: ' '

So do you have loadbalancer in your setup or not? The error message is saying that you do not have loadbalancer in your region (or at least api does not exist). If you do not have then you should remove the entire api structure from the config or use:

  api: {}

This function should return false in your setup https://github.com/kubernetes/kops/blob/master/pkg/model/context.go#L298-L304

chuonglh commented 4 years ago

Hi @zetaab, Currently, I don't have a loadbalancer. I follow this instruction here to not using loadbalancer https://kops.sigs.k8s.io/getting_started/openstack/#using-openstack-without-lbaas.

I tried remove api structure or set like you guide above but not success.

# Found fields that are not recognized
# ...
#     name: hcm.k8s.local
#   spec:
# -   api:
# -     loadBalancer:
# -       type: Public
#     authorization:
#       rbac: {}
# ...
#
#
# Found fields that are not recognized
# ...
#     name: hcm.k8s.local
#   spec:
# -   api:
# -     loadBalancer:
# +   api: {}
# -       type: Public
#     authorization:
#       rbac: {}
# ...
#
#
olemarkus commented 4 years ago

This has been fixed in master (9b0d235554a) together with a few other related issues. Are you able to test kops master version?

chuonglh commented 4 years ago

Hi @olemarkus,

I have tested with kops master version and it works. So, when we have a new release for it?

olemarkus commented 4 years ago

Good to know, thanks. There will probably be another pre-release soon. Stable release sometime in late september/early october if all goes well.

If all is good, could you close the issue?

chuonglh commented 4 years ago

Yes, sure. Thanks, I really appreciate it!