kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.84k stars 4.64k forks source link

Fails to install on Openstack #9834

Closed oz123 closed 3 years ago

oz123 commented 4 years ago

**1. What kops version are you running? The command kops version, will display

 $ kops version
Version 1.18.0 (git-698bf974d8)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

ubuntu@master-de-nbg6-1a-1-kops1-k8s-local:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 127.0.0.1 was refused - did you specify the right host or port?

3. What cloud provider are you using? Openstack

4. What commands did you run? What is the simplest way to reproduce this issue?

export NAME=kops1.k8s.local

kops create cluster \
  --cloud openstack \
  --name kops1.k8s.local \
  --state ${KOPS_STATE_STORE} \
  --zones  de-nbg6-1a,de-nbg6-1b,de-nbg6-1c \
  --master-zones de-nbg6-1a,de-nbg6-1b,de-nbg6-1c \
  --network-cidr 192.168.1.0/24 \
  --image "ubuntu-focal-cloud" \
  --master-count=3 \
  --node-count=2 \
  --node-size ECS.UC1.4-4 \
  --master-size ECS.C1.4-8 \
  --etcd-storage-type BSS-Ultra-SSD-Storage \
  --topology private \
  --bastion \
  --networking calico \
  --api-loadbalancer-type public \
  --os-ext-net ext01

I edited the subnetworks for utilities:

 - 0.0.0.0/0
  subnets:
  - cidr: 192.168.1.32/27
    name: de-nbg6-1a
    type: Private
    zone: de-nbg6-1a
  - cidr: 192.168.1.64/27
    name: de-nbg6-1b
    type: Private
    zone: de-nbg6-1b
  - cidr: 192.168.1.96/27
    name: de-nbg6-1c
    type: Private
    zone: de-nbg6-1c
  - cidr: 192.168.1.0/29
    name: utility-de-nbg6-1a
    type: Utility
    zone: de-nbg6-1a
  - cidr: 192.168.1.8/29
    name: utility-de-nbg6-1b
    type: Utility
    zone: de-nbg6-1b
  - cidr: 192.168.1.16/29
    name: utility-de-nbg6-1c
    type: Utility
    zone: de-nbg6-1c
  topology:

Than I did:

kops update cluster ${NAME} --yes

5. What happened after the commands executed? The infrastructure was provisioned, but the cluster failed to start. All components start, except for etcd cluster and the api server:

$ sudo docker logs 1053ccb835b9 # this is the api server
Flag --insecure-bind-address has been deprecated, This flag will be removed i
I0828 21:27:22.457134       1 server.go:661] Initializing cache sizes based on 0MB limit
I0828 21:27:22.457465       1 server.go:148] Version: v1.18.8
I0828 21:27:22.458548       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/srv/kubernetes/server.cert::/srv/kubernetes/server.key"
I0828 21:27:23.214860       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/srv/kubernetes/ca.crt"
I0828 21:27:23.215106       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/srv/kubernetes/apiserver-aggregator-ca.cert"
W0828 21:27:23.216222       1 admission.go:78] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0828 21:27:23.217010       1 plugins.go:158] Loaded 13 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0828 21:27:23.217067       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0828 21:27:23.217244       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/srv/kubernetes/ca.crt"
I0828 21:27:23.217441       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/srv/kubernetes/apiserver-aggregator-ca.cert"
W0828 21:27:23.217940       1 admission.go:78] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0828 21:27:23.218317       1 plugins.go:158] Loaded 13 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0828 21:27:23.218369       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0828 21:27:23.220822       1 client.go:361] parsed scheme: "endpoint"
I0828 21:27:23.221039       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:4001  <nil> 0 <nil>}]
W0828 21:27:23.221490       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
I0828 21:27:24.216120       1 client.go:361] parsed scheme: "endpoint"
I0828 21:27:24.216222       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:4001  <nil> 0 <nil>}]
W0828 21:27:24.216936       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:24.221878       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:25.217880       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:25.990465       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:27.016854       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:28.771703       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:29.434322       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:33.093554       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:33.416971       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:39.895743       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
W0828 21:27:40.447818       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:4001  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4001: connect: connection refused". Reconnecting...
panic: context deadline exceeded

goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition.NewREST(0xc0006cea10, 0x50ea9e0, 0xc0001d46c0, 0xc0001d48e8)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition/etcd.go:56 +0x3e7
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.completedConfig.New(0xc000b39980, 0xc000498b88, 0x51a93a0, 0x7748858, 0x10, 0x0, 0x0)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:145 +0x14ef
k8s.io/kubernetes/cmd/kube-apiserver/app.createAPIExtensionsServer(0xc000498b80, 0x51a93a0, 0x7748858, 0x0, 0x50ea540, 0xc000a89e50)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/apiextensions.go:102 +0x59
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateServerChain(0xc000104580, 0xc0002fe5a0, 0x455ea14, 0xc, 0xc000a31c48)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:181 +0x2b8
k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc000104580, 0xc0002fe5a0, 0x0, 0x0)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:150 +0x101
k8s.io/kubernetes/cmd/kube-apiserver/app.NewAPIServerCommand.func1(0xc0000e5680, 0xc0005abb00, 0x0, 0x23, 0x0, 0x0)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:117 +0x104
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc0000e5680, 0xc0000d2010, 0x23, 0x23, 0xc0000e5680, 0xc0000d2010)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826 +0x460
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0000e5680, 0x162f8bbe5c3a2924, 0x772a680, 0xc000068750)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
    /workspace/anago-v1.18.8-rc.1-3+e2dc4848ea15e7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
main.main()
    _output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:43 +0xcd

The logs of etcd:

$ sudo docker logs k8s_etcd-manager_etcd-manager-main-master-de-nbg6-1a-1-kops1-k8s-local_kube-system_33d59d492bbce75677abcdac17ac0eb3_0 | head
etcd-manager
I0828 21:24:49.461341    3773 volumes.go:203] Found project="fe79eafa030046ec83bc05c9f156866d"
I0828 21:24:49.461514    3773 volumes.go:212] Found instanceName="master-de-nbg6-1a-1-kops1-k8s-local"
I0828 21:24:49.909462    3773 volumes.go:232] Found internalIP="192.168.1.57" and zone="de-nbg6-1a"
I0828 21:24:49.909539    3773 main.go:279] Mounting available etcd volumes matching tags [KubernetesCluster=kops1.k8s.local k8s.io/etcd/main k8s.io/role/master=1]; nameTag=k8s.io/etcd/main
I0828 21:24:49.909609    3773 volumes.go:302] Listing Openstack disks in fe79eafa030046ec83bc05c9f156866d/de-nbg6-1a
I0828 21:24:50.344722    3773 mounter.go:294] Trying to mount master volume: "3c85cf0c-0a8f-4bc6-9b74-37f804db6e2e"
I0828 21:24:51.010162    3773 mounter.go:308] Currently attached volumes: [0xc0005a0280]
I0828 21:24:51.010783    3773 mounter.go:72] Master volume "3c85cf0c-0a8f-4bc6-9b74-37f804db6e2e" is attached at "/dev/vdc"
I0828 21:24:51.010903    3773 mounter.go:86] Doing safe-format-and-mount of /dev/vdc to /mnt/master-a.etcd-main.kops1.k8s.local
ubuntu@master-de-nbg6-1a-1-kops1-k8s-local:~$ sudo docker logs k8s_etcd-manager_etcd-manager-main-master-de-nbg6-1a-1-kops1-k8s-local_kube-system_33d59d492bbce75677abcdac17ac0eb3_0 | tail
I0828 22:15:40.805897    3773 commands.go:22] not refreshing commands - TTL not hit
I0828 22:15:40.805930    3773 swiftfs.go:422] Reading file "swift://kops/kops1.k8s.local/backups/etcd/main/control/etcd-cluster-created"
I0828 22:15:42.691389    3773 volumes.go:302] Listing Openstack disks in fe79eafa030046ec83bc05c9f156866d/de-nbg6-1a
I0828 22:15:43.510391    3773 hosts.go:84] hosts update: primary=map[], fallbacks=map[etcd-a.internal.kops1.k8s.local:[192.168.1.57]], final=map[192.168.1.57:[etcd-a.internal.kops1.k8s.local]]
I0828 22:15:43.510591    3773 hosts.go:181] skipping update of unchanged /etc/hosts
I0828 22:15:50.850757    3773 controller.go:333] detected that there is no existing cluster
I0828 22:15:50.851028    3773 commands.go:25] refreshing commands
I0828 22:15:50.939712    3773 vfs.go:104] listed commands in swift://kops/kops1.k8s.local/backups/etcd/main/control: 0 commands
I0828 22:15:50.939763    3773 swiftfs.go:422] Reading file "swift://kops/kops1.k8s.local/backups/etcd/main/control/etcd-cluster-spec"
I0828 22:15:50.981506    3773 controller.go:346] insufficient peers in our gossip group to build a cluster of size 3

6. What did you expect to happen? The cluster starts.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2020-08-28T21:10:30Z"
  generation: 1
  name: kops1.k8s.local
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudConfig:
    openstack:
      blockStorage:
        bs-version: v3
        ignore-volume-az: false
      loadbalancer:
        floatingNetwork: ext01
        floatingNetworkID: dc5e2750-408b-4e4d-bffa-479ce01be3d4
        method: ROUND_ROBIN
        provider: haproxy
        useOctavia: false
      monitor:
        delay: 1m
        maxRetries: 3
        timeout: 30s
      router:
        externalNetwork: ext01
  cloudProvider: openstack
  configBase: swift://kops/kops1.k8s.local
  containerRuntime: docker
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-de-nbg6-1a
      name: a
      volumeType: BSS-Ultra-SSD-Storage
    - instanceGroup: master-de-nbg6-1b
      name: b
      volumeType: BSS-Ultra-SSD-Storage
    - instanceGroup: master-de-nbg6-1c
      name: c
      volumeType: BSS-Ultra-SSD-Storage
    memoryRequest: 100Mi
    name: main
    version: 3.2.24
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-de-nbg6-1a
      name: a
      volumeType: BSS-Ultra-SSD-Storage
    - instanceGroup: master-de-nbg6-1b
      name: b
      volumeType: BSS-Ultra-SSD-Storage
    - instanceGroup: master-de-nbg6-1c
      name: c
      volumeType: BSS-Ultra-SSD-Storage
    memoryRequest: 100Mi
    name: events
    version: 3.2.24
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.18.8
  masterInternalName: api.internal.kops1.k8s.local
  masterPublicName: api.kops1.k8s.local
  networkCIDR: 192.168.1.0/24
  networking:
    calico:
      majorVersion: v3
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 192.168.1.32/27
    name: de-nbg6-1a
    type: Private
    zone: de-nbg6-1a
  - cidr: 192.168.1.64/27
    name: de-nbg6-1b
    type: Private
    zone: de-nbg6-1b
  - cidr: 192.168.1.96/27
    name: de-nbg6-1c
    type: Private
    zone: de-nbg6-1c
  - cidr: 192.168.1.0/29
    name: utility-de-nbg6-1a
    type: Utility
    zone: de-nbg6-1a
  - cidr: 192.168.1.8/29
    name: utility-de-nbg6-1b
    type: Utility
    zone: de-nbg6-1b
  - cidr: 192.168.1.16/29
    name: utility-de-nbg6-1c
    type: Utility
    zone: de-nbg6-1c
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-08-28T21:10:33Z"
  labels:
    kops.k8s.io/cluster: kops1.k8s.local
  name: bastions
spec:
  image: ubuntu-focal-cloud
  machineType: ECS.UC1.1-1
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: bastions
  role: Bastion
  subnets:
  - de-nbg6-1a
  - de-nbg6-1b
  - de-nbg6-1c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-08-28T21:10:31Z"
  labels:
    kops.k8s.io/cluster: kops1.k8s.local
  name: master-de-nbg6-1a
spec:
  image: ubuntu-focal-cloud
  machineType: ECS.C1.4-8
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-de-nbg6-1a
  role: Master
  subnets:
  - de-nbg6-1a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-08-28T21:10:31Z"
  labels:
    kops.k8s.io/cluster: kops1.k8s.local
  name: master-de-nbg6-1b
spec:
  image: ubuntu-focal-cloud
  machineType: ECS.C1.4-8
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-de-nbg6-1b
  role: Master
  subnets:
  - de-nbg6-1b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-08-28T21:10:32Z"
  labels:
    kops.k8s.io/cluster: kops1.k8s.local
  name: master-de-nbg6-1c
spec:
  image: ubuntu-focal-cloud
  machineType: ECS.C1.4-8
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-de-nbg6-1c
  role: Master
  subnets:
  - de-nbg6-1c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-08-28T21:10:32Z"
  labels:
    kops.k8s.io/cluster: kops1.k8s.local
  name: nodes
spec:
  image: ubuntu-focal-cloud
  machineType: ECS.UC1.4-4
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - de-nbg6-1a
  - de-nbg6-1b
  - de-nbg6-1c

9. Anything else do we need to know?

I am using the ubuntu focal cloud image.

olemarkus commented 4 years ago

By the looks of it, it seems that the etcd cluster isn't forming because masters cannot talk to each other over gossip. Would you be able to check that masters can reach each other?

oz123 commented 4 years ago

I think you are correct. Previously when I installed k8s on Openstack I had to allow communication with IP protocol 4 (IP over IP). I don't know how to do that with kops. Can I add my own custom security rules ?

olemarkus commented 4 years ago

It could be a firewall rule is missing from Openstack. You can have a look at what kops does here: https://github.com/kubernetes/kops/blob/master/pkg/model/openstackmodel/firewall.go#L402.

I am not 100% sure what Calico needs. But I had to recently add vxlan for cilium for the openstack setup I am using.

zetaab commented 4 years ago

calico needs that protocol 4, and kops should add that automatically if you use calico. However, etcd is not using calico. Instead it uses host networking, so it should not matter. You could go through the different masters and etcd logs. Please note that there are two etcd clusters: main and event. So each master have two etcd pods, you could check the logs of them - is there anything interesting?

adam-thg commented 3 years ago

I ran into a similar issue as the above, and in my instance the etcd logs (of both etcd clusters) listed only the local deployment as existing, so for whatever reason they couldn't see each other.

I haven't invested much time in debugging further, as my installation was also unsupported (I believe) lacking any sort of LBaaS or DNS resolution in OpenStack.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

adam-thg commented 3 years ago

I don't think this is stale, how's the maintainership of the OpSt code?

olemarkus commented 3 years ago

Best effort :)

OpenStack comes in so many different forms that it is not as easy to maintain as public cloud providers. I am pretty sure the original issue was solved in 1.19 though.

Can you create a new issue with the errors you are seeing?

/close

k8s-ci-robot commented 3 years ago

@olemarkus: Closing this issue.

In response to [this](https://github.com/kubernetes/kops/issues/9834#issuecomment-785095935): >Best effort :) > >OpenStack comes in so many different forms that it is not as easy to maintain as public cloud providers. >I am pretty sure the original issue was solved in 1.19 though. > >Can you create a new issue with the errors you are seeing? > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.