search

rancher / rancher

Complete container management platform
http://rancher.com
Apache License 2.0
23.45k stars 2.97k forks source link

Digital ocean custom node freeze at waiting to register with Kubernetes #12103

Closed marufbd closed 6 years ago

marufbd commented 6 years ago

Rancher versions: rancher/server:preview rancher/agent:v2.0.3

Infrastructure Stack versions: kubernetes (if applicable):

Docker version: (docker version,docker info preferred) Master Version: 17.12.0-ce API version: 1.35 Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:11:19 2017 OS/Arch: linux/amd64

Custom Node Version: 17.09.1-ce API version: 1.32 Go version: go1.8.5 Git commit: 19e2cf6 Built: Thu Dec 7 22:19:00 2017 OS/Arch: linux/amd64

Operating system and kernel: (cat /etc/os-release, uname -r preferred) Master NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.3 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial

Custom Node NAME="Container Linux by CoreOS" ID=coreos VERSION=1632.3.0 VERSION_ID=1632.3.0 BUILD_ID=2018-02-14-0338 PRETTY_NAME="Container Linux by CoreOS 1632.3.0 (Ladybug)" ANSI_COLOR="38;5;75" HOME_URL="https://coreos.com/" BUG_REPORT_URL="https://issues.coreos.com" COREOS_BOARD="amd64-usr"

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) Digital Ocean

Steps to Reproduce: On Master run: docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/server:preview

Results: One node comes up on server web cluster->nodes showing indefinitlely registering waiting to register with Kubernetes

The agent log seems getting stuck on connecting to proxy: -----BEGIN CERTIFICATE----- MIIC7jCCAdagAwIBAgIBADANBgkqhkiG9w0BAQsFADAoMRIwEAYDVQQKEwl0aGUt cmFuY2gxEjAQBgNVBAMTCWNhdHRsZS1jYTAeFw0xODAzMTQwODExNTVaFw0yODAz MTEwODExNTVaMCgxEjAQBgNVBAoTCXRoZS1yYW5jaDESMBAGA1UEAxMJY2F0dGxl LWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo2NRJ0P/o75eHQVA If6xtYY9B10p9AKVg11AJNgkdzuYNA3IXXJz1K96SrheMGSJQpfrtqJGoaevqNpE uYJ9+MYEIfN71WuNTtyVKbCsYgMT3faom4ACx2g7lbFeZb/MebX7hwCMOfrgImdM TCGhcEbHY2aw26LqViZc3WNzSJUxqeBlNjAptTNXzO7TpvdiWyXxYnl4XW9Pj3GP /tgkWzoa7AON5p4yqH7qe56KsMkAWv73XLryhibtuSg+/o9FnS5k/AqdXMCDD+US lrxbfvOnnnG3Bo6jIAyucABdTWyPV3SyFe1pEKG2hrkTYQNDZltjT7JJKm5y4bGP zQUWiQIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAN BgkqhkiG9w0BAQsFAAOCAQEAPkdNQMY2mP2E4j16OOJiglCBwCZSkpoFIT2se65C x42RSHVKmQi3tm/wd0rTTXK/gkiZK/2q8/erRtzkQrO3QTNdgFOYolew6wRNxowa TbNCaNtvBl0T+fjQiWL88iJkvkBD//a11zW5dUL+h+3iBGIR2sT/7dSLVokiMRH0 8p4JPDiitFri6on5J7ZGDMEpna7/95JqNUsYbpzNrc0/wfPL8P4kLH27Ftb08OO5 VrbovE2Yz8kNxvXL7/CE06ZfEkoixMFzkmVvuhXb20Geoa6kmwXfWg0lB9lgRxhG 8kvzs8vf6s4grlRPzVw8vKdNveNBFEu6D2rdx+Nv0++1UA== -----END CERTIFICATE----- time="2018-03-14T08:24:36Z" level=info msg="Option customConfig=map[address:139.59.189.10 internalAddress: roles:[etcd worker controlplane]]" time="2018-03-14T08:24:36Z" level=info msg="Option etcd=true" time="2018-03-14T08:24:36Z" level=info msg="Option controlPlane=true" time="2018-03-14T08:24:36Z" level=info msg="Option worker=true" time="2018-03-14T08:24:36Z" level=info msg="Option requestedHostname=kube-control-node-lon1-01" time="2018-03-14T08:24:36Z" level=info msg="Connecting to wss://myserverdomain.com/v3/connect with token h5djl5kmvsqz7j95hwtbnnbtlq2tdxfs7xkvhpmqvtk9x27gzndv9r" time="2018-03-14T08:24:36Z" level=info msg="Connecting to proxy" url="wss://myserverdomain.com/v3/connect"

Seems gets stuck there, no log after that.

Master server logs: 2018/03/14 08:11:30 [INFO] Activating driver gke 2018/03/14 08:11:30 [INFO] Activating driver gke done 2018/03/14 08:11:30 [INFO] Activating driver rke 2018/03/14 08:11:30 [INFO] Activating driver rke done 2018/03/14 08:11:30 [INFO] Activating driver aks 2018/03/14 08:11:30 [INFO] Activating driver aks done 2018/03/14 08:11:30 [INFO] Activating driver import 2018/03/14 08:11:30 [INFO] Activating driver import done 2018/03/14 08:11:30 [INFO] [certificates] Generating CA kubernetes certificates 2018/03/14 08:11:31 [INFO] [certificates] Generating Kubernetes API server certificates 2018/03/14 08:11:32 [INFO] [certificates] Generating Kube Controller certificates 2018/03/14 08:11:32 [INFO] [certificates] Generating Kube Scheduler certificates 2018/03/14 08:11:34 [INFO] [certificates] Generating Kube Proxy certificates 2018/03/14 08:11:35 [INFO] [certificates] Generating Node certificate 2018/03/14 08:11:35 [INFO] [certificates] Generating admin certificates and kubeconfig 2018/03/14 08:11:36 [INFO] [certificates] Generating etcd-127.0.0.1 certificate and key 2018/03/14 08:11:36 [INFO] Running kube-apiserver --insecure-bind-address=127.0.0.1 --bind-address=0.0.0.0 --insecure-port=0 --secure-port=6443 --cloud-provider= --allow_privileged=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-cluster-ip-range=10.233.0.0/18 --admission-control=ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds --storage-backend=etcd3 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --advertise-address=10.233.0.1 --bind-address=127.0.0.1 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-servers=https://127.0.0.1:2379 --etcd-prefix=/registry --authorization-mode=Node,RBAC -v=1 --logtostderr=false --alsologtostderr=false 2018/03/14 08:11:36 [INFO] Running etcd --name=etcd-master --data-dir=/var/lib/rancher/etcd --advertise-client-urls=https://127.0.0.1:2379,https://127.0.0.1:4001 --listen-client-urls=https://0.0.0.0:2379 --initial-advertise-peer-urls=https://127.0.0.1:2380 --listen-peer-urls=https://0.0.0.0:2380 --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-master=https://127.0.0.1:2380 --initial-cluster-state=new --peer-client-cert-auth --client-cert-auth --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cert-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem --key-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem 2018-03-14 08:11:36.387773 I | etcdmain: etcd Version: 3.2.13 2018-03-14 08:11:36.387983 I | etcdmain: Git SHA: Not provided (use ./build instead of go build) 2018-03-14 08:11:36.388088 I | etcdmain: Go Version: go1.9.2 2018-03-14 08:11:36.388191 I | etcdmain: Go OS/Arch: linux/amd64 2018-03-14 08:11:36.388287 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1 2018-03-14 08:11:36.388535 I | embed: peerTLS: cert = /etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem, key = /etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem, ca = , trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true 2018-03-14 08:11:36.389926 I | embed: listening for peers on https://0.0.0.0:2380 2018-03-14 08:11:36.390247 I | embed: listening for client requests on 0.0.0.0:2379 2018-03-14 08:11:36.393323 I | etcdserver: name = etcd-master 2018-03-14 08:11:36.393616 I | etcdserver: data dir = /var/lib/rancher/etcd 2018-03-14 08:11:36.393707 I | etcdserver: member dir = /var/lib/rancher/etcd/member 2018-03-14 08:11:36.393790 I | etcdserver: heartbeat = 100ms 2018-03-14 08:11:36.393859 I | etcdserver: election = 1000ms 2018-03-14 08:11:36.393925 I | etcdserver: snapshot count = 100000 2018-03-14 08:11:36.394012 I | etcdserver: advertise client URLs = https://127.0.0.1:2379,https://127.0.0.1:4001 2018-03-14 08:11:36.394084 I | etcdserver: initial advertise peer URLs = https://127.0.0.1:2380 2018-03-14 08:11:36.394191 I | etcdserver: initial cluster = etcd-master=https://127.0.0.1:2380 2018-03-14 08:11:36.399494 I | etcdserver: starting member e92d66acd89ecf29 in cluster 7581d6eb2d25405b 2018-03-14 08:11:36.400370 I | raft: e92d66acd89ecf29 became follower at term 0 2018-03-14 08:11:36.400506 I | raft: newRaft e92d66acd89ecf29 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] 2018-03-14 08:11:36.400623 I | raft: e92d66acd89ecf29 became follower at term 1 2018-03-14 08:11:36.419642 W | auth: simple token is not cryptographically signed 2018/03/14 08:11:36 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:36 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018-03-14 08:11:36.459896 I | etcdserver: starting server... [version: 3.2.13, cluster version: to_be_decided] 2018-03-14 08:11:36.461282 I | embed: ClientTLS: cert = /etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem, key = /etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem, ca = , trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true 2018/03/14 08:11:36 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018-03-14 08:11:36.465824 I | etcdserver/membership: added member e92d66acd89ecf29 [https://127.0.0.1:2380] to cluster 7581d6eb2d25405b [restful] 2018/03/14 08:11:37 log.go:33: [restful/swagger] listing is available at https://10.233.0.1:6443/swaggerapi [restful] 2018/03/14 08:11:37 log.go:33: [restful/swagger] https://10.233.0.1:6443/swaggerui/ is mapped to folder /swagger-ui/ 2018-03-14 08:11:37.240286 I | raft: e92d66acd89ecf29 is starting a new election at term 1 2018-03-14 08:11:37.240542 I | raft: e92d66acd89ecf29 became candidate at term 2 2018-03-14 08:11:37.240681 I | raft: e92d66acd89ecf29 received MsgVoteResp from e92d66acd89ecf29 at term 2 2018-03-14 08:11:37.240822 I | raft: e92d66acd89ecf29 became leader at term 2 2018-03-14 08:11:37.240927 I | raft: raft.node: e92d66acd89ecf29 elected leader e92d66acd89ecf29 at term 2 2018-03-14 08:11:37.264565 I | etcdserver: published {Name:etcd-master ClientURLs:[https://127.0.0.1:2379 https://127.0.0.1:4001]} to cluster 7581d6eb2d25405b 2018-03-14 08:11:37.265452 I | etcdserver: setting up the initial cluster version to 3.2 2018-03-14 08:11:37.266440 I | embed: ready to serve client requests 2018-03-14 08:11:37.268272 I | embed: serving client requests on [::]:2379 2018-03-14 08:11:37.271781 N | etcdserver/membership: set the initial cluster version to 3.2 2018-03-14 08:11:37.272470 I | etcdserver/api: enabled capabilities for version 3.2 2018/03/14 08:11:38 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:38 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:38 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused [restful] 2018/03/14 08:11:39 log.go:33: [restful/swagger] listing is available at https://10.233.0.1:6443/swaggerapi [restful] 2018/03/14 08:11:39 log.go:33: [restful/swagger] https://10.233.0.1:6443/swaggerui/ is mapped to folder /swagger-ui/ 2018/03/14 08:11:40 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:40 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:40 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:42 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:42 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:42 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:44 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:44 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:44 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused 2018/03/14 08:11:46 [INFO] Running kube-controller-manager --address=0.0.0.0 --cloud-provider= --leader-elect=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --enable-hostpath-provisioner=false --node-monitor-grace-period=40s --pod-eviction-timeout=5m0s --v=2 --allocate-node-cidrs=true --cluster-cidr=10.233.64.0/18 --service-cluster-ip-range=10.233.0.0/18 --service-account-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --use-service-account-credentials=true -v=1 --logtostderr=false --alsologtostderr=false 2018/03/14 08:11:46 [INFO] Running kube-scheduler --leader-elect=true --v=2 --address=0.0.0.0 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -v=1 --logtostderr=false --alsologtostderr=false E0314 08:11:46.744346 1 server.go:173] unable to register configz: register config "componentconfig" twice E0314 08:11:46.940565 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope E0314 08:11:46.941012 1 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: endpoints "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get endpoints in the namespace "kube-system" E0314 08:11:46.941202 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope E0314 08:11:46.941425 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope E0314 08:11:46.941655 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1.Service: services is forbidden: User "system:kube-scheduler" cannot list services at the cluster scope E0314 08:11:46.941834 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope E0314 08:11:46.941998 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope E0314 08:11:46.942173 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/server.go:103: Failed to list v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list pods at the cluster scope E0314 08:11:46.942345 1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list nodes at the cluster scope 2018/03/14 08:11:46 [INFO] Creating CRD authconfigs.management.cattle.io 2018/03/14 08:11:46 [INFO] Creating CRD apps.project.cattle.io 2018/03/14 08:11:46 [INFO] Creating CRD clusteralerts.management.cattle.io 2018/03/14 08:11:46 [INFO] Creating CRD projectalerts.management.cattle.io 2018/03/14 08:11:46 [INFO] Waiting for CRD apps.project.cattle.io to become available 2018/03/14 08:11:47 [INFO] Creating CRD notifiers.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD catalogs.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD clusterevents.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD clusterloggings.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD clusterregistrationtokens.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD clusterroletemplatebindings.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD clusters.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD dynamicschemas.management.cattle.io 2018/03/14 08:11:47 [INFO] Creating CRD globalrolebindings.management.cattle.io 2018/03/14 08:11:47 [INFO] Done waiting for CRD apps.project.cattle.io to become available 2018/03/14 08:11:48 [INFO] Creating CRD globalroles.management.cattle.io 2018/03/14 08:11:48 [INFO] Creating CRD groupmembers.management.cattle.io 2018/03/14 08:11:48 [INFO] Creating CRD groups.management.cattle.io 2018/03/14 08:11:48 [INFO] Creating CRD listenconfigs.management.cattle.io 2018/03/14 08:11:48 [INFO] Creating CRD nodes.management.cattle.io 2018/03/14 08:11:49 [INFO] Creating CRD nodepools.management.cattle.io 2018/03/14 08:11:49 [INFO] Creating CRD nodedrivers.management.cattle.io 2018/03/14 08:11:49 [INFO] Creating CRD nodetemplates.management.cattle.io 2018/03/14 08:11:49 [INFO] Creating CRD podsecuritypolicytemplates.management.cattle.io 2018/03/14 08:11:49 [INFO] Creating CRD preferences.management.cattle.io 2018/03/14 08:11:50 [INFO] Creating CRD projectloggings.management.cattle.io 2018/03/14 08:11:50 [INFO] Creating CRD projectnetworkpolicies.management.cattle.io 2018/03/14 08:11:50 [INFO] Creating CRD projectroletemplatebindings.management.cattle.io 2018/03/14 08:11:50 [INFO] Creating CRD projects.management.cattle.io 2018/03/14 08:11:50 [INFO] Creating CRD roletemplates.management.cattle.io 2018/03/14 08:11:51 [INFO] Creating CRD settings.management.cattle.io 2018/03/14 08:11:51 [INFO] Creating CRD templates.management.cattle.io 2018/03/14 08:11:51 [INFO] Creating CRD templateversions.management.cattle.io 2018/03/14 08:11:51 [INFO] Creating CRD clusterpipelines.management.cattle.io E0314 08:11:51.682172 1 certificates.go:48] Failed to start certificate controller: error reading CA cert file "/etc/kubernetes/ca/ca.pem": open /etc/kubernetes/ca/ca.pem: no such file or directory 2018/03/14 08:11:51 [INFO] Creating CRD pipelines.management.cattle.io 2018/03/14 08:11:52 [INFO] Creating CRD pipelineexecutions.management.cattle.io E0314 08:11:52.080298 1 core.go:70] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail. 2018/03/14 08:11:52 [INFO] Creating CRD pipelineexecutionlogs.management.cattle.io 2018/03/14 08:11:52 [INFO] Creating CRD sourcecodecredentials.management.cattle.io 2018/03/14 08:11:52 [INFO] Creating CRD sourcecoderepositories.management.cattle.io 2018/03/14 08:11:52 [INFO] Creating CRD tokens.management.cattle.io 2018/03/14 08:11:53 [INFO] Creating CRD users.management.cattle.io 2018/03/14 08:11:53 [INFO] Starting API controllers 2018/03/14 08:11:53 [INFO] Syncing ServiceController Controller 2018/03/14 08:11:53 [INFO] Syncing ListenConfigController Controller 2018/03/14 08:11:53 [INFO] Syncing RoleController Controller 2018/03/14 08:11:53 [INFO] Syncing PodController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterRegistrationTokenController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterController Controller 2018/03/14 08:11:53 [INFO] Syncing NodeController Controller 2018/03/14 08:11:53 [INFO] Syncing NodeController Controller 2018/03/14 08:11:53 [INFO] Syncing UserController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterRoleTemplateBindingController Controller 2018/03/14 08:11:53 [INFO] Syncing ProjectRoleTemplateBindingController Controller 2018/03/14 08:11:53 [INFO] Syncing TokenController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterLoggingController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterPipelineController Controller 2018/03/14 08:11:53 [INFO] Syncing SourceCodeCredentialController Controller 2018/03/14 08:11:53 [INFO] Syncing SourceCodeRepositoryController Controller 2018/03/14 08:11:53 [INFO] Syncing PipelineController Controller 2018/03/14 08:11:53 [INFO] Syncing GroupMemberController Controller 2018/03/14 08:11:53 [INFO] Syncing GroupController Controller 2018/03/14 08:11:53 [INFO] Syncing DynamicSchemaController Controller 2018/03/14 08:11:53 [INFO] Syncing SettingController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterRoleBindingController Controller 2018/03/14 08:11:53 [INFO] Syncing RoleBindingController Controller 2018/03/14 08:11:53 [INFO] Syncing ClusterRoleController Controller 2018/03/14 08:11:53 [INFO] Syncing RoleBindingController Controller Done 2018/03/14 08:11:53 [INFO] Syncing ServiceController Controller Done 2018/03/14 08:11:53 [INFO] Syncing RoleController Controller Done 2018/03/14 08:11:53 [INFO] Syncing PodController Controller Done 2018/03/14 08:11:53 [INFO] Syncing SettingController Controller Done 2018/03/14 08:11:53 [INFO] Syncing ClusterRoleBindingController Controller Done 2018/03/14 08:11:53 [INFO] Syncing ClusterRoleController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ListenConfigController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleTemplateBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterRegistrationTokenController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterController Controller Done 2018/03/14 08:11:54 [INFO] Syncing NodeController Controller Done 2018/03/14 08:11:54 [INFO] Syncing NodeController Controller Done 2018/03/14 08:11:54 [INFO] Syncing UserController Controller Done 2018/03/14 08:11:54 [INFO] Syncing PipelineController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ProjectRoleTemplateBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing TokenController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterLoggingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterPipelineController Controller Done 2018/03/14 08:11:54 [INFO] Syncing SourceCodeCredentialController Controller Done 2018/03/14 08:11:54 [INFO] Syncing SourceCodeRepositoryController Controller Done 2018/03/14 08:11:54 [INFO] Syncing GroupMemberController Controller Done 2018/03/14 08:11:54 [INFO] Syncing GroupController Controller Done 2018/03/14 08:11:54 [INFO] Syncing DynamicSchemaController Controller Done 2018/03/14 08:11:54 [INFO] Starting catalog controller 2018/03/14 08:11:54 [INFO] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cattle-controllers", UID:"5b49fa5d-275f-11e8-afc7-0242ac110002", APIVersion:"v1", ResourceVersion:"262", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 0daff664d6ef became leader 2018/03/14 08:11:54 [INFO] Starting management controllers 2018/03/14 08:11:54 [INFO] Syncing NamespaceController Controller 2018/03/14 08:11:54 [INFO] Syncing NodePoolController Controller 2018/03/14 08:11:54 [INFO] Syncing RoleController Controller 2018/03/14 08:11:54 [INFO] Syncing RoleTemplateController Controller 2018/03/14 08:11:54 [INFO] Syncing ProjectController Controller 2018/03/14 08:11:54 [INFO] Syncing ClusterController Controller 2018/03/14 08:11:54 [INFO] Syncing GlobalRoleController Controller 2018/03/14 08:11:54 [INFO] Syncing ProjectRoleTemplateBindingController Controller 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleTemplateBindingController Controller 2018/03/14 08:11:54 [INFO] Syncing GlobalRoleBindingController Controller 2018/03/14 08:11:54 [INFO] Syncing CatalogController Controller 2018/03/14 08:11:54 [INFO] Syncing ClusterEventController Controller 2018/03/14 08:11:54 [INFO] Syncing ProjectAlertController Controller 2018/03/14 08:11:54 [INFO] Syncing NodeController Controller 2018/03/14 08:11:54 [INFO] Syncing DynamicSchemaController Controller 2018/03/14 08:11:54 [INFO] Syncing NodeDriverController Controller 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleBindingController Controller 2018/03/14 08:11:54 [INFO] Syncing RoleBindingController Controller 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleController Controller 2018/03/14 08:11:54 [INFO] Syncing NamespaceController Controller Done 2018/03/14 08:11:54 [INFO] Syncing RoleController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ProjectRoleTemplateBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleTemplateBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing NodeController Controller Done 2018/03/14 08:11:54 [INFO] Syncing DynamicSchemaController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing RoleBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterRoleController Controller Done 2018/03/14 08:11:54 [INFO] Syncing NodePoolController Controller Done 2018/03/14 08:11:54 [INFO] Syncing RoleTemplateController Controller Done 2018/03/14 08:11:54 [INFO] Syncing GlobalRoleBindingController Controller Done 2018/03/14 08:11:54 [INFO] Syncing CatalogController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ClusterEventController Controller Done 2018/03/14 08:11:54 [INFO] Syncing ProjectAlertController Controller Done 2018/03/14 08:11:54 [INFO] Syncing NodeDriverController Controller Done 2018/03/14 08:11:55 [INFO] Syncing ProjectController Controller Done 2018/03/14 08:11:55 [INFO] Syncing GlobalRoleController Controller Done 2018/03/14 08:11:55 [INFO] Reconciling GlobalRoles 2018/03/14 08:11:55 [INFO] Listening on :443 2018/03/14 08:11:55 [INFO] Listening on :80 2018/03/14 08:11:55 [INFO] Creating manage-users 2018/03/14 08:11:55 [INFO] Creating manage-node-templates 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-users for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating use-node-templates 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-node-templates for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating user-base 2018/03/14 08:11:55 [INFO] Creating manage-clusters 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-use-node-templates for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-user-base for corresponding GlobalRole E0314 08:11:55.481580 1 generic_controller.go:200] GlobalRoleController manage-users [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-users": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating manage-node-drivers E0314 08:11:55.509353 1 generic_controller.go:200] GlobalRoleController manage-node-templates [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-node-templates": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-clusters for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating manage-catalogs 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-node-drivers for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating user 2018/03/14 08:11:55 [INFO] Creating admin 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-catalogs for corresponding GlobalRole E0314 08:11:55.566264 1 generic_controller.go:200] GlobalRoleController user-base [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "user-base": the object has been modified; please apply your changes to the latest version and try again E0314 08:11:55.573706 1 generic_controller.go:200] GlobalRoleController manage-clusters [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-clusters": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-user for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating create-clusters 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-admin for corresponding GlobalRole E0314 08:11:55.596053 1 generic_controller.go:200] GlobalRoleController manage-node-drivers [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-node-drivers": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating use-catalogs 2018/03/14 08:11:55 [INFO] Creating manage-roles E0314 08:11:55.616270 1 generic_controller.go:200] GlobalRoleController manage-catalogs [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-catalogs": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-create-clusters for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-use-catalogs for corresponding GlobalRole E0314 08:11:55.639425 1 generic_controller.go:200] GlobalRoleController user [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "user": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating manage-settings E0314 08:11:55.661146 1 generic_controller.go:200] GlobalRoleController admin [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "admin": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating manage-authn 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-roles for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-settings for corresponding GlobalRole 2018/03/14 08:11:55 [INFO] Reconciling RoleTemplates E0314 08:11:55.696125 1 generic_controller.go:200] GlobalRoleController use-catalogs [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "use-catalogs": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating create-projects 2018/03/14 08:11:55 [INFO] Creating new ClusterRole cattle-globalrole-manage-authn for corresponding GlobalRole E0314 08:11:55.720337 1 generic_controller.go:200] GlobalRoleController manage-roles [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-roles": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating view-workloads E0314 08:11:55.732686 1 generic_controller.go:200] GlobalRoleController manage-settings [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-settings": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating view-configmaps E0314 08:11:55.742654 1 generic_controller.go:200] GlobalRoleController manage-authn [mgmt-auth-gr-controller] failed with : Operation cannot be fulfilled on globalroles.management.cattle.io "manage-authn": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:55 [INFO] Creating view-serviceaccounts 2018/03/14 08:11:55 [INFO] Creating manage-cluster-members 2018/03/14 08:11:55 [INFO] Creating view-services 2018/03/14 08:11:55 [INFO] Creating manage-serviceaccounts 2018/03/14 08:11:55 [INFO] Creating admin 2018/03/14 08:11:55 [INFO] Creating view 2018/03/14 08:11:55 [INFO] Creating view-volumes 2018/03/14 08:11:55 [INFO] Creating create-ns 2018/03/14 08:11:55 [INFO] Creating manage-workloads 2018/03/14 08:11:55 [INFO] Creating view-ingress 2018/03/14 08:11:55 [INFO] Creating view-secrets 2018/03/14 08:11:55 [INFO] Creating cluster-owner 2018/03/14 08:11:55 [INFO] Creating manage-projects 2018/03/14 08:11:55 [INFO] Creating manage-volumes 2018/03/14 08:11:55 [INFO] Creating view-cluster-members 2018/03/14 08:11:55 [INFO] Creating project-owner 2018/03/14 08:11:55 [INFO] Creating manage-service 2018/03/14 08:11:55 [INFO] Creating manage-configmaps 2018/03/14 08:11:55 [INFO] Creating view-persistentvolumeclaims 2018/03/14 08:11:55 [INFO] Creating view-projects 2018/03/14 08:11:55 [INFO] Creating manage-nodes 2018/03/14 08:11:55 [INFO] Creating read-only 2018/03/14 08:11:55 [INFO] Creating manage-project-members 2018/03/14 08:11:55 [INFO] Creating view-project-members 2018/03/14 08:11:55 [INFO] Creating view-events 2018/03/14 08:11:55 [INFO] Creating cluster-admin 2018/03/14 08:11:55 [INFO] Creating view-nodes 2018/03/14 08:11:55 [INFO] Creating project-member 2018/03/14 08:11:55 [INFO] Creating manage-ingress 2018/03/14 08:11:55 [INFO] Creating manage-persistentvolumeclaims 2018/03/14 08:11:55 [INFO] Creating edit 2018/03/14 08:11:55 [INFO] Creating cluster-member 2018/03/14 08:11:55 [INFO] Creating manage-secrets 2018/03/14 08:11:56 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding globalrolebinding-5vth2 E0314 08:11:56.132533 1 generic_controller.go:200] GlobalRoleBindingController globalrolebinding-5vth2 [mgmt-auth-grb-controller] failed with : Operation cannot be fulfilled on globalrolebindings.management.cattle.io "globalrolebinding-5vth2": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:56 [INFO] Creating node driver amazonec2 2018/03/14 08:11:56 [INFO] Creating node driver digitalocean 2018/03/14 08:11:56 [INFO] Creating node driver exoscale 2018/03/14 08:11:56 [INFO] Creating node driver openstack 2018/03/14 08:11:56 [INFO] Creating node driver otc E0314 08:11:56.237148 1 generic_controller.go:200] NodeDriverController exoscale [node-driver-controller] failed with : Operation cannot be fulfilled on nodedrivers.management.cattle.io "exoscale": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:56 [INFO] Creating node driver packet 2018/03/14 08:11:56 [INFO] Creating node driver rackspace 2018/03/14 08:11:56 [INFO] Creating node driver softlayer E0314 08:11:56.329232 1 generic_controller.go:200] NodeDriverController openstack [node-driver-controller] failed with : Operation cannot be fulfilled on nodedrivers.management.cattle.io "openstack": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:56 [INFO] Creating node driver vmwarevsphere 2018/03/14 08:11:56 [INFO] uploading digitaloceanConfig to node schema 2018/03/14 08:11:56 [INFO] uploading digitaloceanConfig to node schema 2018/03/14 08:11:56 [INFO] uploading amazonec2Config to node schema 2018/03/14 08:11:56 [INFO] uploading amazonec2Config to node schema E0314 08:11:56.587235 1 generic_controller.go:200] NodeDriverController vmwarevsphere [node-driver-controller] failed with : Operation cannot be fulfilled on nodedrivers.management.cattle.io "vmwarevsphere": the object has been modified; please apply your changes to the latest version and try again 2018/03/14 08:11:56 [INFO] uploading vmwarevsphereConfig to node schema 2018/03/14 08:11:56 [INFO] uploading vmwarevsphereConfig to node schema 2018/03/14 08:12:06 [INFO] Updating catalog helm 2018/03/14 08:12:06 [INFO] Creating template helm-kubernetes-etcd-operator 2018/03/14 08:12:07 [INFO] Creating template helm-kubernetes-grafana 2018/03/14 08:12:07 [INFO] Creating template helm-kubernetes-sugarcrm 2018/03/14 08:12:08 [INFO] Creating template helm-kubernetes-sonarqube 2018/03/14 08:12:08 [INFO] Creating template helm-kubernetes-spinnaker 2018/03/14 08:12:09 [INFO] Creating template helm-kubernetes-metabase 2018/03/14 08:12:10 [INFO] Creating template helm-kubernetes-magento 2018/03/14 08:12:12 [INFO] Creating template helm-kubernetes-orangehrm 2018/03/14 08:12:12 [INFO] Creating template helm-kubernetes-verdaccio 2018/03/14 08:12:12 [INFO] Creating template helm-kubernetes-voyager 2018/03/14 08:12:13 [INFO] Creating template helm-kubernetes-redis-ha 2018/03/14 08:12:13 [INFO] Creating template helm-kubernetes-telegraf 2018/03/14 08:12:13 [INFO] Creating template helm-kubernetes-fluent-bit 2018/03/14 08:12:13 [INFO] Creating template helm-kubernetes-weave-scope 2018/03/14 08:12:13 [INFO] Creating template helm-kubernetes-postgresql 2018/03/14 08:12:14 [INFO] Creating template helm-kubernetes-redmine 2018/03/14 08:12:15 [INFO] Creating template helm-kubernetes-kapacitor 2018/03/14 08:12:15 [INFO] Creating template helm-kubernetes-minecraft 2018/03/14 08:12:15 [INFO] Creating template helm-kubernetes-rabbitmq 2018/03/14 08:12:16 [INFO] Creating template helm-kubernetes-ipfs 2018/03/14 08:12:16 [INFO] Creating template helm-kubernetes-spotify-docker-gc 2018/03/14 08:12:16 [INFO] Creating template helm-kubernetes-chaoskube 2018/03/14 08:12:16 [INFO] Creating template helm-kubernetes-lamp 2018/03/14 08:12:17 [INFO] Creating template helm-kubernetes-gitlab-ee 2018/03/14 08:12:17 [INFO] Creating template helm-kubernetes-spartakus 2018/03/14 08:12:17 [INFO] Creating template helm-kubernetes-phpbb 2018/03/14 08:12:18 [INFO] Creating template helm-kubernetes-mailhog 2018/03/14 08:12:18 [INFO] Creating template helm-kubernetes-cluster-autoscaler 2018/03/14 08:12:18 [INFO] Creating template helm-kubernetes-mediawiki 2018/03/14 08:12:18 [INFO] Creating template helm-kubernetes-drupal 2018/03/14 08:12:20 [INFO] Creating template helm-kubernetes-traefik 2018/03/14 08:12:21 [INFO] Creating template helm-kubernetes-coredns 2018/03/14 08:12:21 [INFO] Creating template helm-kubernetes-newrelic-infrastructure 2018/03/14 08:12:21 [INFO] Creating template helm-kubernetes-uchiwa 2018/03/14 08:12:21 [INFO] Creating template helm-kubernetes-kube-ops-view 2018/03/14 08:12:22 [INFO] Creating template helm-kubernetes-anchore-engine 2018/03/14 08:12:22 [INFO] Creating template helm-kubernetes-nginx-lego 2018/03/14 08:12:22 [INFO] Creating template helm-kubernetes-linkerd 2018/03/14 08:12:22 [INFO] Creating template helm-kubernetes-aerospike 2018/03/14 08:12:22 [INFO] Creating template helm-kubernetes-zetcd 2018/03/14 08:12:22 [INFO] Creating template helm-kubernetes-datadog 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-artifactory 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-percona 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-buildkite 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-dokuwiki 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-kibana 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-kubed 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-elasticsearch-exporter 2018/03/14 08:12:23 [INFO] Creating template helm-kubernetes-sapho 2018/03/14 08:12:24 [INFO] Creating template helm-kubernetes-minio 2018/03/14 08:12:24 [INFO] Creating template helm-kubernetes-mariadb 2018/03/14 08:12:25 [INFO] Creating template helm-kubernetes-prestashop 2018/03/14 08:12:26 [INFO] Creating template helm-kubernetes-sensu 2018/03/14 08:12:26 [INFO] Creating template helm-kubernetes-sonatype-nexus 2018/03/14 08:12:26 [INFO] Creating template helm-kubernetes-selenium 2018/03/14 08:12:27 [INFO] Creating template helm-kubernetes-redis 2018/03/14 08:12:27 [INFO] Creating template helm-kubernetes-jasperreports 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-gcp-night-king 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-searchlight 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-g2 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-kube-state-metrics 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-kube2iam 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-sumokube 2018/03/14 08:12:28 [INFO] Creating template helm-kubernetes-prometheus 2018/03/14 08:12:31 [INFO] Creating template helm-kubernetes-swift 2018/03/14 08:12:31 [INFO] Creating template helm-kubernetes-phabricator 2018/03/14 08:12:33 [INFO] Creating template helm-kubernetes-ghost 2018/03/14 08:12:34 [INFO] Creating template helm-kubernetes-rethinkdb 2018/03/14 08:12:34 [INFO] Creating template helm-kubernetes-spark 2018/03/14 08:12:35 [INFO] Creating template helm-kubernetes-dask-distributed 2018/03/14 08:12:35 [INFO] Creating template helm-kubernetes-sentry 2018/03/14 08:12:35 [INFO] Creating template helm-kubernetes-memcached 2018/03/14 08:12:35 [INFO] Creating template helm-kubernetes-risk-advisor 2018/03/14 08:12:36 [INFO] Creating template helm-kubernetes-parse 2018/03/14 08:12:37 [INFO] Creating template helm-kubernetes-gcloud-sqlproxy 2018/03/14 08:12:37 [INFO] Creating template helm-kubernetes-kube-lego 2018/03/14 08:12:37 [INFO] Creating template helm-kubernetes-docker-registry 2018/03/14 08:12:37 [INFO] Creating template helm-kubernetes-consul 2018/03/14 08:12:37 [INFO] Creating template helm-kubernetes-opencart 2018/03/14 08:12:38 [INFO] Creating template helm-kubernetes-elastalert 2018/03/14 08:12:38 [INFO] Creating template helm-kubernetes-namerd 2018/03/14 08:12:38 [INFO] Creating template helm-kubernetes-centrifugo 2018/03/14 08:12:38 [INFO] Creating template helm-kubernetes-sumologic-fluentd 2018/03/14 08:12:38 [INFO] Creating template helm-kubernetes-gitlab-ce 2018/03/14 08:12:39 [INFO] Creating template helm-kubernetes-stash 2018/03/14 08:12:39 [INFO] Creating template helm-kubernetes-coscale 2018/03/14 08:12:39 [INFO] Creating template helm-kubernetes-mongodb-replicaset 2018/03/14 08:12:40 [INFO] Creating template helm-kubernetes-rocketchat 2018/03/14 08:12:40 [INFO] Creating template helm-kubernetes-joomla 2018/03/14 08:12:40 [INFO] Creating template helm-kubernetes-jenkins 2018/03/14 08:12:42 [INFO] Creating template helm-kubernetes-wordpress 2018/03/14 08:12:44 [INFO] Creating template helm-kubernetes-openvpn 2018/03/14 08:12:44 [INFO] Creating template helm-kubernetes-owncloud 2018/03/14 08:12:45 [INFO] Creating template helm-kubernetes-heapster 2018/03/14 08:12:45 [INFO] Creating template helm-kubernetes-keel 2018/03/14 08:12:45 [INFO] Creating template helm-kubernetes-hadoop 2018/03/14 08:12:45 [INFO] Creating template helm-kubernetes-prometheus-to-sd 2018/03/14 08:12:45 [INFO] Creating template helm-kubernetes-factorio 2018/03/14 08:12:45 [INFO] Creating template helm-kubernetes-moodle 2018/03/14 08:12:46 [INFO] Creating template helm-kubernetes-mssql-linux 2018/03/14 08:12:46 [INFO] Creating template helm-kubernetes-suitecrm 2018/03/14 08:12:46 [INFO] Creating template helm-kubernetes-testlink 2018/03/14 08:12:47 [INFO] Creating template helm-kubernetes-kubernetes-dashboard 2018/03/14 08:12:47 [INFO] Creating template helm-kubernetes-influxdb 2018/03/14 08:12:47 [INFO] Creating template helm-kubernetes-rabbitmq-ha 2018/03/14 08:12:47 [INFO] Creating template helm-kubernetes-cert-manager 2018/03/14 08:12:47 [INFO] Creating template helm-kubernetes-zeppelin 2018/03/14 08:12:47 [INFO] Creating template helm-kubernetes-external-dns 2018/03/14 08:12:48 [INFO] Creating template helm-kubernetes-neo4j 2018/03/14 08:12:48 [INFO] Creating template helm-kubernetes-mongodb 2018/03/14 08:12:48 [INFO] Creating template helm-kubernetes-acs-engine-autoscaler 2018/03/14 08:12:48 [INFO] Creating template helm-kubernetes-sematext-docker-agent 2018/03/14 08:12:49 [INFO] Creating template helm-kubernetes-mysql 2018/03/14 08:12:49 [INFO] Creating template helm-kubernetes-sysdig 2018/03/14 08:12:49 [INFO] Creating template helm-kubernetes-chronograf 2018/03/14 08:12:50 [INFO] Creating template helm-kubernetes-msoms 2018/03/14 08:12:50 [INFO] Creating template helm-kubernetes-aws-cluster-autoscaler 2018/03/14 08:12:50 [INFO] Creating template helm-kubernetes-cockroachdb 2018/03/14 08:12:50 [INFO] Creating template helm-kubernetes-gcloud-endpoints 2018/03/14 08:12:50 [INFO] Creating template helm-kubernetes-concourse 2018/03/14 08:12:52 [INFO] Creating template helm-kubernetes-nginx-ingress 2018-03-14 08:12:53.395998 I | wal: segmented wal file /var/lib/rancher/etcd/member/wal/0000000000000001-0000000000000885.wal is created 2018/03/14 08:12:53 [INFO] Creating template helm-kubernetes-kong 2018/03/14 08:12:53 [INFO] Creating template helm-kubernetes-weave-cloud 2018/03/14 08:12:53 [INFO] Creating template helm-kubernetes-osclass 2018/03/14 08:12:54 [INFO] Creating template helm-kubernetes-mcrouter 2018/03/14 08:12:54 [INFO] Creating template helm-kubernetes-locust 2018/03/14 08:12:54 [INFO] Creating template helm-kubernetes-pachyderm 2018/03/14 08:12:54 [INFO] Creating template helm-kubernetes-odoo 2018/03/14 08:12:56 [INFO] Creating template helm-kubernetes-percona-xtradb-cluster E0314 08:17:52.895273 1 generic_controller.go:200] ProjectController cluster-b2gg9/project-9dwh2 [mgmt-project-rbac-remove] failed with : Operation cannot be fulfilled on projects.management.cattle.io "project-9dwh2": the object has been modified; please apply your changes to the latest version and try again E0314 08:17:53.058835 1 generic_controller.go:200] ProjectController cluster-b2gg9/project-9dwh2 [mgmt-project-rbac-remove] failed with : Operation cannot be fulfilled on projects.management.cattle.io "project-9dwh2": the object has been modified; please apply your changes to the latest version and try again E0314 08:17:53.061250 1 generic_controller.go:200] ClusterRoleTemplateBindingController cluster-b2gg9/creator [mgmt-auth-crtb-controller] failed with : couldn't create role cluster-owner: roles.rbac.authorization.k8s.io "cluster-owner" already exists E0314 08:17:53.106530 1 generic_controller.go:200] ClusterRoleTemplateBindingController cluster-b2gg9/creator [mgmt-auth-crtb-controller] failed with : Operation cannot be fulfilled on roles.rbac.authorization.k8s.io "cluster-owner": the object has been modified; please apply your changes to the latest version and try again E0314 08:17:53.131482 1 generic_controller.go:200] ClusterController cluster-b2gg9 [cluster-scoped-gc] failed with : Operation cannot be fulfilled on clusters.management.cattle.io "cluster-b2gg9": the object has been modified; please apply your changes to the latest version and try again, [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered 2018/03/14 08:17:53 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered 2018/03/14 08:17:53 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered 2018/03/14 08:17:53 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered 2018-03-14 08:21:36.843129 I | mvcc: store.index: compact 2579 2018-03-14 08:21:36.924387 I | mvcc: finished scheduled compaction at 2579 (took 78.807479ms) 2018/03/14 08:22:59 [INFO] Handling backend connection request. 2018/03/14 08:22:59 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered 2018/03/14 08:22:59 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered 2018/03/14 08:22:59 [INFO] Provisioning cluster [cluster-b2gg9] 2018/03/14 08:22:59 [INFO] Creating cluster [cluster-b2gg9] 2018/03/14 08:22:59 [INFO] cluster [cluster-b2gg9] provisioning: Building Kubernetes cluster 2018/03/14 08:22:59 [INFO] cluster [cluster-b2gg9] provisioning: [dialer] Setup tunnel for host [139.59.189.10] 2018/03/14 08:22:59 [ERROR] cluster [cluster-b2gg9] provisioning: Failed to set up SSH tunneling for host [139.59.189.10]: Unsupported Docker version found [17.09.1-ce], supported versions are [1.12.6 1.13.1 17.03.2] 2018/03/14 08:22:59 [ERROR] cluster [cluster-b2gg9] provisioning: Removing host [139.59.189.10] from node lists E0314 08:22:59.293544 1 generic_controller.go:200] ClusterController cluster-b2gg9 [cluster-provisioner-controller] failed with : Cluster must have at least one etcd plane host 2018/03/14 08:22:59 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 30s 2018/03/14 08:22:59 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 30s 2018/03/14 08:23:29 [INFO] Provisioning cluster [cluster-b2gg9] 2018/03/14 08:23:29 [INFO] Creating cluster [cluster-b2gg9] 2018/03/14 08:23:29 [INFO] cluster [cluster-b2gg9] provisioning: Building Kubernetes cluster 2018/03/14 08:23:29 [INFO] cluster [cluster-b2gg9] provisioning: [dialer] Setup tunnel for host [139.59.189.10] 2018/03/14 08:23:29 [ERROR] cluster [cluster-b2gg9] provisioning: Failed to set up SSH tunneling for host [139.59.189.10]: Unsupported Docker version found [17.09.1-ce], supported versions are [1.12.6 1.13.1 17.03.2] 2018/03/14 08:23:29 [ERROR] cluster [cluster-b2gg9] provisioning: Removing host [139.59.189.10] from node lists E0314 08:23:29.451237 1 generic_controller.go:200] ClusterController cluster-b2gg9 [cluster-provisioner-controller] failed with : Cluster must have at least one etcd plane host 2018/03/14 08:23:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 1m0s 2018/03/14 08:23:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 1m0s 2018/03/14 08:23:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 1m0s 2018/03/14 08:24:29 [INFO] Provisioning cluster [cluster-b2gg9] 2018/03/14 08:24:29 [INFO] Creating cluster [cluster-b2gg9] 2018/03/14 08:24:29 [INFO] cluster [cluster-b2gg9] provisioning: Building Kubernetes cluster 2018/03/14 08:24:29 [INFO] cluster [cluster-b2gg9] provisioning: [dialer] Setup tunnel for host [139.59.189.10] 2018/03/14 08:24:29 [ERROR] cluster [cluster-b2gg9] provisioning: Failed to set up SSH tunneling for host [139.59.189.10]: Can't retrieve Docker Info: error during connect: Get http://%!F(MISSING)var%!F(MISSING)run%!F(MISSING)docker.sock/v1.24/info: failed to find session for client m-e4cc6f165d3f 2018/03/14 08:24:29 [ERROR] cluster [cluster-b2gg9] provisioning: Removing host [139.59.189.10] from node lists E0314 08:24:29.701351 1 generic_controller.go:200] ClusterController cluster-b2gg9 [cluster-provisioner-controller] failed with : Cluster must have at least one etcd plane host 2018/03/14 08:24:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 2m0s 2018/03/14 08:24:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 2m0s 2018/03/14 08:24:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 2m0s 2018/03/14 08:24:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 2m0s 2018/03/14 08:24:36 [INFO] Handling backend connection request. 2018/03/14 08:26:29 [INFO] Provisioning cluster [cluster-b2gg9] 2018/03/14 08:26:29 [INFO] Creating cluster [cluster-b2gg9] 2018/03/14 08:26:29 [INFO] cluster [cluster-b2gg9] provisioning: Building Kubernetes cluster 2018/03/14 08:26:29 [INFO] cluster [cluster-b2gg9] provisioning: [dialer] Setup tunnel for host [139.59.189.10] 2018/03/14 08:26:29 [ERROR] cluster [cluster-b2gg9] provisioning: Failed to set up SSH tunneling for host [139.59.189.10]: Unsupported Docker version found [17.09.1-ce], supported versions are [1.12.6 1.13.1 17.03.2] 2018/03/14 08:26:29 [ERROR] cluster [cluster-b2gg9] provisioning: Removing host [139.59.189.10] from node lists E0314 08:26:29.928685 1 generic_controller.go:200] ClusterController cluster-b2gg9 [cluster-provisioner-controller] failed with : Cluster must have at least one etcd plane host 2018/03/14 08:26:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 4m0s 2018/03/14 08:26:29 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 4m0s 2018/03/14 08:26:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 4m0s 2018/03/14 08:26:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 4m0s 2018-03-14 08:26:36.885360 I | mvcc: store.index: compact 3074 2018-03-14 08:26:36.891605 I | mvcc: finished scheduled compaction at 3074 (took 3.934002ms) E0314 08:30:10.107671 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus 2018/03/14 08:30:29 [INFO] Provisioning cluster [cluster-b2gg9] 2018/03/14 08:30:29 [INFO] Creating cluster [cluster-b2gg9] 2018/03/14 08:30:29 [INFO] cluster [cluster-b2gg9] provisioning: Building Kubernetes cluster 2018/03/14 08:30:30 [INFO] cluster [cluster-b2gg9] provisioning: [dialer] Setup tunnel for host [139.59.189.10] 2018/03/14 08:30:30 [ERROR] cluster [cluster-b2gg9] provisioning: Failed to set up SSH tunneling for host [139.59.189.10]: Unsupported Docker version found [17.09.1-ce], supported versions are [1.12.6 1.13.1 17.03.2] 2018/03/14 08:30:30 [ERROR] cluster [cluster-b2gg9] provisioning: Removing host [139.59.189.10] from node lists E0314 08:30:30.171032 1 generic_controller.go:200] ClusterController cluster-b2gg9 [cluster-provisioner-controller] failed with : Cluster must have at least one etcd plane host 2018/03/14 08:30:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 8m0s 2018/03/14 08:30:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 8m0s 2018/03/14 08:30:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 8m0s 2018/03/14 08:30:30 [INFO] ClusterController cluster-b2gg9 completed with dropped err: [cluster-provisioner-controller] failed with : backing off failure, delay: 8m0s 2018-03-14 08:31:36.901640 I | mvcc: store.index: compact 3554 2018-03-14 08:31:36.908311 I | mvcc: finished scheduled compaction at 3554 (took 3.163853ms)

zionwu commented 6 years ago

2018/03/14 08:26:29 [ERROR] cluster [cluster-b2gg9] provisioning: Failed to set up SSH tunneling for host [139.59.189.10]: Unsupported Docker version found [17.09.1-ce], supported versions are [1.12.6 1.13.1 17.03.2]

@marufbd the above message indicates the docker version on the host is not supported. Please use the supported version.

marufbd commented 6 years ago

Thanks a lot @zionwu It seems then digital ocean CoreOS container distribution none of them have any suported docker version. https://coreos.com/releases/

I ended up using Rancher OS for the custom node and using the script: wget https://releases.rancher.com/install-docker/17.03.2.sh -O - | sh to make the docker version supported.

And it's working great!

farajfarook commented 6 years ago

An easier solution would be to cloud-init the core os with 1.12 version of docker.

storage:
  files:
    - filesystem: root
      path: /etc/coreos/docker-1.12
      contents:
        inline: yes

Refer the official documentation https://coreos.com/blog/toward-docker-17-in-container-linux

Remigius2011 commented 6 years ago

@marufbd How did you get these logs? Did you ssh into the droplets - if so, how? I can't login via the DO ssh console (don't know the root password - if there is one at all) nor via the ssh command line (using the public keys from rancher ui gives me Permission denied (publickey).). In my case it could either be a docker version issue (I'm running rancher on 18.03.0-ce) or a communication problem (droplets can only access rancher via https / rancher cannot ssh into droplets).

marufbd commented 6 years ago

@Remigius2011 the logs here are from two nodes:

  1. Rancher server node which you must have already access, otherwise you cannot even launch rancher
  2. Custom node which also you must already have access otherwise you cannot bring that under the kubernetes cluster

However, if you launch your cluster from rancher, you can always download the keys for a node from the cluster->nodes list. Once downloaded and extracted the zip use the id_rsa file as your key file when doing ssh as: ssh -i id_rsa rancher@node_ip Here my node was rancher OS hence the root password is rancher, for coreos the root password would be core and otherwise its root in Digital Ocean

Remigius2011 commented 6 years ago

@marufbd Thanks for the information. Actually, I was able to have a root password sent to me by email (droplet settings -> Access, then push Reset root password). It looks like docker is not installed on the droplet. Does rancher ssh into the droplets once created?