search

contiv-experimental / cluster

cluster management infrastructure using serf, collins and ansible!
Other
19 stars 9 forks source link

UCP controller node re-commissioning results in controller state failure #149

Closed rkharya closed 8 years ago

rkharya commented 8 years ago

Docker - 1.11.1.cs2 UCP - 1.1.0 Contiv_cluster - v0.1-05-14-2016.00-33-02.UTC

Problem - 3 node UCP master setup. Docker-2/3/4 were commissioned as service-master and UCP dashboard displaying all 3 controller as healthy. Node Docker-4 was de-commissioned from Contive-Cluster and recommissioned again. After successful re-commissioning UCP services failed to start on it and UCP dashboard reporting this node in failed state.

steps -

  1. Commissioned 3 nodes as service-master.
  2. All 3 successfully commissioned and Docker-2 became etcd 'Leader' rest 2 slaves
  3. De-commissioned 3rd node Docker-4 from contiv-cluster
  4. Did CPU replacement only leaving internal disk un-touched
  5. Rebooted node previously installed OS come up cleanly with all the required services up and running
  6. Re-commissioned the node into Contiv_Cluster, it got successfully re-commissioned
  7. UCP containers and service is in failed state
  8. UCP dashboard reports state as 'Failed' for this re-commissioned node

[cluster-admin@Docker-2 ~]$ etcdctl member list c5583e158c122eef: name=Docker-2-FLM19379EU8 peerURLs=http://10.65.122.65:2380,http://10.65.122.65:7001 clientURLs=http://10.65.122.65:2379,http://10.65.122.65:4001 isLeader=false c69f2991ffeeb3a6: name=Docker-3-FCH19517CF9 peerURLs=http://10.65.122.66:2380,http://10.65.122.66:7001 clientURLs=http://10.65.122.66:2379,http://10.65.122.66:4001 isLeader=true

[cluster-admin@Docker-2 ~]$ ifconfig -a|grep enp6s0_0 enp6s0_0: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500

[cluster-admin@Docker-4 ~]$ etcdctl member list aae67246ba4f3b45: name=Docker-4-FCH19517CER peerURLs=http://10.65.122.67:2380,http://10.65.122.67:7001 clientURLs=http://10.65.122.67:2379,http://10.65.122.67:4001 isLeader=true

[cluster-admin@Docker-4 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 94499aa40c13 quay.io/coreos/etcd:v2.3.1 "/etcd" 18 hours ago Up 18 hours etcd 3af4192f05bb skynetservices/skydns:latest "/skydns" 44 hours ago Up 18 hours 53/tcp, 53/udp defaultdns

[cluster-admin@Docker-4 ~]$ sudo systemctl status -l -n 1000 ucp.service ● ucp.service - Ucp Loaded: loaded (/etc/systemd/system/ucp.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2016-06-02 18:27:07 IST; 18h ago Process: 30039 ExecStop=/usr/bin/ucp.sh stop (code=exited, status=0/SUCCESS) Process: 28442 ExecStart=/usr/bin/ucp.sh start (code=exited, status=1/FAILURE) Main PID: 28442 (code=exited, status=1/FAILURE)

Jun 02 18:26:21 Docker-4.cisco.com systemd[1]: Started Ucp. Jun 02 18:26:21 Docker-4.cisco.com systemd[1]: Starting Ucp... Jun 02 18:26:21 Docker-4.cisco.com ucp.sh[28442]: starting ucp on Docker-4-FCH19517CER[10.65.122.67] Jun 02 18:26:22 Docker-4.cisco.com ucp.sh[28442]: INFO[0000] Your engine version 1.11.1-cs2 is compatible Jun 02 18:26:22 Docker-4.cisco.com ucp.sh[28442]: WARN[0000] Your system uses devicemapper. We can not accurately detect available storage space. Please make sure you have at least 3.00 GB available in /var/lib/docker Jun 02 18:26:24 Docker-4.cisco.com ucp.sh[28442]: INFO[0002] All required images are present Jun 02 18:26:25 Docker-4.cisco.com ucp.sh[28442]: INFO[0000] This engine will join UCP and advertise itself with host address 10.65.122.67 - If this is incorrect, please specify an alternative address with the '--host-address' flag Jun 02 18:26:25 Docker-4.cisco.com ucp.sh[28442]: INFO[0000] Verifying your system is compatible with UCP Jun 02 18:26:25 Docker-4.cisco.com ucp.sh[28442]: INFO[0000] Checking that required ports are available and accessible Jun 02 18:27:02 Docker-4.cisco.com ucp.sh[28442]: INFO[0037] Starting local swarm containers Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: INFO[0040] Starting UCP Controller replica containers Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: ERRO[0040] Server response: {"message":"etcdserver: peerURL exists"} Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: ERRO[0040] Failed to start KV store. Run "docker logs ucp-kv" for more details Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: FATA[0040] Failed to add member to KV store: {"message":"etcdserver: peerURL exists"} Jun 02 18:27:06 Docker-4.cisco.com systemd[1]: ucp.service: main process exited, code=exited, status=1/FAILURE Jun 02 18:27:06 Docker-4.cisco.com ucp.sh[30039]: 986ef385fd95 Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: 2d60e11357c4 Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: 986ef385fd95 Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: 2d60e11357c4 Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-auth-api-certs Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-auth-store-certs Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-auth-store-data Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-auth-worker-certs Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-auth-worker-data Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-client-root-ca Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-cluster-root-ca Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-controller-client-certs Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-controller-server-certs Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-kv-certs Jun 02 18:27:07 Docker-4.cisco.com ucp.sh[30039]: ucp-node-certs Jun 02 18:27:07 Docker-4.cisco.com systemd[1]: Unit ucp.service entered failed state. Jun 02 18:27:07 Docker-4.cisco.com systemd[1]: ucp.service failed.

[cluster-admin@Docker-1 ~]$ clusterctl nodes get | egrep "inventory_name|status" Ceph-1-FCH1936V1EJ: prev_status: Unallocated Ceph-1-FCH1936V1EJ: status: Unallocated Ceph-1-FCH1936V1EJ: inventory_name: Ceph-1-FCH1936V1EJ Ceph-2-FCH1936V1EX: prev_status: Unallocated Ceph-2-FCH1936V1EX: status: Unallocated Ceph-2-FCH1936V1EX: inventory_name: Ceph-2-FCH1936V1EX Ceph-3-FCH1936V1EZ: prev_status: Unallocated Ceph-3-FCH1936V1EZ: status: Unallocated Ceph-3-FCH1936V1EZ: inventory_name: Ceph-3-FCH1936V1EZ Docker-1-FLM19379EUC: prev_status: Provisioning Docker-1-FLM19379EUC: status: Allocated Docker-1-FLM19379EUC: inventory_name: Docker-1-FLM19379EUC Docker-2-FLM19379EU8: prev_status: Allocated Docker-2-FLM19379EU8: status: Allocated Docker-2-FLM19379EU8: inventory_name: Docker-2-FLM19379EU8 Docker-3-FCH19517CF9: prev_status: Allocated Docker-3-FCH19517CF9: status: Allocated Docker-3-FCH19517CF9: inventory_name: Docker-3-FCH19517CF9 Docker-4-FCH19517CER: prev_status: Provisioning Docker-4-FCH19517CER: status: Allocated Docker-4-FCH19517CER: inventory_name: Docker-4-FCH19517CER Docker-5-FCH19517CAT: prev_status: Allocated Docker-5-FCH19517CAT: status: Allocated Docker-5-FCH19517CAT: inventory_name: Docker-5-FCH19517CAT Docker-6-FCH1945JJ4F: prev_status: Allocated Docker-6-FCH1945JJ4F: status: Allocated Docker-6-FCH1945JJ4F: inventory_name: Docker-6-FCH1945JJ4F [cluster-admin@Docker-1 ~]$ clusterctl job get last

Description: commissionEvent: nodes:[Docker-4-FCH19517CER] extra-vars:{} host-group:service-master Status: Complete Error: Logs: [DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [devtest] *****************************************************************
skipping: no hosts matched

PLAY [volplugin-test] **********************************************************
skipping: no hosts matched

PLAY [cluster-node] ************************************************************
skipping: no hosts matched

PLAY [cluster-control] *********************************************************
skipping: no hosts matched

PLAY [service-master] **********************************************************

TASK [setup] *******************************************************************
ok: [Docker-4-FCH19517CER]

TASK [base : include] **********************************************************
skipping: [Docker-4-FCH19517CER]

TASK [base : include] **********************************************************
included: /home/cluster-admin/ansible/roles/base/tasks/redhat_tasks.yml for Docker-4-FCH19517CER

TASK [base : install epel release package (redhat)] ****************************
ok: [Docker-4-FCH19517CER]

TASK [base : install/upgrade base packages (redhat)] ***************************
ok: [Docker-4-FCH19517CER] => (item=[u'yum-utils', u'ntp', u'unzip', u'bzip2', u'curl', u'python-requests', u'bash-completion', u'kernel', u'libselinux-python'])

TASK [base : install and start ntp] ********************************************
changed: [Docker-4-FCH19517CER]

TASK [base : include] **********************************************************
included: /home/cluster-admin/ansible/roles/base/tasks/os_agnostic_tasks.yml for Docker-4-FCH19517CER

TASK [base : download consul binary] *******************************************
ok: [Docker-4-FCH19517CER]

TASK [base : install consul] ***************************************************
ok: [Docker-4-FCH19517CER]

TASK [ucarp : download and install ucarp service (Redhat)] *********************
ok: [Docker-4-FCH19517CER]

TASK [ucarp : download and install ucarp service (Ubuntu)] *********************
skipping: [Docker-4-FCH19517CER]

TASK [ucarp : copy the ucarp start/stop script] ********************************
ok: [Docker-4-FCH19517CER]

TASK [ucarp : copy the vip up and down scripts used by ucarp] ******************
ok: [Docker-4-FCH19517CER]

TASK [ucarp : copy systemd units for ucarp] ************************************
ok: [Docker-4-FCH19517CER]

TASK [ucarp : start ucarp] *****************************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : check docker version] *******************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : include] ********************************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : include] ********************************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : create docker daemon's config directory] ************************
ok: [Docker-4-FCH19517CER]

TASK [docker : setup docker daemon's environment] ******************************
ok: [Docker-4-FCH19517CER]

TASK [docker : setup iptables for docker] **************************************
changed: [Docker-4-FCH19517CER] => (item=2385)

TASK [docker : copy systemd units for docker(enable cluster store) (debian)] ***
skipping: [Docker-4-FCH19517CER]

TASK [docker : copy systemd units for docker(enable cluster store) (redhat)] ***
ok: [Docker-4-FCH19517CER]

TASK [docker : check docker-tcp socket state] **********************************
changed: [Docker-4-FCH19517CER]

TASK [docker : include] ********************************************************
included: /home/cluster-admin/ansible/roles/docker/tasks/create_docker_device.yml for Docker-4-FCH19517CER

TASK [docker : pvcreate check for /dev/sdb] ************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : pvcreate /dev/sdb] **********************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : vgcreate check for /dev/sdb] ************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : vgcreate contiv] ************************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : lvcreate check for /dev/sdb] ************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : lvcreate contiv-dockerthin] *************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : copy systemd units for docker tcp socket settings] **************
ok: [Docker-4-FCH19517CER]

TASK [docker : reload systemd configuration] ***********************************
changed: [Docker-4-FCH19517CER]
 [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather
than running sudo

TASK [docker : stop docker] ****************************************************
ok: [Docker-4-FCH19517CER]

TASK [docker : start docker-tcp service] ***************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : check docker service state] *************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : remove the docker key file, if any. It shall be regenerated by docker on restart] ***
changed: [Docker-4-FCH19517CER]

TASK [docker : reload docker systemd configuration] ****************************
changed: [Docker-4-FCH19517CER]

TASK [docker : restart docker (first time)] ************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : ensure docker is started] ***************************************
changed: [Docker-4-FCH19517CER]

TASK [docker : stat] ***********************************************************
ok: [Docker-4-FCH19517CER]

TASK [docker : Import saved docker images] *************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : check docker-compose version] ***********************************
changed: [Docker-4-FCH19517CER]

TASK [docker : download and install docker-compose] ****************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : check contiv-compose version] ***********************************
changed: [Docker-4-FCH19517CER]

TASK [docker : download contiv-compose] ****************************************
skipping: [Docker-4-FCH19517CER]

TASK [docker : install contiv-compose] *****************************************
skipping: [Docker-4-FCH19517CER]

TASK [etcd : download etcdctl v2.3.1] ******************************************
ok: [Docker-4-FCH19517CER]

TASK [etcd : install etcdctl] **************************************************
changed: [Docker-4-FCH19517CER]
 [WARNING]: Consider using unarchive module rather than running tar

TASK [etcd : install etcd v2.3.1] **********************************************
changed: [Docker-4-FCH19517CER]

TASK [etcd : setup iptables for etcd] ******************************************
changed: [Docker-4-FCH19517CER] => (item=2379)
changed: [Docker-4-FCH19517CER] => (item=4001)
changed: [Docker-4-FCH19517CER] => (item=2380)
changed: [Docker-4-FCH19517CER] => (item=7001)

TASK [etcd : copy the etcd start/stop script] **********************************
changed: [Docker-4-FCH19517CER]

TASK [etcd : copy systemd units for etcd] **************************************
ok: [Docker-4-FCH19517CER]

TASK [etcd : start etcd] *******************************************************
changed: [Docker-4-FCH19517CER]

TASK [swarm : check for swarm image] *******************************************
skipping: [Docker-4-FCH19517CER]

TASK [swarm : download swarm container image] **********************************
skipping: [Docker-4-FCH19517CER]

TASK [swarm : setup iptables for swarm] ****************************************
skipping: [Docker-4-FCH19517CER] => (item=2375)

TASK [swarm : copy the swarm start/stop script] ********************************
skipping: [Docker-4-FCH19517CER]

TASK [swarm : copy systemd units for swarm] ************************************
skipping: [Docker-4-FCH19517CER]

TASK [swarm : start swarm] *****************************************************
skipping: [Docker-4-FCH19517CER]

TASK [ucp : download and install ucp images] ***********************************
changed: [Docker-4-FCH19517CER]

TASK [ucp : setup iptables for ucp] ********************************************
changed: [Docker-4-FCH19517CER] => (item=12376)
changed: [Docker-4-FCH19517CER] => (item=12379)
changed: [Docker-4-FCH19517CER] => (item=12380)
changed: [Docker-4-FCH19517CER] => (item=12381)
changed: [Docker-4-FCH19517CER] => (item=12382)
changed: [Docker-4-FCH19517CER] => (item=12383)
changed: [Docker-4-FCH19517CER] => (item=12384)
changed: [Docker-4-FCH19517CER] => (item=12385)
changed: [Docker-4-FCH19517CER] => (item=12386)
changed: [Docker-4-FCH19517CER] => (item=2375)
changed: [Docker-4-FCH19517CER] => (item=2376)
changed: [Docker-4-FCH19517CER] => (item=443)

TASK [ucp : copy the ucp license to the remote machine] ************************
skipping: [Docker-4-FCH19517CER]

TASK [ucp : copy the ucp start/stop script] ************************************
ok: [Docker-4-FCH19517CER]

TASK [ucp : copy systemd units for ucp] ****************************************
ok: [Docker-4-FCH19517CER]

TASK [ucp : start ucp] *********************************************************
changed: [Docker-4-FCH19517CER]

TASK [ucp : create a local fetch directory if it doesn't exist] ****************
ok: [Docker-4-FCH19517CER -> localhost]

TASK [ucp : wait for ucp files to be created, which ensures the service has started] ***
skipping: [Docker-4-FCH19517CER] => (item=ucp-fingerprint)
skipping: [Docker-4-FCH19517CER] => (item=ucp-instance-id)

TASK [ucp : fetch the ucp files from master nodes] *****************************
skipping: [Docker-4-FCH19517CER] => (item=ucp-fingerprint)
skipping: [Docker-4-FCH19517CER] => (item=ucp-instance-id)

TASK [ucp : copy the ucp files to replicas and worker nodes] *******************
changed: [Docker-4-FCH19517CER] => (item=ucp-fingerprint)
changed: [Docker-4-FCH19517CER] => (item=ucp-instance-id)

TASK [contiv_network : check dns container image] ******************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : pull dns container image] *******************************
skipping: [Docker-4-FCH19517CER]

TASK [contiv_network : include] ************************************************
included: /home/cluster-admin/ansible/roles/contiv_network/tasks/ovs.yml for Docker-4-FCH19517CER

TASK [contiv_network : download ovs binaries (redhat)] *************************
ok: [Docker-4-FCH19517CER] => (item={u'url': u'https://cisco.box.com/shared/static/zzmpe1zesdpf270k9pml40rlm4o8fs56.rpm', u'dest': u'/tmp/openvswitch-2.3.1-2.el7.x86_64.rpm'})

TASK [contiv_network : install ovs (redhat)] ***********************************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : download ovs binaries (debian)] *************************
skipping: [Docker-4-FCH19517CER] => (item={u'url': u'https://cisco.box.com/shared/static/v1dvgoboo5zgqrtn6tu27vxeqtdo2bdl.deb', u'dest': u'/tmp/ovs-common.deb'})
skipping: [Docker-4-FCH19517CER] => (item={u'url': u'https://cisco.box.com/shared/static/ymbuwvt2qprs4tquextw75b82hyaxwon.deb', u'dest': u'/tmp/ovs-switch.deb'})

TASK [contiv_network : install ovs-common (debian)] ****************************
skipping: [Docker-4-FCH19517CER]

TASK [contiv_network : install ovs (debian)] ***********************************
skipping: [Docker-4-FCH19517CER]

TASK [contiv_network : start ovs service] **************************************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : setup ovs] **********************************************
changed: [Docker-4-FCH19517CER] => (item=tcp:127.0.0.1:6640)
changed: [Docker-4-FCH19517CER] => (item=ptcp:6640)

TASK [contiv_network : check selinux status] ***********************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : permit openvswitch_t type in selinux] *******************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : setup iptables for vxlan vtep port] *********************
changed: [Docker-4-FCH19517CER] => (item=4789)

TASK [contiv_network : setup iptables for contiv network control plane] ********
changed: [Docker-4-FCH19517CER] => (item=9001)
changed: [Docker-4-FCH19517CER] => (item=9002)
changed: [Docker-4-FCH19517CER] => (item=9003)
changed: [Docker-4-FCH19517CER] => (item=9999)
changed: [Docker-4-FCH19517CER] => (item=8080)
changed: [Docker-4-FCH19517CER] => (item=179)

TASK [contiv_network : download netmaster and netplugin] ***********************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : ensure netplugin directory exists] **********************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : install netmaster and netplugin] ************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : create links for netplugin binaries] ********************
ok: [Docker-4-FCH19517CER] => (item=netctl)
ok: [Docker-4-FCH19517CER] => (item=netmaster)
ok: [Docker-4-FCH19517CER] => (item=netplugin)
ok: [Docker-4-FCH19517CER] => (item=contivk8s)

TASK [contiv_network : copy environment file for netplugin] ********************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : copy systemd units for netplugin] ***********************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : copy bash auto complete file for netctl] ****************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : start netplugin] ****************************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : setup netmaster host alias] *****************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : copy environment file for netmaster] ********************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : copy systemd units for netmaster] ***********************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : start netmaster] ****************************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : download contivctl] *************************************
ok: [Docker-4-FCH19517CER]

TASK [contiv_network : install contivctl] **************************************
changed: [Docker-4-FCH19517CER]

TASK [contiv_network : include] ************************************************
skipping: [Docker-4-FCH19517CER]

PLAY [service-worker] **********************************************************
skipping: no hosts matched

PLAY [netplugin-node] **********************************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************
Docker-4-FCH19517CER       : ok=72   changed=36   unreachable=0    failed=0
mapuri commented 8 years ago

@rkharya

looks like UCP is not able to add the re-commissioned node to it's internal KV store:

Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: INFO[0040] Starting UCP Controller replica containers
Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: ERRO[0040] Server response: {"message":"etcdserver: peerURL exists"}
Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: ERRO[0040] Failed to start KV store. Run "docker logs ucp-kv" for more details
Jun 02 18:27:05 Docker-4.cisco.com ucp.sh[28442]: FATA[0040] Failed to add member to KV store: {"message":"etcdserver: peerURL exists"}

Right now when we decommission a node we stop UCP containers as based on my understanding of UCP installer there is no option to cleanly take a node out of UCP cluster.

Unless this is a UCP bug, my guess is that the current way of decommissioning might be leading UCP into trouble. In any case we will need to bring this issue up with Docker folks and ask them for recommended steps for cleanly removing a node from UCP cluster.

rkharya commented 8 years ago

@mapuri Informed Uday about this.

mapuri commented 8 years ago

closing this as not a cluster issue, needs to be tracked with UCP folks.