Adding/removing a master node to the cluster is kicking off a whole cluster restart (canary deployment)

[lgunta2018]

What happened: Adding/removing a master node to the cluster is kicking off the whole cluster restart (canary deployment)

What you expected to happen: Adding/removing a master node should not kick off the whole cluster restart

How to reproduce it (as minimally and precisely as possible):

Steps:

1. Create a cluster with 3 master and 3 worker nodes
2. Update the number of master nodes instances to 2 in cfcr.yml file
3. Run the bosh deployment command, you can see the whole cluster restart

runlog:

Removing master node:

bosh deploy -d cfcr ${KD}/manifests/cfcr.yml -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml -o cfcr-ops.yml -l <(bbl outputs) Using environment 'https://10.0.0.6:25555' as client 'admin'

Using deployment 'cfcr'

Release 'cfcr-etcd/1.5.0' already exists.

Release 'bpm/0.12.3' already exists.

addons:

• include: stemcells:
  • os: ubuntu-trusty
  • os: ubuntu-xenial name: bosh-dns
• jobs:
  • name: kubo-dns-aliases release: kubo name: bosh-dns-aliases features: use_dns_addresses: true instance_groups:
• azs:
  • z1 instances: 1 jobs:
  • name: apply-specs properties: addons:
    • kube-dns
    • metrics-server
    • heapster
    • kubernetes-dashboard admin-password: ((kubo-admin-password)) admin-username: admin api-token: ((kubelet-password)) tls: heapster: ((tls-heapster)) influxdb: ((tls-influxdb)) kubernetes: ((tls-kubernetes)) kubernetes-dashboard: ((tls-kubernetes-dashboard)) metrics-server: ((tls-metrics-server)) release: kubo lifecycle: errand name: apply-addons networks:
  • name: default stemcell: default "kubo-deployment/manifests/cfcr.yml" 297L, 7524C type: certificate
• name: tls-etcdctl options: ca: kubo_ca common_name: etcdClient extended_key_usage:
  • client_auth type: certificate
• name: tls-metrics-server options: alternative_names:
  • metrics-server.kube-system.svc ca: kubo_ca common_name: metrics-server type: certificate
• name: tls-heapster options: alternative_names:
  • heapster.kube-system.svc.cluster.local ca: kubo_ca common_name: heapster type: certificate
• name: tls-influxdb options: alternative_names: [] ca: kubo_ca common_name: monitoring-influxdb type: certificate
• name: kubernetes-dashboard-ca options: common_name: ca is_ca: true type: certificate
• name: tls-kubernetes-dashboard options: alternative_names: [] ca: kubernetes-dashboard-ca common_name: kubernetesdashboard.cfcr.internal type: certificate ?instances certificate: ((tls-etcdctl.certificate)) Release 'bosh-dns/1.8.0' already exists.

Release 'docker/32.0.0' already exists.

instance_groups:

• name: master
  • instances: 3
  • instances: 2

Continue? [yN]: y

Task 120

Task 120 | 23:47:07 | Preparing deployment: Preparing deployment (00:00:07) Task 120 | 23:47:39 | Preparing package compilation: Finding packages to compile (00:00:00) Task 120 | 23:47:39 | Deleting unneeded instances master: master/a02c36fb-a6d0-4ca9-8178-8a929087d32e (2) (00:00:43) Task 120 | 23:48:22 | Updating instance master: master/ba1fbbda-5080-4094-b8c8-671d9abb34a6 (0) (canary) (00:01:05) Task 120 | 23:49:27 | Updating instance master: master/c5dd8b42-ee9e-4607-ad44-152144a7eebf (1) (00:01:22) Task 120 | 23:50:49 | Updating instance worker: worker/43ac3278-5a5b-4a9b-a782-e3b52254f98d (0) (canary) (00:00:33) Task 120 | 23:51:22 | Updating instance worker: worker/41823365-c567-4acc-a2c3-d3df897ee8b3 (1) (00:00:35) Task 120 | 23:51:57 | Updating instance worker: worker/598eec97-2305-4eac-a784-2fee04d6121b (2) (00:00:41)

Same behavior is observed for adding a new master node:

→ bosh deploy -d cfcr ${KD}/manifests/cfcr.yml -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml -o cfcr-ops.yml -l <(bbl outputs) Using environment 'https://10.0.0.6:25555' as client 'admin'

Using deployment 'cfcr'

Release 'bpm/0.12.3' already exists.

Release 'docker/32.0.0' already exists.

Release 'cfcr-etcd/1.5.0' already exists.

Release 'bosh-dns/1.8.0' already exists.

instance_groups:

• name: master
  • instances: 2
  • instances: 3

Continue? [yN]: y

Task 234

Task 234 | 00:10:30 | Preparing deployment: Preparing deployment (00:00:06) Task 234 | 00:11:07 | Preparing package compilation: Finding packages to compile (00:00:00) Task 234 | 00:11:07 | Creating missing vms: master/09e9e7e6-7e1f-4d51-b591-1c0f1e8ca12d (2) (00:01:27) Task 234 | 00:12:34 | Updating instance master: master/ba1fbbda-5080-4094-b8c8-671d9abb34a6 (0) (canary) (00:01:07) Task 234 | 00:13:41 | Updating instance master: master/c5dd8b42-ee9e-4607-ad44-152144a7eebf (1) (00:01:05) Task 234 | 00:14:46 | Updating instance master: master/09e9e7e6-7e1f-4d51-b591-1c0f1e8ca12d (2) (00:01:27) Task 234 | 00:16:13 | Updating instance worker: worker/43ac3278-5a5b-4a9b-a782-e3b52254f98d (0) (canary) (00:00:33) Task 234 | 00:16:46 | Updating instance worker: worker/41823365-c567-4acc-a2c3-d3df897ee8b3 (1) (00:00:42) Task 234 | 00:17:28 | Updating instance worker: worker/598eec97-2305-4eac-a784-2fee04d6121b (2) (00:00:35)

Task 234 Started Sat Sep 15 00:10:30 UTC 2018 Task 234 Finished Sat Sep 15 00:18:03 UTC 2018 Task 234 Duration 00:07:33 Task 234 done

Anything else we need to know?: Adding worker nodes is working fine, it is not restarting the whole cluster. kube-deployment : v0.21.0

Environment:

• Deployment Info (bosh -d <deployment> deployment): bosh deploy -d cfcr ${KD}/manifests/cfcr.yml -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml -o cfcr-ops.yml -l <(bbl outputs)

ame Release(s) Stemcell(s) Config(s) Team(s) cfcr bosh-dns/1.8.0 bosh-aws-xen-hvm-ubuntu-xenial-go_agent/97.16 1 cloud/default - bpm/0.12.3 2 runtime/dns cfcr-etcd/1.5.0 docker/32.0.0 kubo/0.21.0

• Environment Info (bosh -e <environment> environment): Name bosh-cfcr-lgunta UUID 58cd4562-b14b-4310-aaa3-bd1582250f34 Version 267.5.0 (00000000) CPI aws_cpi Features compiled_package_cache: disabled config_server: enabled dns: disabled snapshots: disabled User admin

• Kubernetes version (kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-10T11:44:36Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider (e.g. aws, gcp, vsphere): AWS

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/160542355

The labels on this github issue will be updated when the story is started.

[alex-slynko]

Hi @lgunta2018

This is intended behaviour right now with current way CFCR is deployed. Each master is collocated with etcd node. Each worker uses flannel networking that is connected to the etcd. Flannel is reconfigured each time etcd node is added or removed. Bosh team is considering improving the flow so that only single job will get restarted, but this is not in their short or mid-term plans.

Out of curiosity, what was the reason to scale masters down to 2 VMs?

[lgunta2018]

Hi @alex-slynko Thanks for the quick update, just to see how the cluster reacts when we remove a master node. So, if we lose the master node due to some issue, it means it restart the whole cluster to bring a new master node in place of the existing master node? Why do we need to restart the whole cluster when we adding a master node to the cluster?

[lgunta2018]

Hey @alex-slynko , Thank you for the response, however we would like our apps deployed in the workers not to be restarted while handling a master node scale down/up/. Our expectation was borne out of the fact that though this is a control plane update, it shouldn't need to restart other parts of the control plane like the workers. Would it be possible to externalize the etcd from the master nodes (create etcd cluster with LB) and would that solve this issue of a rolling restart across the cluster? All our workloads are not cloud native apps and high have stateful sessions which we would like to keep from restarting unless necessary to reduce downtime on those apps.

[youreddy]

@lgunta2018,

if we lose the master node due to some issue, it means it restart the whole cluster to bring a new master node in place of the existing master node?

It depends, if bosh resurrector has noticed one of your master vms has gone missing then it will recreate it without touching the workers. Also, say your manifest states master instances: 3 but one of the masters is failing, in this case bosh hasn't seen a change to the instances count and it won't re-template the jobs and touch the workers

Why do we need to restart the whole cluster when we adding a master node to the cluster?

Basically what Oleksandr said above. flanneld is running on every vm in the cluster. The flannel job consumes etcd bosh links and iterates through the list of etcd instances. So every time the number of etcds change (aka the number of masters since they're colocated), the flannel job will get re-templated on every vm and bosh will update the instances.

Would it be possible to externalize the etcd from the master nodes (create etcd cluster with LB) and would that solve this issue of a rolling restart across the cluster?

Externalizing etcd would be a larger architectural change and may not necessarily solve the problem. We might be able to solve this in simpler way by not iterating through the list of etcd links and configuring flanneld to use the etcd bosh dns entry. We discussed this approach this morning but we need to spike it out because there's probably more than just those etcd links alone. There's a spike in our public tracker if you want to follow progress.

[alex-slynko]

Hi @lgunta2018

1. If you want to test how the cluster react to removing master node, you need to use delete-vm or stop command. I wrote a tiny blog post where I tried to explain the difference.
2. We created a spike to investigate it more, but I can't guarantee we will work on it or fix it soon. It might require big redesign for CFCR or some architectural Bosh changes. We might prioritize it if we see a business value in improving it.

There are two workarounds for this I can think of, but we haven't tried them

• create external load balancer and use manual links to pass traffic through this load balancer.
  • split deployments into two: control plane and workers and use cross-deployment links

Feel free to ask more questions in Slack channel or here.

[lgunta2018]

Thanks, @youreddy && @alex-slynko for looking into this issue. I will try to use your workarounds to solve this problem for now. But its very useful feature in my case. I hope we will see some traction in this regard. I will let you know if the workarounds do not work for me.