search

colemickens / azure-kubernetes-status

Status of Kubernetes on Azure (DEPRECATED! See https://github.com/Azure/ACS)
28 stars 2 forks source link

Clarify limitations of Azure Load Balancers in Kubernetes in combination with Availabilitty Sets #11

Closed codablock closed 7 years ago

codablock commented 7 years ago

Today I got two different errors when trying to use the Azure Cloud Provider and a service of type LoadBalancer. Unfortunately I did not make a copy of the log output. Basically the errors told me that there are limitations when using LBs in combination with Availability Sets.

In my current deployment, I have 2 Availability Sets. One for the master nodes and one for the minion nodes. If I understand the cloud provider correctly, it tries to add all nodes to the Azure LB and then fails because the nodes are not in the same Availability Sets.

Can someone clarify the limitations here? Would it be possible to have more control about the behaviour of the Azure cloud provider? For example, only attach nodes to the LB when they have a desired label set? Or would it be possible that the cloud provider creates one LB per Availability Set?

edevil commented 7 years ago

I also have different availability sets for master and node vms, and it works fine. Only the node vms need to be added to the load-balancer, unless you expose services on the master nodes as well.

codablock commented 7 years ago

@edevil How do you ensure only the node vms are added to the LB? In my case, it tried to add the master nodes as well.

colemickens commented 7 years ago

Mark the master node as --register-node=true and --register-schedulabe=false and you will avoid this problem.

colemickens commented 7 years ago

Can I close this out?

codablock commented 7 years ago

Yepp, can be closed (will do so after the comment).

I could however not really try the proposed solution as kubeadm fails to finish the init phase when the master is not schedulable.

colemickens commented 7 years ago

This is the first mention of kubeadm. What are you using kubeadm for? Why would you run kubeadm init when the master is already up? etc?

codablock commented 7 years ago

I'm currently working on an Ansible+Kubeadm+Azure solution which I'm going to release in the next days. I run "kubeadm init" with the Azure cloud provider correctly set up. When kubeadm init sets up the master, it does a "test deployment" to figure out if everything went fine. This however requires a schedulable node, which is assumed to be the master itself. If I make the master unschedulable, the init waits forever as there are no joined nodes yet.

colemickens commented 7 years ago

In that case I think your only option is to ensure the master and nodes are in the same availability set.

Also, have you seen the ACS-Engine that we open sourced last week?