allow creation of clusters w/o ingress controller load balancer

[r7vme]

Azure does not allow to associate more than ONE load balancer per VM (per NIC). By default azure-operator creates ingress load balancer attached to all worker VMs. So only one load balancer slot already occupied, so users will not be able to create azure load balancer via kubernetes.

This bug just opens a discussion.

I have two questions: 1) Do we want to support ability to create Azure load balancers via kubernetes? Of not this bug can be closed. 2) If yes, how we want to design solution?

cc: @puja108 @kopiczko

[puja108]

This is interesting, for now I would say IC is more important than service LBs, but we should talk to MS if there's another way.

My team will anyway work on managed services inside guest clusters in Q1 so we might make IC optional/customizable anyway and then we might just open up option for starting cluster without IC.

[puja108]

/cc @teemow @marians as this is sth of kinda high impact

[xh3b4sd]

This is lying around in the SIG operator backlog. Any movement? What can we do? Should we just ignore it?

[kopiczko]

We can

• Add more NICs to all machines. Like 11, and tell customers they can create up to 10 LBs per cluster. (simple)
• Write kind of crazy operator that adds NICs as LB services are created. (hard)

I don't know user impact. This is just technical POV

[teemow]

How likely is that solved upstream?

[kopiczko]

I'll ask in kubernetes slack.

[kopiczko]

https://kubernetes.slack.com/archives/C5HJXTT9Q/p1520503302000423

[kopiczko]

@r7vme

According to https://kubernetes.slack.com/archives/C5HJXTT9Q/p1520510302000070

Single LB in azure can handle 10 frontend IPs. One is allocated for ingress so the rest is free for LB Services. According to that you can create up to 9 LB services. Is that correct?

So it looks like we have just a limitation. Please confirm.

[r7vme]

Single LB in azure can handle 10 frontend IPs. One is allocated for ingress so the rest is free for LB Services. According to that you can create up to 9 LB services. Is that correct?

Yes, it's correct. :) But it does not solve the problem.

Problem that if you do kubectl apply -f xxx.yaml, where xxx.yaml is smth like

kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP 
      port: 80
  type: LoadBalancer

Kubernetes will create NEW load-balancer. It does not support reusing same load-balancer afaik.

[kopiczko]

Maybe it'll create a new LB and then you can have 9 more services running in the same LB. Adding single NIC has potential to solve the problem to some extent. Need to check.

[puja108]

What I saw in action few days ago on Azure was that new service of type loadbalancer only resulted in a new IP (but not sure what went on behind the scenes). This seems to be the intended way on Azure, not adding LBs, but giving IPs, cause each LB supports more than a single IP. If we have questions here, we can ping Stuart to see if he can inquire internally.

[puja108]

We can also ask Dennis, who showed above service creation in his training.

[r7vme]

Someone just need to do simple test: 1) create guest cluster 2) create service with type LoadBalancer 3) See what happened.

Next steps depending on result:

• delete ingress LB and repeat 2
• create one more service with type LB and see if it will reuse LB created by Kubernetes.

If it will reuse LB created by kubernetes (e.g. it filters LB based on tags), then we can use these tags for our ingress LB, so Kubernetes will reuse ingress LB.

Let me know if chat needed.

[xh3b4sd]

This is lying around in the SIG operator backlog. Any movement? What can we do? Should we just ignore it?

[puja108]

I think we still need to test it, or did someone do the test described above? It should work, at least from what I'm hearing from the MS guys.

[kopiczko]

Are we fine with closing it? We have a few pilots on Azure. Nobody complains and nobody is keen to test that.

[xh3b4sd]

Fine with me.

[r7vme]

Oki doki!