Public networking should be optional

[klausenbusk]

Follow up on https://github.com/digitalocean/digitalocean-cloud-controller-manager/pull/75#discussion_r181517290. I do not see any particular reason why we require "private networking", it should be optional.

cc @andrewsykim

[cagedmantis]

@klausenbusk Can you give us a use case for when you would not want to use private networking?

[klausenbusk]

@klausenbusk Can you give us a use case for when you would not want to use private networking?

The idea is distribute the master nodes across different datacenters for redundancy (ex: FRA1, AMS3, LON1), the worker nodes run in the same datacenter (for latency reason) (ex: FRA1). This make it possible for me to recover from a datacenter outage, pretty quickly by just adding worker nodes in another datacenter.

[aybabtme]

Makes sense to me.

[lxfontes]

worth mentioning split brain is more likely to happen in this scenario ( masters in many regions ).

• different network paths from workers to each master
• latency between masters ( etcd consensus )

Still think it makes sense to make private networking optional ( in fact, keep private networking as default and make 'use public network' an option )

[klausenbusk]

worth mentioning split brain is more likely to happen in this scenario ( masters in many regions ).

A split-brain situation isn't possible with ETCD, either the master is healthy or it isn't.

different network paths from workers to each master

I'm not sure how big a concern that is these days.

[peterver]

@klausenbusk what about having a multi-datacenter federated kubernetes cluster ? https://kubernetes.io/docs/tasks/federation/. Wouldn't the masters need to communicate over the public internet with the federation server ( unless you setup some form of ssh tunnel between clusters ).

Then again not like that will be deployed or needed to be taken care of in the CCM.

[klausenbusk]

@klausenbusk what about having a multi-datacenter federated kubernetes cluster ? https://kubernetes.io/docs/tasks/federation/.

The overhead of using Federation is too big for small cluster (IMHO).

Wouldn't the masters need to communicate over the public internet with the federation server ( unless you setup some form of ssh tunnel between clusters ).

The connection is encrypted, so why is this a concern?

[andrewsykim]

Private networking on droplets is on a VPC by default now (see https://www.digitalocean.com/docs/release-notes/2018/private-networking/). I can't think of a reason why you would not run a droplet on a private network to at least isolate L4 proxy traffic. Though private networking is not strictly required, I think we should move forward here assuming that it will be. Will let @lxfontes have the final say.

[klausenbusk]

I can't think of a reason why you would not run a droplet on a private network to at least isolate L4 proxy traffic

It is way easier to recover from a datacenter outage, if the masters is spread out across multiple datacenters, but that is probably a corner-case? BTW: Can't I achieve the same kind of VPC over the public link by using Cloud Firewalls? According to https://blog.digitalocean.com/whats-new-with-the-digitalocean-network/, all traffic is traveling over DO controlled links.

[andrewsykim]

Yeah for public addresses you probably do want firewalls. What I meant was that there are very few downsides of enabling private network (especially with the new default VPC isolation). Even in your case where your masters strictly talk over a public address, having private network enabled would have no consequence and would allow for future k8s nodes added in that DC to talk over that private network.

[peterver]

It is way easier to recover from a datacenter outage, if the masters is spread out across multiple datacenters, but that is probably a corner-case?

@klausenbusk @andrewsykim Which is exactly what happened when last weekend FRA-1 networking went down for several hours. Private networking issue FRA1

Our team wasn't effected because we run multi-datacenter and with cloudflare monitors on top to steer traffic with geo-ip load balancing, but I don't expect other teams to have as complex a stack as we do ?

[lxfontes]

I'm 💯 for allowing communication over public networks. However, I disagree with making it the default.

Why? It's an advanced setup.

• cross-region clusters are not the norm ( and within same region, they should use private net )
• private network is isolated by default, public is not and users should firewall it ( default to secure )
• failure mode change ( as mentioned before )

We gonna carve out time to work on this, likely mid september or as part of hacktoberfest 🤘