search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.94k stars 1.32k forks source link

Connecting non-cluster hosts to a Kubernetes cluster #6722

Open geekorous opened 2 years ago

geekorous commented 2 years ago

I noticed there are several issues on this matter already, but with relatively confusing flows that is hard to follow. The documentation also does not provide a repeatable scenario, specially for our case of having a bare-metal hosted cluster (on-prem).

In my case I have a Kubernetes cluster with Calico as its networking infrastructure. It is using BGP with no encapsulation. I have one other machine that is outside the cluster which the goal is to connect it to the cluster without joining to the cluster. The possible solutions based on my research are the following:

  1. Join the cluster and cordon the node: This is a viable solution where joining is an option.
  2. Peer the networking infrastructure with Calico: This might not be applicable in many cases where either the access to the infrastructure is limited or the infrastructure is not suited for BGP.
  3. Use Calico node in docker mode: Given a docker installation on the non-cluster host, there is documentation on how to configure and run Calico node, but unfortunately it seems it cannot distribute the networking information and could be used only for networking policies. I assume running Calico node without docker (binary mode) is not an option here, which could be great if it was.

The second case is where the confusion starts: First of all, there are two options for the data store: etcd and Kubernetes API. The documentation at https://projectcalico.docs.tigera.io/getting-started/bare-metal/about says only etcd could be used for networking. The related questions are:

  1. Whether one could use the etcd of Kubernetes cluster for that purpose? Given that Calico is not configured to use etcd directly but through the Kubernetes API.
  2. I have seen some issues like #3407 or #3551 mention a dummy, virtual node to be added into the cluster. Is this approach correct and does it work with Kubernetes API as the datastore?

I could confirm that I have tried both 1 and 2 and none of them worked.

Expected Behavior

Be able to connect a non-cluster host to the cluster without joining the cluster to get not only the policies but the networking without encapsulation (BGP mode).

Current Behavior

None of the suggested solutions above seems suitable or working.

Possible Solution

I suggest to improve the documentation and provide a repeatable scenario for this case. It would be awesome if the binary mode could also work without docker installation.

Steps to Reproduce (for bugs)

N/A

Context

In my case I have a Kubernetes cluster with Calico as its networking infrastructure. It is using BGP with no encapsulation. I have one other machine that is outside the cluster which the goal is to connect it to the cluster without joining to the cluster to allow non-containerized workloads connect to the pods in the cluster. The options are limited given the networking infrastructure is not suitable for BGP and we prefer to not join the node to the cluster.

Your Environment

lwr20 commented 2 years ago

Yep, we clearly need to do some work here. One for @caseydavenport to schedule.