search

vmware-archive / kube-prod-runtime

A standard infrastructure environment for Kubernetes
Apache License 2.0
767 stars 135 forks source link

Support for non-cloud environments #275

Open EamonKeane opened 5 years ago

EamonKeane commented 5 years ago

This looks great... a lot more maintainable than wrangling the same charts individually.

Are there plans to make this work on e.g. hetzner?

falfaro commented 5 years ago

Yes, we have plans for this. At the moment we are focusing mostly on public cloud, but we will support Kops and perhaps with Kops we can support non-public environments. Is this something that would fit your use case?

falfaro commented 5 years ago

My previous answer was not completely accurate. At the moment we are focusing on adding support to BKPR for AWS/EKS and AWS/Kops. The Kops documentation clearly states that only AWS is stable, with GCE marked as beta and VMware vSphere as alpha.

This said, nothing prevents you from deploying BKPR on your Kubernetes cluster, whichever platform you are using. For instance, I have been using Canonical's Kubernetes deployed using Juju and MaaS on top of a Hetzner machine in germany. This single machine from Hetzner was configured as a KVM hypervisor running 8 KVM virtual machines managed by MaaS. The only issues I had with BKPR is that I had to configure and tune Ingress and DNS myself. On GKE and AKS this is handled for you transparently.

Please let me know if this answers your question.

EamonKeane commented 5 years ago

Thanks @falfaro that's helpful. I guess the one area I was thinking of was load balancing. I have a stack of helm charts (similar components to BKPR) that works on Hetzner and GKE with the only difference is that metal-lb is installed on Hetzner (it uses google's Cloud DNS). Does your adjusted setup include that?

anguslees commented 5 years ago

@EamonKeane At a technical level, BKPR should work with metallb just fine. We don't assume anything more than Service type=LoadBalancer works, and gives us a public IP (required by letsencrypt(*)).

(*) ... until we switch to DNS01 challenges. Then we just need a supported managed (public) DNS zone.

The big items for a new platform are:

  1. Find a managed-DNS solution and OIDC (OAuth) gateway.
  2. Write kubeprod installer code to handle the simple/normal case.
  3. Set up automated testing for the new platform so we can be sure that all the functionality continues to work going forward.

Solving (1) for bare-metal is easy: just point to any of the many hosted solutions supported by the external-dns and oauth proxy controllers (ie: this doesn't have to be served "locally" on the bare-metal cluster).

(2) and (3) will be particularly challenging for us as a project given the enormous variation in bare-metal environments. For an individual deployment however, it will be quite simple since you only care about your specific setup. So I suspect the way this will play out is that individual deployments of BKPR onto bare-metal using the "manual" deployment method (ie: not kubeprod install ...) will be straightforward and common long before we can declare any "official" support for bare-metal. You could do it right now, for example, except that we don't have any docs to help you.

@EamonKeane (or anyone else): how much do you want this? I'd be happy to work with someone to develop a "BKPR the hard way" doc, that talked through how to install it onto any environment. I just need someone with some time to be the canary deployment - to give feedback and verify the doc is sufficient.

EamonKeane commented 5 years ago

@anguslees I'd be happy to beta "BKPR the hard way" if you plan on putting together instructions.

I agree that the last mile of configuring bare metal could be left out as it is very varied.