Enable multi-tenant, managed KaaS offerings using ref.-impl.

[joshmue]

As a cloud customer, I want to create Kubernetes clusters that are operated by my CSP who takes responsibility for basic cluster setup and maintenance. As a cloud customer, I want to be responsible only for my own workloads and as little else as possible.

Relation to "cluster stacks"

AFAIU by @joshmue:

321 and #326 track the idea of "cluster stacks", which should enable people to create similar clusters on different (IaaS-)providers. Being a Cluster API based solution, "cluster stacks" would require a bunch of prerequisites, like first and foremost: A management/bootstrap Kubernetes cluster.

• Option 1: Create a virtual machine on a IaaS provider, installing e. g. kind
• Option 2: Use a different managed Kubernetes offering by some CSP. E. g. AKS, EKS, GKE, Gridscale-managed-K8s, PSKE, Cleura Kubernetes or others.

Then, having a management Kubernetes cluster, roughly the following steps are required:

• Setting up access credentials to some IaaS provider(s)
• Defining cluster (stacks), making sure that the provisioning process works (given that IaaS API's work)
• Fixing any errors that may arise during normal operation or when doing upgrades

By default, the customer is responsible for every single of these steps (and more). Hence, the customer would be well-advised to have experts employed (or contracted) that are specialized in operating Kubernetes using e. g. "cluster stacks" and Cluster API.

Challenge

That is kind of the anti-thesis to usual "Kubernetes as a Service" offerings, which generally move as much work as it is reasonable into the realm of "provider responsibility". Without additional glue for providers to make "cluster stacks" a "true" KaaS, "cluster stacks" would not play in the same market as usual KaaS does.

Solution

Create tooling for providers to offer an API (see #136, as opposed to a ticket-based workflow) to create "cluster stack"-based clusters that are operated and managed by the provider.

Definition of Ready:

• [ ] User Story is small enough to be finished within one sprint
• [ ] User Story is clear and understood by the whole team
• [ ] Acceptance criteria are defined
• [ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

• [ ] All acceptance criteria are met
• [ ] Changes have been reviewed
• [ ] CI tests have run successfully
• [ ] Documentation has been updated
• [ ] Release Notes have been updated

[joshmue]

Happy to collect your perspective on this, @batistein, @janiskemper.

[janiskemper]

I would say that 'Make "Cluster Stacks" a "Kubernetes as a Service"' is what we want to reach with the reference implementation. This reference implementation should be easy to use, but it is self-service. If people don't want to do that, then the reference implementation is not what they look for.

Completely independent of the reference implementation, there is the task of standardizing SCS clusters. If these standards are fixed, then CSPs can get certified for providing their customers with standardized SCS clusters.

You are only referring to the second part with your idea of having not only standardized clusters but also a standardized API for customers to get/manage these clusters.

Whether this is a good goal for SCS, I can't say. I would say standardizing the clusters themselves is definitely more important. But a unified API can be handy, of course.

As such an API is completely independent of the cluster stacks and reference implementation (there we have the Kubernetes API via management cluster and won't be able to change that), I would say that we change the title to something like "Unified SCS API for SCS-certified Managed Kubernetes providers".

[o-otte]

I have to say that for me it is quite unclear what "as a Service" means within the SCS Context. My understanding, and I think @joshmue has the same in mind, of "as a Service" is that most of the Kubernetes offering, at least the control plane, is handled and operated by CSPs. When thinking about the Hyperscaler Kubernetes offerings, the customers have not even access to the control plane. Also customers just specify some cluster and node parameters and the rest is handled by the logic of the CSPs. The manual steps @joshmue pointed out above and the term "self-service" seems to contradict this. The SCS About page states:

the real service here is k8s aaS – we are offering the k8s cluster API as interface to manage k8s clusters; providers can of course use it internally as well to create managed services.

As Kubernetes as a Service is a main goal, I think we need a clear definition what that means and entails.

Furthermore, to ensure customers can move workload from one cloud to another cloud and setup clusters the same way in SCS-compliant clouds I think we need to ensure that the clusters are also operated in a standardized way. At least we should have a clear understanding what parts are managed by the CSP and which are not.

Hence, I propose that we should create one or more ADRs with the goal to specify what the SCS means with the terms "Kubernetes as a Service" or "managed Kubernetes" and also what the responsibilities of CSPs and Customers are. We should also flesh out the UX Workflows for Scenarios like: Creating Clusters, Updating Clusters Configurations (Changing Node Sizes, Node count), or Upgrading of Kubernetes Versions, etc. We should also align the outcome of the ADRs to the needs and constrains of the CSPs as we want them to use the reference implementation.

@jschoone, @garloff, @fkr What do you think?

[joshmue]

This reference implementation should be easy to use, but it is self-service. If people don't want to do that, then the reference implementation is not what they look for.

I very much like the clear distinction, here.

I would say that 'Make "Cluster Stacks" a "Kubernetes as a Service"' is what we want to reach with the reference implementation.

As @o-otte pointed out, this wording will probably turn out to be an issue. [^1]

You are only referring to the second part with your idea of having not only standardized clusters but also a standardized API for customers to get/manage these clusters.

Whether this is a good goal for SCS, I can't say. I would say standardizing the clusters themselves is definitely more important. But a unified API can be handy, of course.

As such an API is completely independent of the cluster stacks and reference implementation (there we have the Kubernetes API via management cluster and won't be able to change that), I would say that we change the title to something like "Unified SCS API for SCS-certified Managed Kubernetes providers".

This issue was not so much intended to be about a technical, standardized API (I intended #136 to track that). Rather, it was intended to track work to Make "Cluster Stacks" a "Kubernetes as a Service", making it possible for CSP's to use the reference implementation as technical base to build a "Kubernetes as a Service" in the typical "managed service" sense.

This does indeed go a bit beyond what "About SCS" says: "providers can of course use it internally as well to create managed services". But in order to make this statement really true, it should be realistically feasible for providers to actually do that - without an unreasonably huge amount of work on their side. For example, if a hosting company decides to build an SCS based cloud, they should by default be able to offer a "Kubernetes as a Service" in the "managed" sense, right? If SCS does not provide an implementation for that, the provider is effectively required to use non-SCS tech like Gardener (like IIRC Plusserver and Regio.digital do). That may very well be the most technically viable option, but IMHO should be reflected in documents like the to-be-written ADR (like @o-otte mentioned) about KaaS definition and goals.

[^1]: I think that while the term "Kubernetes as a Service" has many blurry interpretations, they all have in common that at least some parts of "Kubernetes" is offered "as a Service" by a service provider, going beyond features that "Infrastructure as a Service" CSP's are offering (in turn, providers offering "Infrastructure as a Service" generally take care and responsibility of stacking server racks and plugging cables, so service consumers do not have to). In sum, I do not think that building a "Do it yourself" Kubernetes on IaaS infrastructures does fit well to the literal content of the term "Kubernetes as a Service". But +1 for the ADR approach, so a common understanding can emerge.

[janiskemper]

@joshmue thanks for clarifying your intentions with this issue.

Kubernetes-as-a-Service was the tender that we won. If SCS wants to do this, it will always be, a priori, without any CSP. They will always come only in the second step when they choose to use the reference implementation that we build. According to your definition, the wording in the tender would be incorrect I suppose.

But anyway, the wording is not so important to me. You or SCS can name it however you want :smile:

The other point you made was that you wanted to make sure that CSPs can use the reference implementation. Well, this is what we want to do, right? Same as in the quote from SCS website - the reference implementation should be open to use for everyone!

I do not really know what exactly you have in mind when you want to make an even better job for CSPs... In what way?

And btw: We are mingling the two parts of a) cluster stacks and b) the operator that helps you using the cluster stacks.

The cluster stacks themselves are supposed to be provided by CSPs who want to be part of SCS. Other than that users won't be able to get standardized clusters on their infrastructure.

The operator can be used by both CSPs and other users that want to have clusters on a certain infrastructure.

[joshmue]

The other point you made was that you wanted to make sure that CSPs can use the reference implementation. Well, this is what we want to do, right? Same as in the quote from SCS website - the reference implementation should be open to use for everyone!

I do not really know what exactly you have in mind when you want to make an even better job for CSPs... In what way?

Simply put: Enable CSP's that use the SCS reference implementation at Kubernetes "level" to participate in the same market as CSP's that provide managed KaaS using multi-tenant-aware management tooling already [^1].

General UX of this "market" being (kind of 1:1 the same as for IaaS offerings, just for IaaS resources instead of Kubernetes clusters):

• Registration at CSP
• Being granted web frontend access and/or CLI and/or API access to create/manage Kubernetes clusters
• If any error occurs: Ask CSP for help, who is responsible for all general cluster management topics

I'm sure both "cluster stacks" and the "cluster stack operator" can be wrapped to provide such UX, but as-is, without such wrapping (or AFAIK also without Proof of Concept), CSP's are left to implement this wrapping themselves, which may not be a trivial task to do well. As such, CSP's intending to participate in this market, may not even consider using the reference implementation for that purpose, but use/wrap e. g. Gardener, which seems to work well for multiple companies. Again, leaving this market to Gardener or other non-SCS-tooling...

may very well be the most technically viable option, but IMHO should be reflected in documents like the to-be-written ADR (like @o-otte mentioned) about KaaS definition and goals.

In that case, this issue indeed may describe a feature out of SCS's scope and hence should be closed.

[^1]: By rolling their own multi-tenant cluster management (like Azure/AKS, AWS/EKS, Google/GKE) or using/wrapping e. g. Gardener for that purpose.

[janiskemper]

So you say that SCS should build a UI for managing clusters?

[joshmue]

An UI may be a logical part of such offering, indeed. But the point would be that the CSP offers it "as a Service", and is technically and organizationally responsible for UI hosting and general cluster management.

[janiskemper]

Sorry, I didn't get that. Who else should be responsible for cluster management and hosting other than the CSP that offers Managed Kubernetes?

[joshmue]

Yes, indeed. In this managed KaaS scenario, the provider would be responsible, no one else.

---

With...

A UI may be a logical part of such offering, indeed. But the point would be that the CSP offers it "as a Service", and is technically and organizationally responsible for UI hosting and general cluster management.

...I just wanted to say that "building"/programming a UI ("SCS should build a UI for managing clusters") that is usable in a self-service context, but does not implement multi-tenancy (and other features that CSP's need to build a managed KaaS) would not be sufficient, of course.

[janiskemper]

Okay. Then let's come to the title and description of the issue: I would say that the title should be something like "Add UI for SCS Kubernetes-as-a-Service". What would you say?

[janiskemper]

And as a separate issue: "Add multi-tenancy feature to K8s-as-a-Service"

[joshmue]

:+1:

Okay. Then let's come to the title and description of the issue: I would say that the title should be something like "Add UI for SCS Kubernetes-as-a-Service". What would you say?

Created #341.

And as a separate issue: "Add multi-tenancy feature to K8s-as-a-Service"

Renamed this issue; Omitted K8s-as-a-Service term to avoid confusion.

[janiskemper]

cool :)

[joshmue]

For clarification:

I do not really know what exactly you have in mind when you want to make an even better job for CSPs... In what way?

I basically had exactly in mind what you offer with the Syself Autopilot in "SaaS" mode, now. To help other CSP's to provide similar services.

[janiskemper]

makes sense :) But it's straightforward: Giving a namespace and making sure that the user has only access to this one namespace!