KubePlus is a turn-key solution to transform any containerized application into a multi-instance SaaS.
Multi-instance multi-tenancy (MIMT) is a software architecture pattern in which a separate instance of an application is provided per tenant. The typical adopters of this pattern are application hosting providers, platform engineering teams, and B2B software vendors that need to host and manage dedicated instances of a software application for different tenants and effectively deliver that application as a managed service. KubePlus is a turn-key solution to build such managed services on Kubernetes. It comes with end to end automation to help you deploy and manage your application on Kubernetes following the MIMT pattern. This includes isolation and security between instances along with easy to use APIs for managing upgrades, customization and resource utilization.
KubePlus takes an application Helm chart and wraps it under a Kubernetes API (CRD). Whenever an application instance is created using this API, KubePlus ensures that every instance is created in a separate namespace and the required multi-tenancy policies are applied in order to ensure isolation between instances. The API supports CRUD operations on the instances of the CRD, RBAC, version upgrades, and additional customizations for each instance.
KubePlus takes an application Helm chart and wraps it in a Kubernetes API (CRD). This API is used to provision application instances on a cluster. KubePlus isolates each application instance in a separate namespace. It adds a safety perimeter around such namespaces using Kubernetes network policies and non-shared persistent volumes ensuring that each application instance is appropriately isolated from other instances. Additionally, it provides controls for application providers to deploy different tenant application instances on different worker nodes for node isolation.
The KubePlus Operator does not need any admin-level permissions on a cluster for application providers. This allows application providers to offer their managed services on any K8s clusters including those owned by their customers. KubePlus comes with a small utility that allows you to create provider specific kubeconfig on a cluster in order to enable application deployments and management. Providers have an ability to create a consumer specific further limited kubeconfig to allow for self-service provisioning of application instances as well.
KubePlus provides controls to set per-namespace resource quotas. It also monitors usage of CPU, memory, storage, and network traffic at the application instance level. The collected metrics are available in different formats and can be pulled into Prometheus for historical usage tracking. KubePlus also supports ability to define licenses for the CRDs. A license defines the number of application instances that can be created for that CRD, and an expiry date. KubePlus prevents creation of application instances if the license terms are not met.
A running application instance can be updated by making changes to the spec properties of the CRD instance and applying it. KubePlus will update that application instance (i.e. helm upgrade of the corresponding helm release). A new version of an application can be deployed by updating the application Helm chart under the existing Kubernetes CRD or registering the new chart under a new Kubernetes CRD. If the existing Kubernetes CRD object is updated, KubePlus will update all the running application instances (helm releases) to the new version of the application Helm chart.
The spec properties of the Kubernetes CRD wrapping the application Helm chart are the fields defined in the chart’s values.yaml file. Application deployments can be customized by specifying different values for these spec properties.
https://github.com/cloud-ark/kubeplus/assets/732525/efb255ff-fc73-446b-a583-4b89dbf61638
Let’s look at an example of creating a multi-instance WordPress Service using KubePlus. The WordPress service provider goes through the following steps towards this on their cluster:
Create cluster or use an existing cluster. For testing purposes you can create a minikube or kind cluster:
minikube start
or
kind create cluster
Unzip KubePlus plugins and set up the PATH
wget https://github.com/cloud-ark/kubeplus/raw/master/kubeplus-kubectl-plugins.tar.gz
tar -zxvf kubeplus-kubectl-plugins.tar.gz
export KUBEPLUS_HOME=`pwd`
export PATH=$KUBEPLUS_HOME/plugins:$PATH
kubectl kubeplus commands
Set the Namespace in which to deploy KubePlus
export KUBEPLUS_NS=default
Create provider kubeconfig using provider-kubeconfig.py
wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/requirements.txt
wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/provider-kubeconfig.py
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
apiserver=`kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'`
python3 provider-kubeconfig.py -s $apiserver create $KUBEPLUS_NS
deactivate
Install KubePlus Operator using the generated provider kubeconfig
helm install kubeplus "https://github.com/cloud-ark/operatorcharts/blob/master/kubeplus-chart-4.0.0.tgz?raw=true" --kubeconfig=kubeplus-saas-provider.json -n $KUBEPLUS_NS
until kubectl get pods -A | grep kubeplus | grep Running; do echo "Waiting for KubePlus to start.."; sleep 1; done
Create Kubernetes CRD representing WordPress Helm chart.
The WordPress Helm chart can be specified as a public url or can be available locally.
kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/application-hosting/wordpress/wordpress-service-composition.yaml --kubeconfig=kubeplus-saas-provider.json
kubectl get resourcecompositions
kubectl describe resourcecomposition wordpress-service-composition
If the status of the wordpress-service-composition
indicates that the new CRD has been created successfully, verify it:
kubectl get crds
You should see wordpressservices.platformapi.kubeplus
CRD registered.
Create WordpressService instance wp-tenant1
kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/application-hosting/wordpress/tenant1.yaml --kubeconfig=kubeplus-saas-provider.json
Create WordpressService instance wp-tenant2
kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/application-hosting/wordpress/tenant2.yaml --kubeconfig=kubeplus-saas-provider.json
Check created WordpressService instances
kubectl get wordpressservices
NAME AGE
wp-tenant1 86s
wp-tenant2 26s
Check the details of created instance:
kubectl describe wordpressservices wp-tenant1
Check created application resources. Notice that the WordpressService
instance resources are deployed in a Namespace wp-tenant1
, which was created by KubePlus.
kubectl appresources WordpressService wp-tenant1 –k kubeplus-saas-provider.json
NAMESPACE KIND NAME
default WordpressService wp-tenant1
wp-tenant1 PersistentVolumeClaim mysql-pv-claim
wp-tenant1 PersistentVolumeClaim wp-for-tenant1
wp-tenant1 Service wordpress-mysql
wp-tenant1 Service wp-for-tenant1
wp-tenant1 Deployment mysql
wp-tenant1 Deployment wp-for-tenant1
wp-tenant1 Pod mysql-76d6d9bdfd-2wl2p
wp-tenant1 Pod wp-for-tenant1-87c4c954-s2cct
wp-tenant1 NetworkPolicy allow-external-traffic
wp-tenant1 NetworkPolicy restrict-cross-ns-traffic
wp-tenant1 ResourceQuota wordpressservice-wp-tenant1
Check application resource consumption
kubectl metrics WordpressService wp-tenant1 $KUBEPLUS_NS -k kubeplus-saas-provider.json
----------------------------------------------------------
Kubernetes Resources created:
Number of Sub-resources: -
Number of Pods: 2
Number of Containers: 2
Number of Nodes: 1
Number of Not Running Pods: 0
Underlying Physical Resoures consumed:
Total CPU(cores): 0.773497m
Total MEMORY(bytes): 516.30859375Mi
Total Storage(bytes): 40Gi
Total Network bytes received: 0
Total Network bytes transferred: 0
----------------------------------------------------------
Cleanup
kubectl delete wordpressservice wp-tenant1 --kubeconfig=kubeplus-saas-provider.json
kubectl delete wordpressservice wp-tenant2 --kubeconfig=kubeplus-saas-provider.json
kubectl delete resourcecomposition wordpress-service-composition --kubeconfig=kubeplus-saas-provider.json
helm delete kubeplus -n $KUBEPLUS_NS
python3 provider-kubeconfig.py delete $KUBEPLUS_NS
KubePlus can be used in the following use cases.
In order to try these examples, follow these steps:
git clone --depth 1 https://github.com/cloud-ark/kubeplus.git
cd kubeplus
export KUBEPLUS_HOME=`pwd`
export PATH=$KUBEPLUS_HOME/plugins:$PATH
Go to appropriate examples directory and follow steps.txt
there in.
Make sure you have installed latest version of kubectl and you have created a minikube/kind cluster.
KubePlus architecture details are available here. KubePlus is a referenced solution for multi-customer tenancy in Kubernetes.
If you are using KubePlus chart version < 4.0.0, follow these steps to migrate to 4.0.0+ versions.
In versions < 4.0.0, the KubePlus's built-in CRDs like ResourceComposition
were included in the chart's
templates
folder. This led to them getting deleted on KubePlus's upgrade. In 4.0.0+ version, the CRDs
have been moved to the crds
folder, which avoids this issue.
kubectl annotate customresourcedefinition resourcepolicies.workflows.kubeplus helm.sh/resource-policy=keep
kubectl annotate customresourcedefinition resourceevents.workflows.kubeplus helm.sh/resource-policy=keep
kubectl annotate customresourcedefinition resourcemonitors.workflows.kubeplus helm.sh/resource-policy=keep
kubectl annotate customresourcedefinition resourcecompositions.workflows.kubeplus helm.sh/resource-policy=keep
helm upgrade kubeplus <4.0.0+ version> -n $KUBEPLUS_NS --kubeconfig=kubeplus-saas-provider.json
Check the contributing guidelines.
KubePlus is part of CNCF landscape's Application Definition section.
As enterprise teams build their custom Kubernetes platforms using community or in house developed Operators, they need a set of guidelines for Operator readiness in multi-Operator and multi-tenant environments. We have developed the Operator Maturity Model for this purpose. Operator developers are using this model today to ensure that their Operator is a good citizen of the multi-Operator world and ready to serve multi-tenant workloads. It is also being used by Kubernetes cluster administrators for curating community Operators towards building their custom platforms.
KubePlus presentation at community meetings (CNCF sig-app-delivery, Kubernetes sig-apps, Helm)
DevOps.com Webinar: Deliver your Kubernetes Applications as-a-Service
Being a good citizen of the Multi-Operator world, Kubecon NA 2020
We meet every Tuesday at 10.30 a.m. US CST. We use Slack huddle in #kubeplus
channel on CNCF workspace
The meeting agenda is here.
Please join us in our meetings. Your participation is welcome.
Subscribe to KubePlus mailing list.
Join #kubeplus channel on CNCF Slack.
If you don't have an account on the CNCF workspace, get your invitation here. You can join the #kubeplus
channel once your invitation is active.