Project review - KubePlus

[joshgav]

Thank you @devdattakulkarni for presenting KubePlus to TAG App Delivery! Here's a summary:

Meta

• GitHub repo: https://github.com/cloud-ark/kubeplus
• Product: https://cloudark.io/
• Presentation recording: https://youtu.be/Ke5fPT9l3bg?t=1027

Value props

• Major value proposition is "multi-instance multi-tenancy": each tenant is assigned a separate instance of the same application, one instance is assigned to each tenant. Separate data, config, customizations per instance/tenant.
  • Also called "Multi-customer tenancy" in Kubernetes docs
  • Apps that are otherwise single-tenant can be managed in a multi-tenant way - that's what "multi-instance" is
• Use cases:
  • B2B SaaS
  • Enterprise internal SaaS (and possibly internal platform capabilities)
  • Public SaaS
• Story
  • A service described by a Helm chart is to be deployed many times for many tenants in one cluster.
  • Naive approach would be to run helm install against each tenant namespace. But this leaves gaps in isolation, security, monitoring, upgrades, and customizations for tenants.
  • KubePlus manages the installation of a Helm chart to a namespace and adds features to ensure and enable isolation, security, monitoring, upgrades, and customizations for each tenant and instance.

Implementation

• Components info: https://cloud-ark.github.io/kubeplus/docs/html/html/kubeplus-components.html
  • Includes a kubectl plugin
• ResourceComposition resource includes a link to a Helm chart
• KubePlus generates a CRD based on the ResourceComposition resource. For example, it generates a WordPressService CRD from a WordPress ResourceComposition resource.
  • Helm's values.yaml schema becomes the schema for the new CRD's spec
  • An OpenAPI schema is also generated from values.yaml - not easy, but required in recent Kube versions.
• KubePlus handles instances (CRs) of the generated CRD and does stuff that adds common required features - isolation, security, monitoring, upgrades, customization
• A MutatingWebhook picks up newly-submitted instances of the CRD and calls the Helmer service to render and apply the chart
  • Has been tested alongside other MutatingWebhooks (e.g. from Kyverno) without problems
• KubePlus tracks all the resources created by the chart and by KubePlus itself, and makes them accessible via kubectl appresources (a plugin)
  • NetworkPolicies and ResourceQuotas CRs are generated and injected by KubePlus
• kubectl metrics plugin aggregates metrics across all the resources in the CRD
• CloudArk product offering includes a GUI that gathers metrics periodically and tracks them over time
  • GUI is available publicly and can be deployed by individual researchers but is intentionally not documented or supported
• Tested with all Bitnami Helm charts and addressed bugs
  • Please open issues for Helm charts with problems!

Questions

• If someone just wants to deploy capabilities, not complete applications, should they consider KubePlus? E.g., if tenants want a database in their namespace alongside other components.
  • Yes, but namespace needs to be dedicated to the KubePlus-deployed service.
  • Could bundle the complete application in one chart - including database and app components.
  • Is it okay to use the namespace for other components? Not really, KubePlus expects to own the namespace.
• Compare/contrast with Operator Lifecycle Manager - it also publishes new resource types and allows many instances of them?
  • More complexity with OLM - KubePlus only requires one controller and charts are added directly to it; OLM has notion of channels and packages
  • KubePlus focuses on one pattern: apps bundled as Helm charts; OLM supports Go and other types of operators
• Compare/contrast with Operator Framework's Helm-chart-based operator - it also bundles a Helm chart and exposes a CRD based on the schema in values.yaml?
  • Operators created by Operator SDK yield a lot of boilerplate Go code which must be built and deployed
  • Operator SDK requires a controller per operator type, KubePlus requires just one operator (the KubePlus operator)
• How does KubePlus interact with GitOps tools like ArgoCD?
  • This is work in progress - must update ResourceCompositions, then roll out updates to instances based on those compositions

[devdattakulkarni]

Thanks @joshgav Very well captured 👍

[joshgav]

Done for now so closing. Thanks!