search

kubernetes-sigs / cluster-api

Home for Cluster API, a subproject of sig-cluster-lifecycle
https://cluster-api.sigs.k8s.io
Apache License 2.0
3.49k stars 1.29k forks source link

Add support for clusterctl flavor template mechanism #2339

Closed timothysc closed 4 years ago

timothysc commented 4 years ago

User Story

As a admin I would like to provide my team with a set of well defined templates w/defaults so users only have to input a limited subset of variables when standing up a cluster.

Detailed Description

As of today, the clusterapi.yaml can be intimidating for users to consume, which poses a barrier to broader adoption. An easy way to mitigate this problem is to create well defined flavor.templates that have a limited set of user input variables. This pattern is very common in the kubernetes community, e.g helm charts.

The purpose of this is issue is to enable clusterctl to create a pluggable template model for flavors. For the purpose of simplicity we may want to start with templates that are relatively simple and well defined go-lang libraries and support e.g. https://github.com/k14s/ytt with the option of creating plugable templates for operators who prefer their own secret sauce.

Anything else you would like to add:

On creation of the flavor only the template variables are presented as possible input options in order to simplify the user experience.

This would be highly valuable to the end user community.

/cc @fabriziopandini /kind feature

ncdc commented 4 years ago

As discussed in Slack, pls review https://master.cluster-api.sigs.k8s.io/clusterctl/commands/config-cluster.html#variables and https://master.cluster-api.sigs.k8s.io/clusterctl/configuration.html#variables. We don't have anything other than variable substitution (no conditionals, no loops, etc.), but hopefully this is enough for now.

vincepri commented 4 years ago

/milestone Next

frapposelli commented 4 years ago

Big +1 on this, having defaults, conditionals and loops available will make templates a much better experience.

Also, rendering the template server-side will make it easier to adopt from other programming languages, as currently the clusterctl client library is only available in go.

timothysc commented 4 years ago

So our current user experience could use some work, I'll try to update the ask to outline in more detail the workflow I'm thinking about.

ncdc commented 4 years ago

@frapposelli if you're looking for server-side rendering, it's possible, but because we're CRD-based, it would end up being a request/response pattern and not a blocking call.

fabriziopandini commented 4 years ago

Some initial considerations: There are concerns that are common across templating engines:

The last point is where template engines are generally not adequate, because they are designed to receive in input a list of values in some format (e.g. Helm, ytt).

Instead, what is required for improving the user experience around "workload cluster template/flavors" is something more than a list of values; it is required to have the list of variables included in each template, where each variable have at least:

How to fill the gap between list of values and list of variables in a template agnostic way requires some design work.

vincepri commented 4 years ago

Let's move the discussion to a proposal google doc, we can do a POC using FeatureGates in CAPI as v0.3.0-rc.1

/kind proposal

randomvariable commented 4 years ago

A small, but potentially very beneficial improvement for now would be the ability for providers to provide a variable description manifest, so that 

➜ clusterctl config cluster test --infrastructure=vsphere:v0.0.0

Error: failed to perform variable substitution: value for variables [CLUSTER_CIDR, K8S_IMAGE_REPOSITORY, KUBERNETES_VERSION, SERVICE_CIDR, SERVICE_DOMAIN, SSH_AUTHORIZED_KEY, VSPHERE_DATACENTER, VSPHERE_DATASTORE, VSPHERE_DISK_GIB, VSPHERE_FOLDER_VAR, VSPHERE_MEM_MIB, VSPHERE_NETWORK, VSPHERE_NUM_CPUS, VSPHERE_RESOURCE_POOL, VSPHERE_SERVER, VSPHERE_TEMPLATE] is not set. Please set the value using os environment variables or the clusterctl config file

can be more informative. Wouldn't necessarily require the adoption of a templating engine.

In the past, at least with Ruby, have used i18n packages for messages and ended up with things that looked like the following:

options:
- name: AWS_ACCOUNT
  description: The AWS Account ID for the ${cluster} cluster
  messages:
    invalid: |-
      To find your AWS account ID number in the AWS Management Console, click on
      Support in the navigation bar in the upper-right, and then click Support
      Center. Your currently signed in account ID appears in the upper-right
      corner below the Support menu.
      For further information, see,
      http://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html
casibbald commented 4 years ago

We at WeaveWorks have a set of templates as helm charts that could be utilised. Currently updating these for alpha3 and would consider contributing them and making them available on a helm repository.

vukg commented 4 years ago

We at WeaveWorks have a set of templates as helm charts that could be utilised. Currently updating these for alpha3 and would consider contributing them and making them available on a helm repository.

Essentially what @casibbald is referring to is complete set of templates not only for clusterapi.yaml, but also working method of standing up the cluster fully equipped with selected components like CNI, Ingress controller, Policy agent, Monitoring and Logging stack etc by just pointing to Git dir with templates. We are working with WeaveWorks to apply that work at Deutsche Telekom Technik and working together to expand it to full GitOps based cluster management.

@vincepri @timothysc We could organize dedicated Zoom to walk you through our use case. I originally intended to do that in person in Amsterdam, but Corona....

ncdc commented 4 years ago

I'm interested too, and I imagine @sedefsavas will be as well.

My vision is to use Sedef's work in #2395 to get a minimal set of things installed as soon as a cluster is created:

Getting the "addon manager(s)" installed would get the managers/operators configured for things like the CNI provider that we had to install manually (a bit of chicken & egg), CoreDNS, kube-proxy, and whatever else you may want in all your clusters (monitoring, logging, etc).

I know the above isn't necessarily about cluster templating, but it relates to your suggestion @vukg.

vukg commented 4 years ago

Great! We will plan for it together with WeaveWorks partners and ping the community via Slack when we have things in place (couple of days).

We intend to make whole that concept open and contribute it.

vukg commented 4 years ago

It seems that it will take us a bit more time to make migration from 0.2.0 to 0.3.0 than we originally anticipated. We will come back to demo once we have stable setup.

fabriziopandini commented 4 years ago

I'm interested in this discussion as well

CecileRobertMichon commented 4 years ago

Interested as well.

@ncdc I wonder if what you're describing could also apply for out-of-tree cloud provider managers and CSI (related to https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/456#issuecomment-605171003). In this case we'd want all clusters with a certain K8s version (eg. 1.18+) to install the CCM and CSI addons.

casibbald commented 4 years ago

We have it running for all components and providers and fully managed by helm

On Fri, 27 Mar 2020 at 22:41, Cecile Robert-Michon notifications@github.com wrote:

Interested as well.

@ncdc https://github.com/ncdc I wonder if what you're describing could also apply for out-of-tree cloud provider managers and CSI (related to kubernetes-sigs/cluster-api-provider-azure#456 (comment) https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/456#issuecomment-605171003). In this case we'd want all clusters with a certain K8s version (eg. 1.18+) to install the CCM and CSI addons.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/cluster-api/issues/2339#issuecomment-605344513, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA6C3Y2LZZSTQY7SGYKAADRJUTQRANCNFSM4KVMYTNQ .

vukg commented 4 years ago

We would be ready to demonstrate together with @casibbald from Weaveworks our use case with CAPI incl. cluster templating, bootstrapping of workload clusters and installation of standard components at bootstrapping time using Helm and few other things.

If you agree I would go for dedicated call. I will reach out via Slack to fix suitable slot.

@ncdc @fabriziopandini @timothysc @vincepri @sedefsavas

ncdc commented 4 years ago

Hi @vukg, if you'd like to demo something, I would encourage you to attend the weekly Cluster API Office Hours meeting, Wednesdays at 1pm ET. You can sign up for a demo by adding it in the "Demos/POCs" section here

CecileRobertMichon commented 4 years ago

@devigned and I have been thinking about this overall problem lately, wondering if anyone on this thread has thoughts on https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/540 since it's related.

devigned commented 4 years ago

It would also be useful to provide templating functionality for clusterctl init as well.

For example, it would be useful to template the infrastructure-components.yaml for a provider to change the way the provider is configured.

One concrete example of this is the desire to use a different identity type when initializing a CAPZ provider. Many of the cloud providers offer credentialed access or usage of an implicit identity provisioned on the node. If the configuration was templated, it would be easy to produce one template that allowed a user of clusterctl init to specify the desired configuration as input and produce a yaml with the desired identity configuration.

vincepri commented 4 years ago

/assign @wfernandes

wfernandes commented 4 years ago

/lifecycle active

vincepri commented 4 years ago

As per April 27th 2020 community guidelines, this project follows the process outlined in https://github.com/kubernetes-sigs/cluster-api/blob/master/CONTRIBUTING.md#proposal-process-caep for large features or changes.

Following those guidelines, I'll go ahead and close this issue for now and defer to contributors interested in pushing the proposal forward to open a collaborative document proposal instead, and follow the process as described.

/close

k8s-ci-robot commented 4 years ago

@vincepri: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/2339#issuecomment-620121510): >As per April 27th 2020 community guidelines, this project follows the process outlined in https://github.com/kubernetes-sigs/cluster-api/blob/master/CONTRIBUTING.md#proposal-process-caep for large features or changes. > >Following those guidelines, I'll go ahead and close this issue for now and defer to contributors interested in pushing the proposal forward to open a collaborative document proposal instead, and follow the process as described. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.