add new sub-command to clusterctl to process generic yaml

[wfernandes]

User Story

As a developer I would like to have a sub-command in clusterctl that could process yaml using the built in yaml processor so that we can leverage clusterctl's internal yaml processor within scripts.

Detailed Description

Since we integrated envsubst within clusterctl's simple yaml processor, we can now handle default variables in our cluster templates and our infrastructure component yamls. However, since the addition of these variables (e.g. ${var:=defaultValue} in the infrastructure components changes to the tooling were required in order to process these yamls with the corresponding envsubst binary.

In order to avoid the need for users to have to know and use the internal tool (in this case, drone/envsubst) within their scripts directly, e.g.

curl -sSL https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.3.7/cluster-api-components.yaml | $(ENVSUBST) | kubectl apply -f -

it would be nice to do something like

# Direct usage
clusterctl config raw --core cluster-api:v0.3.7 | kubectl apply -f -

# Using local file
clusterctl config raw --from /tmp/my-yaml-file | kubectl apply -f -

Anything else you would like to add: For context, the comment https://github.com/kubernetes-sigs/cluster-api/issues/3375#issuecomment-662513206 describes configuration in tilt.

As part of the context above, it would be nice to have --list-variables supported for this command as well to show the required variables and their defaults.

/kind feature /area clusterctl

[vincepri]

/milestone v0.3.x

[wfernandes]

/assign

[wfernandes]

As per the discussion happening in #3360, we'll go with the command clusterctl generate yaml to accomplish this user story. /lifecycle active

[wfernandes]

~Do we need to support things like --target-namespace and --kubernetes-version? These are features that clusterctl supports for other commands.~ Asking because I don't know if users will want to do something like

clusterctl generate yaml --from <some-custom-infrastructure-components.yaml> --target-namespace=foobar

or

clusterctl generate yaml --from <some-custom-cluster-template.yaml> --kubernetes-version=v1.16.0

Or are we expecting the sole purpose of this command will be to substitute env vars? That is just take some kind of yaml...whatever it may be and run the our yaml processor on top of it.

@vincepri Any thoughts?

UPDATE: Actually...now that I think about it some more. Maybe we don't implement those for this command specifically since these features will be ported over when we decide to do clusterctl generate cluster.

[wfernandes]

I was planning on implementing something like clusterctl generate yaml --core cluster-api:v0.3.7 that provides the component yaml for the core provider. I thought this might be useful to simplify commands like

curl -sSL https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.3.7/cluster-api-components.yaml | $(ENVSUBST) | kubectl apply -f -

This would work similarly for

clusterctl generate yaml --bootstrap kubeadm:v0.3.0
clusterctl generate yaml --control-plane kubeadm:v0.3.0

However, for the infrastructure provider,

clusterctl generate yaml --infrastructure aws:v0.4.0

the user may be confused with what template they will get. That is, will they get the infrastructure-provider.yaml or will they get the cluster-template.yaml associated with that infra provider?

But at this point, if they'd like to get an unmodified components of a provider they could use a command like clusterctl generate provider --core cluster-api:v0.3.7. This command would provide a components yaml with all the variables substituted. If the user does not want the variables to be substituted then we can provide a --raw flag clusterctl generate provider --core cluster-api:v0.3.7 --raw.

If the user would like an unmodified cluster template for a infra provider they could similarly use clusterctl generate cluster <cluster-name> --infrastructure aws:v0.4.0

Hope this clarifies the usage of the generate sub-commands and the specific intent for clusterctl generate yaml.