Add CLI command to delete a workload cluster

marians commented 1 year ago

User story

As a user I want to simply delete a workload cluster and all resources related to it, via a CLI.

Details

Deleting a cluster is not as straightforward as deleting a Cluster CR, and it varies between product generations and potentially also between providers.

The goal here is to provide one CLI command like kubectl gs delete cluster NAME --namespace NAMESPACE which handles all the internal differentiation.

Tasks

[ ] Spec (Rainbow)

marians commented 1 year ago

@calvix explained cluster deletion for CAPA like this:

# create resource manifests
kubectl gs template cluster --provider=capa --organization giantswarm --name=vac01 > ./cluster-vac01.yaml

# create cluster
kubectl apply -f ./cluster-vac01.yaml

# delete cluster
kubectl delete -f ./cluster-vac01.yaml

The kubectl gs template cluster for CAPI clusters yields four resource manifests: 2 x App, 2 x ConfigMap. To delete the cluster, all four resources have to be deleted.

Based on this thread this is the same in all CAPI providers.

I see a problem with this relying purely on resource naming. Who guarantees that all workload clusters conform with this naming convention? Anyway, this might mean that the implementation could be changed in the future.

Topics to address in the spec:

General Syntax of the command: kubectl gs delete cluster
Which flags should be provided, which are required? --namespace vs. --organization
Should apps installed in a cluster also be deleted? Should there be an option for/against this? What should be the default?
How to treat the case that not all expected resources exist?
If namespace is not specified, where should the search happen? default?
What if the user has only partial permission to delete, list, get resources?
How can we prevent accidental deletion?

marians commented 1 year ago

Spec v1

For your feedback

General description

Command to delete workload cluster resources on a management cluster. In terms of the semantics, we provide this command to express that a certain workload cluster should not exists. This means that the command does whatever required to remove resources belonging to this cluster, even if the cluster has not been created completely.

General syntax

kubectl gs delete cluster NAME [FLAGS]

Flags

--organization: The organization owning the cluster. Can be specified instead of --namespace to indicate the namespace in which resources should be deleted.
--namespace: The namespace in which resources should be deleted (Default: default).
--keep-apps: If set true, App resources for this cluster will not be deleted. Does not affect the cluster-app and the default-apps bundle, which will be deleted in any case. (Default: false)
--keep-app-config: If set true, ConfigMaps and Secrets belonging to App resources will not be deleted. Does not affect the ConfigMap resources for the cluster-app and the default-apps bundle, which will be deleted in any case. (Default: false)
--dry-run: If set true, a list of the resources to be deleted will be printed, but deletion will not take place. (Default: false)

Example commands

Delete cluster abc12 owned by organization acme, which is expected to be found in namespace org-acme.

$ kubectl gs delete cluster abc12 \
  --organization acme

Delete cluster abc12 namespace org-acme. The same as above, but using --namespace instead of --organization.

$ kubectl gs delete cluster abc12 \
  --namespace org-acme

Cluster info and confirmation prompt

If cluster and node pool resources exist for the cluster to be deleted, information will be shown and a prompt asks for confirmation:

You are about to delete this cluster from
installation gollum (region eu-westeurope-1):

Name:              abc12
Service priority:  HIGHEST
Description:       Prod cluster K8s 1.25
Created:           12 May 2022 - 5 months ago

Node pools:        2
Worker nodes:      12

Do you really want to delete this cluster? There is no undo!
If yes, please type the installation and cluster name,
separated by a slash, and hit Return.
If no, enter anything else or hit Ctrl-C.

>

TODO: Decide what to show if no cluster resource, or no node pool resource, exists for the cluster.

Behind the scenes

Based on the flags used (--namespace, --organization, --keep-*) we compile a list of resources to be deleted.
We also look up whether actual main cluster resource exists. We store the region, service priority label, cluster description from the main cluster resource if it exists.
If a cluster resource exists, we also look up whether node pools exist and store the number of worker nodes.
For the resources in the "to be deleted" list, we check get and list permissions, to ensure that the user can find all existing resources to be deleted. If permission is lacking for at least on of the resources, we exit with a specific error message that lists all missing permissions.
Then existing resources to be deleted are looked up.
For all resources to be deleted, we check whether the acting user has permission to delete. If permission is lacking for at least one of the resources, we exit with a specific error message that lists all missing permissions.
If --dry-run is true, we print a list of all resources to be deleted and exit.
If the main cluster resource exists, we show a general info block regarding the cluster.
Then we delete resources one by one and print one line for each one.

Special cases

The cluster specified by name cannot be found at all: This will be reported as an error with exit code > 0. We see it as an indicator that the user may have mistyped the name or namespace.

marians commented 1 year ago

Comments from refinement in Team Rainbow

In other commands, specifically login --workload-cluster, if organization or namespace is not defined, we look up in all accessible organization namespaces. If the result is unique, we use the resulting org/namespace. We should apply the same logic here.
- For now we should not offer the --organization flag here, and only offer --namespace (which is a global flag). Later we can implement the --organization flag as an alternative in all commands where it makes sense.
- --keep-apps should also keep config and secrets. We remove --keep-app-config from the spec.

We change the prompt like below:

You are about to delete this cluster:

Name:              abc12
Installation:      gollum
Region:            eu-westeurope-1
Service priority:  HIGHEST
Description:       Prod cluster K8s 1.25
Created:           12 May 2022 - 5 months ago

Node pools:        2
Worker nodes:      12

Do you really want to delete this cluster? There is no undo!
If yes, please type the installation and cluster name,
separated by a slash, and hit Return.
If no, enter anything else or hit Ctrl-C.

> gollum abc12

Error: Installation and cluster name do not match the format 'gollum/abc12'.

$>

Dry-run output format should be specified
We discussed that the interactive prompt will mean that the command is not easily used in automation. If needed, the expect utility can be used to work around this. Offering a --force flag to circumvent the prompt, like gsctl delete cluster has, can be a way to form dangerous habits.

Question:

In vintage, or specifically: where a cluster namespace is used for apps and config, is there an automation in place to delete that entire namespace if a cluster is deleted?

marians commented 1 year ago

Spec v2

For your feedback

Changes compared to Spec v1:

Removed --organization flag
Removed --keep-app-config flag. Instead --keep-apps will also mean that configmaps and secrets will not be deleted.
Modified confirmation prompt
Added spec for dry-run output

General description

Command to delete workload cluster resources on a management cluster. In terms of the semantics, we provide this command to express that a certain workload cluster should not exists. This means that the command does whatever required to remove resources belonging to this cluster, even if the cluster has not been created completely.

General syntax

kubectl gs delete cluster NAME [FLAGS]

Flags

--namespace: The namespace in which resources should be deleted. If not specified, all organization namespaces the user has access to will be searched for a cluster matching the given NAME.
--keep-apps: If set true, user-installed App resources for this cluster and their config (ConfigMaps, Secrets) will not be deleted. Does not affect the cluster-app and the default-apps bundle, which will be deleted in any case. (Default: false)
--dry-run: If set true, a list of the resources to be deleted will be printed, but deletion will not take place. (Default: false)

Example commands

Delete cluster abc12 and find the namespace automatically:

$ kubectl gs delete cluster abc12

Delete cluster abc12, to be found in namespace org-acme:

$ kubectl gs delete cluster abc12 \
  --namespace org-acme

Show what deleting cluster abc12, to be found in namespace org-acme, would mean:

$ kubectl gs delete cluster abc12 \
  --namespace org-acme \
  --dry-run

Cluster info and confirmation prompt

If cluster and node pool resources exist for the cluster to be deleted, information will be shown and a prompt asks for confirmation:

You are about to delete this cluster:

Name:              abc12
Installation:      gollum
Region:            eu-westeurope-1
Service priority:  HIGHEST
Description:       Prod cluster K8s 1.25
Created:           12 May 2022 - 5 months ago

Node pools:        2
Worker nodes:      12

Do you really want to delete this cluster? There is no undo!
If yes, please type the installation and cluster name,
separated by a slash, and hit Return.
If no, enter anything else or hit Ctrl-C.

> gollum abc12

Error: Installation and cluster name do not match the format 'gollum/abc12'.

$>

TODO: Decide what to show if no cluster resource, or no node pool resource, exists for the cluster.

Dry-run output

The following resources will be deleted directly:

NAMESPACE   KIND       NAME                            API GROUP                  VERSION
org-acme    App        abc12                           application.giantswarm.io  v1alpha1
org-acme    App        abc12-default-apps              application.giantswarm.io  v1alpha1
org-acme    App        abc12-nginx-ingress-controller  application.giantswarm.io  v1alpha1
org-acme    ConfigMap  abc12-userconfig                                           v1
org-acme    ConfigMap  abc12-default-apps-userconfig                              v1

Note: This list only shows resources that will be deleted through kubectl-gs directly.
More resources (cluster, node pools) will be deleted as a consequence, if they exist.

Behind the scenes

Based on the namespace and the --keep-apps flag, we compile a list of resources to be deleted.
We also look up whether actual main cluster resource exists. We store the region, service priority label, cluster description from the main cluster resource if it exists.
If a cluster resource exists, we also look up whether node pools exist and store the number of worker nodes.
For the resources in the "to be deleted" list, we check get and list permissions, to ensure that the user can find all existing resources to be deleted. If permission is lacking for at least on of the resources, we exit with a specific error message that lists all missing permissions.
Then existing resources to be deleted are looked up.
For all resources to be deleted, we check whether the acting user has permission to delete. If permission is lacking for at least one of the resources, we exit with a specific error message that lists all missing permissions.
If --dry-run is true, we print a list of all resources to be deleted and exit.
If the main cluster resource exists, we show a general info block regarding the cluster (confirmation prompt).
Then we delete resources one by one and print one line for each one.

Special cases

The cluster specified by name cannot be found at all: This will be reported as an error with exit code > 0. We see it as an indicator that the user may have mistyped the name or namespace.

marians commented 1 year ago

Addition to Spec v2

If a workload cluster is managed through GitOps/Flux, deleting via the CLI command should be prevented, just like it's the case in the web UI.

We will use the existence of the two labels kustomize.toolkit.fluxcd.io/name and kustomize.toolkit.fluxcd.io/namespace on the cluster app resource as an indicator for this.

In this case, the attempt to delete such a cluster should yield the following error message:

Error: cluster 'abc12' is managed through GitOps. It cannot be deleted using this command. Instead, delete the defining resources from the source git repository.

marians commented 1 year ago

Blocked by #1599

gusevda commented 1 year ago

@marians based on this PR description

All apps for a cluster not being managed by cluster-apps-operator are being deleted when cluster CR is deleted.

How the --keep-apps flag is going to work in this case?

marians commented 1 year ago

Talked with Dmitry.

--keep-apps is actually not possible from the client side, as the cluster-apps-operator will automatically delete apps and config once the cluster is deleted.

Consider it removed from the spec.

weatherhog commented 2 weeks ago

@gusevda @marians are you okay with closing this one? Or is this still needed even with flux?

marians commented 1 week ago

Still relevant to me

giantswarm / roadmap