This is a bit of a catch-all issue that touches on a bunch of things, all related to the way kapp creates, updates, and deletes resources and how this behavior is not necessarily clear to users interacting with Carvel packages rather than kapp directly. It seems like it would make sense to look at this as more of an epic that should be refined and broken down into smaller more concrete issues.
Problem Description
The following takes the Tanzu Crossplane package as a concrete example of a situation in which kapp's default strategies for creation and deletion of resources lead to issues:
Crossplane is already installed in the cluster, along with the AWS provider. There are a number of external resources in AWS that have been provisioned via Crossplane (i.e. RDS instances, S3 buckets, etc).
User installs or updates to the latest version of TAP which newly introduces a Crossplane package. They do not realize that there is such a package now and hence do not explicitly exclude it via the TAP values file.
Crossplane package fails to install due to failing updates for certain properties on pre-existing resources (i.e. immutable selector in Crossplane controller deployment).
User notices the installation failure for that package and subsequently excludes the package via their TAP values file.
kapp removes all pre-installed Crossplane resources including all XRDs and Compositions.
External resources in AWS are now orphaned and not managed by Crossplane anymore.
At the very core of the problem here is the fact that kapp silently adopts pre-existing resources and does that despite the error it throws in (3). To users interacting with tanzu package install this behavior is not obvious at all. Hence they do not realize that removing the failing PackageInstall will cause all pre-existing resources to be deleted.
Proposed Short-term Solution
For the particular case of the Crossplane package, we're currently planning to...
Set --existing-non-labeled-resources-check via spec.deploy[0].kapp.rawOptions
Anotate all Crossplane CRDs, Providers, and ProviderConfigs with kapp.k14s.io/delete-strategy=orphan
Expose optional package values that allow users to configure (1) and (2). This way they can, for example, turn off (1) in order to reinstall the package after having installed and deleted it previously in the same cluster.
While this approach seems to be a good short-term solution for this particular package, one can argue that there's a need for a more general solution that addresses these kinds of problems at a higher level. After all, Crossplane isn't the only package that might run into issues like these. Another example is the cert-manager package.
Desired Long-term Outcomes
Think of the following list as suggestions for improvements to kapp-controller and kapp that came up during previous conversations.
Having more explicit documentation covering the default resource lifecycle, strategies, and expectations for users and authors of Carvel packages (bridging the conceptual gap between kapp and tanzu package install). Right now, it seems we're implicitly expecting users to be aware that at their core our packages deploy kapp applications. Furthermore, we expect them to understand the create, update, and delete behavior of kapp.
Having first-class properties on the App, Package, and/or PackageInstall CRs to configure create, update, and delete strategies for resources.
Think about the right default behavior for create, update, delete strategies on the package-level
Think about an (optional) escape hatch for package installations to error out early (i.e. before creating/updating anything) if one or more pre-existing resources are being detected
Document existing command line flags for kapp, e.g. --existing-non-labeled-resources-check and which version they were introduced in
Document which flags and options are allowed in spec.deploy[0].kapp.rawOptions
This is a bit of a catch-all issue that touches on a bunch of things, all related to the way kapp creates, updates, and deletes resources and how this behavior is not necessarily clear to users interacting with Carvel packages rather than kapp directly. It seems like it would make sense to look at this as more of an epic that should be refined and broken down into smaller more concrete issues.
Problem Description
The following takes the Tanzu Crossplane package as a concrete example of a situation in which kapp's default strategies for creation and deletion of resources lead to issues:
At the very core of the problem here is the fact that kapp silently adopts pre-existing resources and does that despite the error it throws in (3). To users interacting with
tanzu package install
this behavior is not obvious at all. Hence they do not realize that removing the failingPackageInstall
will cause all pre-existing resources to be deleted.Proposed Short-term Solution
For the particular case of the Crossplane package, we're currently planning to...
--existing-non-labeled-resources-check
viaspec.deploy[0].kapp.rawOptions
Providers
, andProviderConfigs
withkapp.k14s.io/delete-strategy=orphan
While this approach seems to be a good short-term solution for this particular package, one can argue that there's a need for a more general solution that addresses these kinds of problems at a higher level. After all, Crossplane isn't the only package that might run into issues like these. Another example is the cert-manager package.
Desired Long-term Outcomes
Think of the following list as suggestions for improvements to kapp-controller and kapp that came up during previous conversations.
kapp
andtanzu package install
). Right now, it seems we're implicitly expecting users to be aware that at their core our packages deploy kapp applications. Furthermore, we expect them to understand the create, update, and delete behavior of kapp.App
,Package
, and/orPackageInstall
CRs to configure create, update, and delete strategies for resources.kapp
, e.g.--existing-non-labeled-resources-check
and which version they were introduced inspec.deploy[0].kapp.rawOptions