jlegrone commented 4 years ago

Background

In #337 we described a method to provide scoped credentials to bundles that install Kubernetes applications, but did not address application definition or deployment tooling. This proposal lays out a solution to another common thread in #285, preventing invocation images from becoming black boxes and allowing runtimes to swap out deployment tooling at will.

In the Kubernetes space alone, there are a myriad of application bundling formats and deployment tools alike. Without a decentralized model for supporting these tools, CNAB implementations are probably destined to have inconsistent behaviors, be difficult to patch, and generally have a hard time keeping up with the larger Kubernetes ecosystem (see Spinnaker's ongoing evolution around Kubernetes and Helm support as an example).

The story isn't all bad, however. Many tools focus exclusively on either manifest rendering (kustomize, ksonnet), or on deployment (kubectl, kapp), and can thus be composed. Some others, like Helm, could conceivably be split into these two phases as well.

Proposal

Objectives

Enable users and CNAB runtimes to provide last-mile customizations to Kubernetes manifests
Allow CNAB runtimes to swap bundled invocation images with custom images appropriate for the application definition format
Keep bundles portable between CNAB runtimes that do or do not support Kubernetes extensions

Design

In this proposal we'll set forth three different patterns for Kubernetes bundles to follow, each providing different levels of visibility from the CNAB runtime.

1. Transparent manifest and pluggable deployment tooling

This is the simplest case. Bundles that do not require any rendering stage should specify their Kubernetes manifest as an OCI artifact:

{
    "images": {
        "io.kubernetes.manifest": {
            "contentDigest": "sha256:deadbeef",
            "image": "example/app/manifest:0.1.2",
            "mediaType": "application/vnd.cnab.kubernetes.manifest.config.v1+json",
            "size": 1337
        }
    }
}

(note that an OCI representation for Kubernetes manifests has not been standardized)

Bundles SHOULD include their own invocation images, but CNAB runtimes that are aware of this spec extension MAY replace the bundle invocation image with a different image or deployment tool when executing the standard install/upgrade/delete actions.

2. Opaque rendering, pluggable deployment tooling

The application bundling format is unknown to the CNAB runtime, but the invocation image supports a custom action for manifest rendering that allows the runtime to take over standard actions with its own deployment tooling as described in the previous example.

The CNAB runtime MAY intercept the output produced by io.kubernetes.render in order to apply customizations before submitting resources to the cluster.

{
    "actions": {
        "io.kubernetes.render": {
            "title": "Render Manifest",
            "description": "outputs manifest to be applied to a Kubernetes cluster",
            "modifies": false,
            "stateless": false
        }
    },
    "outputs": {
        "io.kubernetes.manifest": {
            "applyTo": [
                "io.kubernetes.render"
            ],
            "path": "/cnab/app/outputs/manifest"
        }
    }
}

3. Opaque rendering and deployment tooling

There are various reasons why custom deployment tooling might be required in order to install an application, including executing imperative lifecycle actions like creating/restoring a database backup, or waiting for readiness of one or more custom resources.

In these situations CNAB runtimes MAY rely on the proposal outlined in #337 for providing properly scoped credentials. Bundles MAY declare support for last-mile transformations via kustomize by accepting a "well known" io.kubernetes.kustomization parameter:

{
    "parameters": {
        "io.kubernetes.kustomization": {
            "destination": {
                "path": "/cnab/app/kustomization.yaml"
            }
        }
    }
}

carolynvs commented 4 years ago

For the first section, is the intent that the runtime would look for bundles that have an image of type io.kubernetes.manifest and then when present swap the invocation image defined in the bundle with its own invocation image? For example, a "trusted" (outside of the scope of this issue how or why it's trusted) invocation image that has just has kubectl and kustomize on it.

carolynvs commented 4 years ago

For the second section, am I correct that the runtime would look for the custom io.kubernetes.render action and automatically execute it before the well-known actions (install/upgrade/uninstall) and use the output as the manifest sent to that action? It would not be necessary for the user to first run render, then figure out how to pipe the manifest output from render into install.

trishankatdatadog commented 4 years ago

So well-written, good job, Jacob!

glyn commented 4 years ago

Yes, thanks for this issue Jacob! You asked for some feedback...

I think this proposal helps CNAB provide close to the middle point in the spectrum of experiences that @jbeda described in issue 285:

From there we can build different types of experiences — these are not exclusive:

A purely declarative install mechanism. All necessary artifacts (with metadata about those artifacts) would be included in the bundle. External tools would be required to interpret and use those artifacts to do... something. It could be installing something but it could also be other things over time.

Move the install tool chain into a container (or set of containers) that are left as a suggestion in the metadata on how to use the artifacts. It would be up to the user to use (or not use) those containers.

Have everything bundled into the install image much as it is today. That image would is opaque and must be used to install the application as some of the required artifacts may be baked into that image. For what it is worth, we are interested in the first couple of options but I really don’t want to support or encourage our users to do the third.

It's not precisely the same experience described in the second bullet, because the invocation image is still present in the bundle and not just a suggestion, but it's in the right ball park.

However, this proposal certainly seems quite different to the experience described in the first bullet. The intention there is that the bundle contains purely declarative metadata, such as Kubernetes manifests, and does not contain an invocation image. If I've understood correctly, this proposal does not require a CNAB spec change to allow the invocation image to be omitted from a bundle.

jlegrone commented 4 years ago

It's not precisely the same experience described in the second ^(first?) bullet, because the invocation image is still present in the bundle and not just a suggestion

@glyn you're right, this proposal still includes an invocation image in bundles that implement the Kubernetes extension. This is necessary in order to keep bundles compatible with CNAB runtimes that do not support the non-normative extension to the spec.

A purely declarative install mechanism. All necessary artifacts (with metadata about those artifacts) would be included in the bundle. External tools would be required to interpret and use those artifacts to do... something.

This is the use case I had in mind for point 1 in this proposal, but I find the wording here to be slightly overly prescriptive. IMO the spec should not enforce that bundles must be installed via some tool external to the bundle, but should instead provide the metadata necessary for an opinionated runtime to enforce policy, scope credentials, or delegate to an alternative deployment tool.

As a bundle vendor, I want to be capable of shipping one bundle that works out of the box with barebones CNAB installers, but also supports progressive security enhancement and introspection for more advanced runtimes. These two goals don't seem like they have to be at odds with one another.

glyn commented 4 years ago

I see, then I think our goals are different. Point 1 is aiming for a bundle without an invocation image that can be installed only by runtimes that understand the relevant Kubernetes metadata/extension(s). This gets rid of all ActiveX connotations and will make CNAB much more broadly usable within the Kubernetes community.

I've raised a separate spec change for making the invocation image optional in issue 352.

carolynvs commented 3 years ago

Closing this after our CNAB meeting today. This isn't something that we feel is a priority right now, but we can certainly revisit at a later time.

cnabio / cnab-spec

Proposal: Extension for Declarative Kubernetes Installers #338