Separate Invocation Image from Artifacts

[SteveLasker]

The current spec includes all artifacts used to deploy a bundle within the invocation image. This means that any change to a helm chart, a compose file, an OPA Bundle will require the invocation image to be rebuilt. It also avoids the ability to have common invocation images shared across different bundles, or have the common invocation image signed by an external entity.

The proposal is to separate invocation images to be just the binaries, and possibly mix-ins required to do a deployment, from any other artifact used for that deployment.

Scenario: Azure, AWS, Google make a invocation images available for deploying to their respective clouds. The cloud specific invocation images have mix-ins packaged within the invocation image. The invocation images are signed by the respective clouds, so they know what is capable.

A developer wishes to deploy an app that includes Kubernetes, storage, IP addresses, and a Redis Cache instance.

They reference the cloud providers invocation image. The Helm chart and ARM template are also referenced in the bundle. When the bundle is "built", the Helm chart is pushed to the registry as an individual OCI artifact type. The ARM template can be pushed as an OCI artifact type, or it could be pushed as a cnab.artifact, meaning CNAB provides for generic artifacts that don't yet have an OCI CLI. An OCI Index is created that references these artifacts, with a reference to an invocation image, and the images used to run the app in AKS. Any CNAB configuration information is placed in the invocation images config. Alternatively, the specific CNAB config for this bundle could be a separate artifact as well.

By separating these elements out, we get:

• Builds don't actually require docker build. In most cases, a developer is likely referencing existing invocation images and only iterating on the artifacts used for the deployment
• Invocation images are well known, signed and approved by the different clouds, or different IT shops of companies. They don't have to worry about any code executing at deployment time.
• By referencing mix-ins within an invocation image, the execution environment can understand the intent of the CNAB and decide if this bundle should have specific mix-in rights. If not, it can be blocked before a partial deployment is started, making it near impossible to rollback.
• Referencing artifacts as external entities allows re-use, as multiple bundles may reference the same helm chart, OBA, singularity image, Terraform template, ...

[SteveLasker]

Responding to: https://github.com/deislabs/duffle/issues/749#issuecomment-491550081

The issue isn't just about having to rebuild the invocation image each time. It's a collection of the 4 bullets at the end of the issue. If I have a set of artifacts, Images, Helm Charts, Singularity Images, ARM or Cloud Formation Templates, OBA Bundles already in the registry, why would I want to duplicate them into an invocation image? A registry is intended to store individual things (layers, images). References to things that already exist reduce registry sprawl.

Each individual artifact can be signed. The CNAB that references artifacts can be signed. We're getting into separate issues that could be logged, but the CNAB author might want to reference a stable tag, so patches to that tag are automatically picked up. They may want to reference the digest, so they're locked to a given artifact version.

[squillace]

I like the flexibility the idea brings, of course. Without taking my time about it, however, I fail to see the immediate need to add the flexibility. The spec currently works well.

At the moment, I see that CNAB heading toward RC and then 1.0 by rebuilding the bundles, which are OCI images and we can ensure guarantees the specification makes. This issue would radically alter the spec, resulting in a few improved scenarios as stated, but also resulting in a ton of thought-work in advance before we end up doing a bunch of coding across all the tools. That alone would mean that 1.0 of a spec a lot of people have worked quite a bit on would be yet further off, so at this point only on those grounds I'm not in support, though it looks like the perfect issue to turn around and tackle after 1.0...

That's my take at the moment, but I'll read and think this through as well because the features are good, although I am not at all convinced that the added flexibility doesn't bring a reduction in some other area of functionality and/or requires all tools to do work for benefits that they do not see. It'll take some time to decide on that.

My two cents for now....

[GreenCee]

I may be wrong, but I interpreted as the door was open to more than just OCI/Docker images in the image map, it just happens that the only imageType used right now has been docker and OCI https://github.com/deislabs/cnab-spec/blob/master/103-bundle-runtime.md#image-map

Should be explicitly draw OCI/Docker as the supported imageType in 1.0?

With the addition of 'relocation' as a concept (Distribute and update bundle.json) a CNAB runtime would avoid the burden of knowing what or how to treat these differently. Typically the invocationImage will be reading the "image": value. imageType: docker vs. imageType: helm

Artifacts (images) could be embedded in the invocationImage, but best practice would be to externalize the artifact and gain the benefits of the registry ecosystem. This does highlight that I think we do need a catch-all cnab.artifact. MediaType/ImageTypes should evolve in paralell with registry support.

Somewhat related, #95 uncovered a lot of these assumptions

[simonferquel]

I like the idea of generic invocation images, and I think we can leverage the "component" images to store artifacts such as compose files etc. without evolving the spec.

I would love to see an experiment of that in docker-app as we already do a docker-build just to package artifacts on top of a common base invocation image. @chris-crone what about doing a quick POC on docker-app to check if it is doable without a change of spec (and maybe adopt it for a future version) ?

[radu-matei]

@GreenCee - I'm not sure we need to necessarily restrict the image type to OCI / Docker - and I'll let @SteveLasker comment on this more, but technically oras could push any image type - although it might make sense to restrict the initial testing to OCI images.

Another question I have is how would a bundle.json keep track of all referenced artifacts, and how should runtimes work with registries to store these artifacts? Would they be pushed in the same repository / registry as the bundle, invocation images and images?

I think I like the idea, but as @squillace mentioned, it feels rather late in the process for v1.0 of CNAB Core - although it might be a perfect item to seriously start discussing what we want from v2.0 of CNAB Core (and a great talking point for the registry specification).

[SteveLasker]

It seems the biggest confusion is around being able to store different things, and differentiate what they are within OCI Artifact Registries. Singularity is one of the best examples of this. Sylabs was stuffing their SIF formats into the docker/OCI image format, but had to convert it after pulling. Which ironically meant a high performance computing optimization, had to first be converted. By defining a different mediaType, they have the freedom to choose their own optimized layout of the layers. The latest work on OPA, from our friend @garethr also leverages this flexibility to store other formats in OCI Artifact Registries. The thing about the single invocation image, that contains all the objects, besides the images they deploy, means we wind up with lots of duplications of artifacts that CNABs will reference. A key value of a registry is being able to resolve duplicate links. Users have options of stable and unique tags. They can patch stable tags, reference specific tags (which could still be updated), or reference non-updated tags, or digest if they wish to truly lock to a specific version. Bundling these together loses this core value of a registry. As for the mediaTypes, and how these differentiate the artifacts; this is all great feedback, and I've been updating my deck for OCI Artifact Registries - Cloud Native Rejekts to help clarify this. I believe the talks are recorded, but I'm sure we'll be doing more to help clarify this.