olm-curated-catalog proposal

[cdjohnson]

This is an initial proposal to support creating curated catalog index images from other catalog index images.

Superceded by PR #14 (packages/channels)( PR for os/architecture filtering forthcoming...

[kfox1111]

It would be great if this was not tied to the openshift cli. Not sure if this is recommending making a new tool or using an existing openshift one.

Potentially even better, It would be interesting if it were an operator such that you put in a CR for creating a curation and it continuously watched an existing CatalogSource and built the curated one from it. Then no special cli would be needed at all. Users could manage curation just like any other k8s object then.

[kevinrizza]

It would be great if this was not tied to the openshift cli. Not sure if this is recommending making a new tool or using an existing openshift one.

Potentially even better, It would be interesting if it were an operator such that you put in a CR for creating a curation and it continuously watched an existing CatalogSource and built the curated one from it. Then no special cli would be needed at all. Users could manage curation just like any other k8s object then.

Interestingly, I believe there is a proposal floating around in the OpenShift space that discusses using some kind of runtime filter to select what architectures are actually available for the cluster so that OLM can only install that subset.

[mhrivnak]

Here is the beginning of a related proposal: https://github.com/operator-framework/operator-registry/issues/203

I suspect the most common use cases will be this:

1. As a user, I want to use a vendor's index image to make my own index image with only the latest version of the operators I care about, and then mirror all associated content into my disconnected environment. I don't care about anything but the most recent releases in each channel, because I haven't installed anything yet.
2. As a user who did ^ last week, I want to efficiently upgrade to the latest versions of the operators I care about. If possible, I prefer to upgrade directly to the head of each channel. I prefer not to download and mirror a bunch of intermediate releases (that could add up fast!) unless absolutely necessary.

To accomplish this, the user is going to look at what's available in the vendor's index and produce a list of operators they have chosen to deploy. That list will include package name, channel name, and maybe a version expression. In order to make the update process automateable, I see a lot of value in designing the tooling to receive such a list as input and produce an index image accordingly. Then the tool can be run daily/weekly/monthly/quarterly/etc with the same input, and produce an index with the latest updates.

The proposal I linked suggests using Subscriptions as the data structure. I again think that will be a common starting point, but the main idea is to make it easy to automate the delivery of updates.

[cdjohnson]

@shawn-hurley

--filter-by-os/arch commands would require manipulation of released content (Bundle images) to work correctly. This would require re-building the bundle image on the fly. I don't think that this is something that we really want to do.

What are the alternatives? The bundle image is currently being used as a tarball, so I was thinking that it would need to be treated as such. Isn't the Operator Certification process already manipulating the bundles to add relatedImages?

--filter-by-version what is the use case for this? If we try to patch the update graph together that we are setting up operator authors for failure, as they no-longer can define their update graph. I think we need to tread carefully here.

The main use case is to reset the START version of the graph. For example, if an operator has been out there for a few years, and the user only cares about >=10.x, why would they want to mirror all of GB/TB of old images?

[shawn-hurley]

The main use case is to reset the START version of the graph. For example, if an operator has been out there for a few years, and the user only cares about >=10.x, why would they want to mirror all of GB/TB of old images?

Index images should solve this already in my opinion, the Operator Author chooses what is available for a given index image (per release today). I don't see folks keeping old releases in the new index images so I am wondering if we even need to do this.

What are the alternatives? The bundle image is currently being used as a tarball, so I was thinking that it would need to be treated as such. Isn't the Operator Certification process already manipulating the bundles to add relatedImages?

The old method, this was possible because the metadata was just a tar.gzip of data so the certification pipeline could do this. In the new world, you will not just be able to change the manifests in a bundle, because the SHA is used in the index image. This all feels very complex and may be needed in the future, but I am wondering if we can add this later and just get the channel and operator filtering and see how far that gets us?

/cc @ecordell

[shawn-hurley]

As for ongoing maintenance @mhrivnak did we consider just re-using the published bundle image for the update, the pruned index image and the opm index add command?

[cdjohnson]

The main use case is to reset the START version of the graph...

Index images should solve this already in my opinion, the Operator Author chooses what is available for a given index image (per release today). I don't see folks keeping old releases in the new index images so I am wondering if we even need to do this.

So, you're envisioning each Package supporting some time-frame of versions (e.g. 1-year support window), where the Package owner/author prunes old versions (using skipRange or semver/implicit catalog versioning). This would never work with replaces unless bundles/csvs could be mutated, since the replaced version would no longer exist.

What are the alternatives? ....

The old method [app registry], this was possible because the metadata was just a tar.gzip of data so the certification pipeline could do this. In the new world, you will not just be able to change the manifests in a bundle, because the SHA is used in the index image. This all feels very complex and may be needed in the future, but I am wondering if we can add this later and just get the channel and operator filtering and see how far that gets us?

/cc @ecordell

I agree that supporting arch/os filters is tricky. I'm happy to split this out into a separate enhancement

This enhancement would then be narrowed to: filtering on: Package and Channel with the assumption that all Channels are using skipRange or the upcoming implicit catalog versioning feature to build the graph.

Then, in theory, if the Channels are using the auto z-build scheme, versions that are not included in those channels would be excluded from consideration.

If you're good with that, I can clean this up, and create a new set of PRs.

[mhrivnak]

As for ongoing maintenance @mhrivnak did we consider just re-using the published bundle image for the update, the pruned index image and the opm index add command?

That's definitely an option. It could be an optimization once we have the ability to create curated index images.

[mhrivnak]

I don't see folks keeping old releases in the new index images so I am wondering if we even need to do this.

AIUI, index images must have the capability to have multiple releases in a channel, because it is possible for an operator to have a strict upgrade graph that does not utilize skipRange, or at least not for the entire range of supported upgrade paths. Even if we were to say that we don't know of any plans to include multiple releases in a channel, I don't think we can assume that it will never happen.

Also, are we really planning to prune index images like that? Would that mean that on every update of the index image, the end user loses access to previous releases?

[shawn-hurley]

This enhancement would then be narrowed to: filtering on: Package and Channel with the assumption that all Channels are using skipRange or the upcoming implicit catalog versioning feature to build the graph.

I am wondering if we can relax the need to have every version that is in a replace field, and then we can handle that type of update graph as well?

I think making this proposal scoped down makes sense to me thanks @cdjohnson!

[cdjohnson]

Closing this PR to separate concerns.