Add support to pull OCI kustomize manifests

[NissesSenap]

Eschewed features

• [X] This issue is not requesting templating, unstuctured edits, build-time side-effects from args or env vars, or any other eschewed feature.

What would you like to have added?

I would like to make it possible to build and pull OCI artifacts containing Kustomize manifests.

This is a feature that the flux community have implemented https://fluxcd.io/flux/installation, https://fluxcd.io/flux/cmd/flux_pull_artifact/

So a simple example, you can write the following and it will download the OCI artifact and unpack it.

flux pull artifact oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1 --output ./grafana-operator/

For those of us that uses flux it's nothing strange, but for none flux users it would be nice to be able to do the same using Kustomize.

So adding oci support to localize and build to be able to do something like this would be nice:

kustomize localize oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1

Why is this needed?

OCI is getting a big foothold when it comes to artifact management in the CNCF landscape, and it will be used more and more. The possibility of signing and easily versioning manifests is something that should be possible through kustomize

Can you accomplish the motivating task without this feature, and if so, how?

It's possible to solve this by using flux, but since it's a third party tool, I don't think the majority of kustomize users will want to do it.

What other solutions have you considered?

N/A

Anything else we should know?

It would also be nice to add support to create OCI artifact from manifests, I haven't thought of a potential good syntax. As I see it, it could be done in two seperate features. I would start with pulling, since OCI artifacts containing kustomize files already are available today.

I started this conversation in https://github.com/kubernetes-sigs/kustomize/issues/4996#issuecomment-1495925893 but I wanted an issue that I could reference.

Feature ownership

• [ ] I am interested in contributing this feature myself! 🎉

[NissesSenap]

Ping @natasha41575 I created a separate issue for this to make it a bit more visible. I hope that was okay.

[FrenchBen]

Dibs

[natasha41575]

/triage accepted

I think we should support OCI for both kustomize build and kustomize localize.

Thanks for filing the issue!

[natasha41575]

Providing some more details on the requirements here:

kustomize localize and kustomize build should support the exact same feature set. If we add oci support to localize, we also need to add it to build.

That means that if we support localizing an oci manifest e.g. kustomize localize oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1, we also need to support building an oci manifest like kustomize build oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1. Note that this is already supported for git URLs.

Oci support also means that we support oci resources in the kustomization file. For example, we currently support:

# kustomization.yaml
resources:
- https://github.com/kubernetes-sigs/kustomize//examples/multibases?ref=v1.0.6

If we say we support oci in addition to git, we need to support:

# kustomization.yaml
resources:
- oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1

This case needs to be handled by both kustomize build and kustomize localize.

Implementation-wise, both localize and build can use the same underlying code as it currently does for pulling remote git urls.

The implementation also needs to include tests for both localize and build. These can be found in api/krusty/remoteload_test.go, kustomize/commands/localize/localize_test.go, and probably some other places. The implementation for build also needs to ensure that origin annotations get added correctly when options.originAnnotations is set in the kustomization file, with corresponding tests added to api/krusty/originannotation_test.go.

[FrenchBen]

@natasha41575 I went ahead and implemented the changes requested, however I couldn't find details around the origins.originAnnotations that you've mentioned

The implementation for build also needs to ensure that origin annotations get added correctly when options.originAnnotations is set in the kustomization file, with corresponding tests added to api/krusty/originannotation_test.go.

Where can I see docs on the above?

[erikgb]

@FrenchBen Did you ever open a PR with your implementation? I am also very interested in this feature!

Where can I see docs on the above?

I don't think this feature was ever documented. It seems to have been added by @natasha41575 in https://github.com/kubernetes-sigs/kustomize/pull/4065. From a user perspective, I expect this new feature to just set the config.kubernetes.io/origin annotation in resources emitted from the OCI repository.

# kustomization.yaml
resources:
- oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1
buildMetadata: [originAnnotations]

Should probably add something like

metadata:
  annotations:
    config.kubernetes.io/origin: |
      path:  oci://ghcr.io/grafana-operator/kustomize/grafana-operator:v5.0.0-rc1/default/kube-prometheus-stack.yaml

to all resources emitted? See https://github.com/kubernetes-sigs/kustomize/blob/336bc14e1a1c5733564ffb6c9d33f0be04671ee1/api/krusty/originannotation_test.go

[NissesSenap]

Fancy seeing you here @erikgb :) @FrenchBen have been working on the implementation, you can find the pr here https://github.com/kubernetes-sigs/kustomize/pull/5147

[petermicuch]

@natasha41575 - has this been put on hold? I've seen last relevant activity in the referenced pr #5147 back in august. It would be great to have this. I am handling dockers and helms as OCIs and am missing this for kustomize as well. This would allow me to handover complete app deployment composed of docker, helm and kustomize as a single reference to OCI registry which would be really cool.