resources: support remote files

[pohly]

Currently, resources entries must be local files. It would be useful to allow references to remote files, for example a file served by a HTTPS server.

The use case I have in mind is about pulling the rbac.yaml files that Kubernetes-CSI publishes for each sidecar app into a deployment: https://github.com/kubernetes-csi/external-provisioner/blob/master/deploy/kubernetes/rbac.yaml

Originally this was suggested in https://github.com/kubernetes-sigs/kustomize/issues/254, but that got closed by just updating the docs. https://github.com/kubernetes-sigs/kustomize/issues/923 is related, but covers only bases.

Pulling remote files has security implications. When using HTTPS, one has to trust the remote server and relies on HTTPS security to prevent man-in-the-middle attacks. It would be nice to support defining and checking an expected file content hash.

[Liujingfang1]

resources should be from local disks, which is different from the bases. We don't plan to support remote files for resources. What you could do is to submit a PR to kubernetes-csi/external-provisioner, adding a kustomization.yaml there. Then you can use it as a remote base.

[pohly]

Jingfang Liu notifications@github.com writes:

resources should be from local disks, which is different from the bases. We don't plan to support remote files for resources. What you could do is to submit a PR to kubernetes-csi/external-provisioner, adding a kustomization.yaml there. Then you can use it as a remote base.

I had thought of that, but it doesn't make sense to me. Why should upstream repos have to had a kustomization.yaml file when they don't actually use kustomize?

[alexpkalinka]

resources should be from local disks

Can you explain why? What is the problem with loading remote resources? If a base kustomization.yaml is loaded from a remote server then it eventually loads resources from a remote server too, right? Then why can't we just load remote resources directly, bypassing loading a base?

[benjamin-bergia]

Forcing people to have this kustomization.yml on their repo makes the remote base feature pretty much useless in real life scenarios. The idea of being able to load some yaml from whatever repo on github, untouched, and create variants from it will not happen if we expect repo maintainers to create kustomizations. In an other hand, if we allow remote resources we automatically give access to any resources to all kustomize users. A work around would be to use git submodules and a kustomization.yml file but that's ugly.

[evrardjp]

I would like to consume resources directly from URL.

I don't see a security issue if the manifest from the resource is audited before use, and I don't see a good technical reason why it cannot be achieved. Could you clarify why there is no plan for this?

I even don't mind tackling the code on my spare time, but I need a longer explanation than We don't plan to support remote files for resources.

[longility]

This is my need for kustomize to support remote files.

I would like to manage my k8s resources in a consistent and declarative way as much as possible. I'm trying out to not be dependent on helm. Kustomize is definitely working out for me at the moment. However, being able to reference remote file urls will help with referencing 3rd party yamls such as for cert-manager, ingress-nginx, jaeger, and kubernetes dashboard. At this time, I'm pulling the code into my repository.

Is there a cleaner way that I don't know about?

[paulgmiller]

Has anyone brought this up at the kustomize sig meeting. Seems like it would be trivial to make a pr to push the discussion further but I agree it's not really clear why from @Liujingfang1 remote resources couln't be done.

[paulgmiller]

Reflected this over to https://github.com/kubernetes/kubernetes/issues/82459 in case kustomize development has moved there.

[paulgmiller]

Another fun example would be installing knative https://github.com/knative/serving/releases/tag/v0.8.1 Maybe should provide a kustomization.yaml but as is they provide a couple of yamls you can combine manually love to refernce them directly from my kustomziation.yaml rather than mirroring.

[sbose78]

I see an example with a remote resources file

DEMO_HOME=$(mktemp -d)

cat <<EOF >$DEMO_HOME/kustomization.yaml
resources:
- github.com/kubernetes-sigs/kustomize//examples/multibases?ref=v1.0.6
namePrefix: remote-
EOF

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md#remote-targets

It doesn't work.

[Liujingfang1]

@sbose78 Which version do you use? I checked with HEAD and it works.

[evrardjp]

@Liujingfang1 does that work for resources that aren't a base? Resources which are bases are fine, it's for the rest of the resources that remote data could be useful.

[lukasmrtvy]

@Liujingfang1 does not work for me, ( kubectl v1.16.3, it should use 2.0.3 -> https://github.com/kubernetes/kubernetes/blob/release-1.16/go.mod#L487)

DEMO_HOME=$(mktemp -d)

cat <<EOF >$DEMO_HOME/kustomization.yaml
resources:
- github.com/kubernetes-sigs/kustomize//examples/multibases?ref=v1.0.6
namePrefix: remote-
EOF

kubectl kustomize .

Error: rawResources failed to read Resources: Load from path github.com/kubernetes-sigs/kustomize//examples/multibases?ref=v1.0.6 failed: evalsymlink failure on '/tmp/tmp.AotZwipVEC/github.com/kubernetes-sigs/kustomize/examples/multibases?ref=v1.0.6' : lstat /tmp/tmp.AotZwipVEC/github.com: no such file or directory

I tried standalone kustomize binary 3.4.0 and its working also with resources. The problem is that You have to need kustomize.yaml inside remote repository, which is useless ( in real world examples ) when You want to include manifest from 3rd party repository that does not use kustomize.

[kYann]

Same here, I need this feature, and needing the kustomize.yaml on the remote side is not possible.

[JohnPaton]

@Liujingfang1 Also putting in my support for this feature. I've had PRs on a couple of projects now trying to get maintainers to add kustomization.yamls which, while good for kustomize's adoption rate, is obviously cumbersome.

Latest example: Here Google provides a DaemonSet for installing Nvidia drivers on nodes which I would like to have included in my build, but it's not possible because I can't use their file directly and they don't have a kustomization.yaml (yet).

The first reply to this issue that

resources should be from local disks, which is different from the bases

also doesn't really apply anymore since bases has been deprecated since 2.1.0; resources is the only place left

[cmorent]

Any update on this issue? I also need this for quite a simple use case: I want to install the Jaeger operator on one of my clusters, so far they only provide yaml files to run the required resources but no kustomize related config so I can't import the base. I only want to have something like the following in my own config:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: default

resources:
- https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml

For now, if I understand well, the best solution I have is to copy the remote file content into a local file.

[alwaysastudent]

Is this supported now?

[DiegoPomares]

For anyone stumbling upon this issue while googling:

It is supported now, but the documentation is outdated as of today as this URL specifications should follow the hashicorp URL format. The directory must contain a kustomization.yaml file. is not true, at least for the standalone kustomize v4.3.0.

Just put a regular URL in the resources section, example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

[karpoftea]

It seems that there is not way to use git-protocol for referencing individual files. So, following will fail:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - git@github.com:argoproj/argo-cd.git/manifests/crds/application-crd.yaml?ref=v2.1.3

with error:

no such file or directory': '/private/var/folders/4x/3kz53w8j2jvgsrzdctn966v40000gn/T/kustomize-194475774/manifests/crds/application-crd.yaml' refers to file 'application-crd.yaml'; expecting directory

While following will work (mind that crds dir has kustimization.yaml inside):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - git@github.com:argoproj/argo-cd.git/manifests/crds?ref=v2.1.3

In my use-case git-repo is private, thus I can't use 'https'-protocol for fetching individual files.

Why there is a limitation for individual files for git-protocol? (upvote please if this feature deserves a separate issue)

[btrepp]

This is a bit of a weird one to stumble upon. Getting the file from git would be a great advantage. Some tools have multiple versions of the .yaml in the directory. E.g argocd. So if you kustomize the whole folder, you will have a bad time.

With the git protocol, we could use the commit hash, so you would also be confident someone sneaky isn't changing the files, and have some form of verification. In the https mode, that's just a URL, someone could change that file and you probably wouldn't know, or get any indication you just built something malicious

[tonglil]

Two years later and running into the issue @karpoftea described above.

Really unknown why the limitation is imposed on the git loader. The https loader is useless in an enterprise context, where storing kustomize files elsewhere is needed...

[HWiese1980]

Has there been any progress here?

[joebowbeer]

The original issue was fixed years ago as this example illustrates:

Resources can reference remote files via their raw GitHub urls, such as - https://raw.githubusercontent.com/kubernetes-sigs/kustomize/8ea501347443c7760217f2c1817c5c60934cf6a5/examples/helloWorld/deployment.yaml

References to remote directories also works.

For references to remote files in private repos using schemes other than file or https, I suggest creating a new issue.

However, given kustomize's preference for folder refs and its restrictions on loading resources not in or below the root folder, even the support for remote files already provided seems questionable to me.

https://kubectl.docs.kubernetes.io/faq/kustomize/#security-file-foo-is-not-in-or-below-bar