remote helmfiles via http (go-getter)

[mseiwald]

Hi,

using this minimal helmfile.yaml...

helmfiles:
  - path: https://raw.githubusercontent.com/cloudposse/helmfiles/master/releases/external-dns/helmfile.yaml

... when running helmfile deps I get this error:

no matches for path: https://raw.githubusercontent.com/cloudposse/helmfiles/master/releases/external-dns/helmfile.yaml
There are no repositories defined in your helmfile.yaml.
This means helmfile cannot update your dependencies or create a lock file.
See https://github.com/roboll/helmfile/issues/878 for more information.
err: no releases found that matches specified selector() and environment(default), in any helmfile

Is fetching via HTTP not supported?

Kind regards, Michael

[mumoshu]

@mseiwald Hey! Every helmfile, even the parent, requiers one or more releases to be processed. Otherwise it fails like it. If you do want to opt-out that check, add --allow-no-matching-release like helmfile --allow-no-matching-release $CMD. That's the error, anyway.

Also, I thought I have never tried using go-getter URLs other than git::http my self as noted in https://github.com/roboll/helmfile/pull/648. If you find it not working even after using allow-no-matching-release, please report back.

[mseiwald]

@mumoshu I'm getting the same error message with --allow-no-matching-release.

[mumoshu]

@mseiwald Seems like this doesn't work due to the nature of helmfile's go-getter support.

It is supposed to grab the remote directory and then point a specific helmfile.yaml to be processed. So helmfile has to download the directory https://raw.githubusercontent.com/cloudposse/helmfiles/master/releases/external-dns/ then locate helmfile.yaml in it but in HTTP there's no really a notion of "directory" so https://raw.githubusercontent.com/cloudposse/helmfiles/master/releases/external-dns/ is 404. That's why it doesn't work.

If you're interested in retrieving the helmfile.yaml hosted in github, can you use git::https instead anyway?

[mseiwald]

@mumoshu We are using git:https in other scenarios already. In this case we specifically need to use https since we want to push our helmfiles as artifacts to an internal Nexus repository. (The github URL above only served as a reproducible example)

Any chance of adding support for either single file retrieval (without directory support) or fetching "directories" as .tar.gz or something similar? That would help us enormously.

[mumoshu]

@mseiwald Adding support for fetching remote.tar.gz file as a directory makes sense to me.

Would you mind submitting a PR for that? I don't have my own use-case for that feature so I won't just spend my spare time developing it myself soon. But I'll gladly review a PR and merge it if you could help.

[mseiwald]

I'll give it a try. Thanks for your help!

[mumoshu]

@mseiwald Thanks! At quick glance go-getter seems to have support for tgz https://github.com/hashicorp/go-getter#unarchiving

We have a special syntax based on go-getter URL that is implemented in https://github.com/roboll/helmfile/blob/master/pkg/remote/remote.go#L118. I haven't tried it yet but there may be a change that you need to tweak that code to make helmfile accepts URLs like http://somehost/path/to/dir/whatever.tgz

[mathieux51]

In case someone else lands here and like me want to download a file from S3, the following code worked for me out of the box:

# helmfile.example.yaml
helmfiles:
  - path: s3::https://s3-eu-central-1.amazonaws.com/my-bucket/my-folder/my-chart-0.1.0.tgz@helmfile.yaml.gotmpl
    values:
    - foo: bar

helm package command and helm-s3 plugin were used to package and push the helmfile.yaml.gotmpl file as part of the my-chart chart.

[iAnomaly]

@mumoshu Is there any chance this is supported for bases too or should I open a new feature request for that? Thanks!