search

roboll / helmfile

Deploy Kubernetes Helm Charts
MIT License
4.04k stars 566 forks source link

Cannot use a helm-git chart that depends on a helm-git chart #726

Open mrtndwrd opened 5 years ago

mrtndwrd commented 5 years ago

I have written two helm charts. The first helm chart lists the second helm chart in the requirements.yaml file. I don't have a helm repository, so I use helm-git to download and install these charts. This means that I have a "repository" listing for each of the charts.

When I run helmfile install, I run into trouble when helm diff tries to download the first chart:

Adding repo nextcloud-onlyoffice git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add nextcloud-onlyoffice git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add nextcloud-onlyoffice git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora: 
Adding repo onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora: "onlyoffice-documentserver" has been added to your repositories

"onlyoffice-documentserver" has been added to your repositories

err: helm exited with status 1:
  Error: no repository definition for @onlyoffice-documentserver. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error
changing working directory back to "/root"
in /oas/source/repos/helmfiles/helmfile.d/20-nextcloud.yaml: helm exited with status 1:
  Error: no repository definition for @onlyoffice-documentserver. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error

I understand that this is all very vague, but I wanted to give a short summary of the problem before I went into specifics.

I am talking about the two charts in this repository and branch:

https://code.greenhost.net/openappstack/nextcloud/tree/4-use-onlyoffice-instead-of-collabora

(in time of writing HEAD is at this commit)

The nextcloud-onlyoffice chart includes a job that installs an integration of NextCloud and OnlyOffice, so it's requirements.yaml file lists:

dependencies:
  - name: nextcloud
    version: 1.1.1
    repository: "@stable"
    tags:
      - nextcloud
  - name: onlyoffice-documentserver
    version: 0.1.0
    repository: "@onlyoffice-documentserver"
    tags:
      - onlyoffice
      - nextcloud

The first dependency is no problem, because it uses the @stable repository, but the second (a link to the onlyoffice-documentserver chart in the same git repository) needs you to have that helm repository available.

In the non-helmfile scenario, a user would run 

helm repo add onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora

To add that repository. In this case, however, we want to use helmfile. So we have this second repository with our helmfile configuration:

https://code.greenhost.net/openappstack/helmfiles/tree/7077c08573bac6b87601d6e1d7b8abd7e16ffae3

Here, in helmfile.d/20-nextcloud.yaml, we have listed the two repositories we need:

repositories:
  - name: nextcloud-onlyoffice
    url: git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora
  - name: onlyoffice-documentserver
    url: git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora

But when we run the following command:

/usr/local/bin/helmfile -b /snap/bin/helm -e oas -f /oas/source/repos/helmfiles/helmfile.d/ --log-level debug apply --suppress-secrets

Helmfile exits with this error:

Adding repo nextcloud-onlyoffice git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add nextcloud-onlyoffice git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add nextcloud-onlyoffice git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora: 
Adding repo onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora
exec: /snap/bin/helm repo add onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora: "onlyoffice-documentserver" has been added to your repositories

"onlyoffice-documentserver" has been added to your repositories

err: helm exited with status 1:
  Error: no repository definition for @onlyoffice-documentserver. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error
changing working directory back to "/root"
in /oas/source/repos/helmfiles/helmfile.d/20-nextcloud.yaml: helm exited with status 1:
  Error: no repository definition for @onlyoffice-documentserver. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error

I have done some debugging, and it turns out that at some point helm-git is run with the following environment variables set:

  HELM_PLUGIN_DIR=/root/.helm/plugins/helm-git
  HELM_PLUGIN_NAME=helm-git
  SNAP_USER_COMMON=/root/snap/helm/common
  HELM_PATH_STARTER=starters
  HELM_PATH_REPOSITORY=repository
  HELM_BIN=/root/.helm/plugins/helm-diff/bin/diff
  HELM_PLUGIN=/root/.helm/plugins
  PWD=/tmp/helm-git.Ii2YPX
  HELM_PATH_REPOSITORY_FILE=repository/repositories.yaml
  SNAP_INSTANCE_NAME=helm
  SNAP_USER_DATA=/root/snap/helm/106
  HELM_PATH_CACHE=repository/cache
  SNAP=/snap/helm/106
  SNAP_COMMON=/var/snap/helm/common
  HELM_HOME=
  SNAP_DATA=/var/snap/helm/106
  XDG_RUNTIME_DIR=/run/user/0/snap.helm
  HELM_PATH_LOCAL_REPOSITORY=repository/local
  SNAP_NAME=helm

Now I'm not sure where this is going wrong, because I'm not sure if these environment variables are set by helm-diff or by helmfile, but the problem is that HELM_HOME is set to an empty string, which results in helm-git using a temporary home directory instead of the user's normal helm home, which is the reason why it can not find the @onlyoffice-documentserver repository definition.

So I'm aware that this might be a problem in one of the plugins, but I'm not experienced enough with the whole toolset to find the exact cause of this problem on my own. Since helmfile supports using helm-git I thought I'd start here, and I'll open an issue with either plugin if we figure out that they're the culprit.

Any help would be appreciated!

mumoshu commented 5 years ago

@mrtndwrd Hey!

First of, do you set HELM_HOME to a custom location yourself, or just leaving it unset when running helmfile?

Also - I may be confused but would you mind removing the @ within your repo name in requirements.yaml? Wondering if it makes any difference.

Lastly, it might not be your case, but when you encounter https://github.com/aslafy-z/helm-git/issues/16 try adding a sparse=0 query to the repo url.

mumoshu commented 5 years ago

helm seems to empty HELM_HOME when calling a helm plugin, but that's the problem other plugins should be affected as well. That's why I'm doubting @ in the repo name is considered as an actual part of the chart name, which doesn't exist in your repo list.

mrtndwrd commented 5 years ago

Hey @mumoshu thanks for the help! It's much appreciated :)

First of, do you set HELM_HOME to a custom location yourself, or just leaving it unset when running helmfile?

I didn't, but I'll try it now to set it and see if that changes behaviour.

...

Nope. Seems like you're right: if I override HELM_HOME helm will empty it.

Also - I may be confused but would you mind removing the @ within your repo name in requirements.yaml? Wondering if it makes any difference.

If I remove the @ from the repo name in the requirements.yaml it doesn't work anymore in the normal way: 

$ helm repo add onlyoffice-documentserver git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora
"onlyoffice-documentserver" has been added to your repositories
$ cd nextcloud/nextcloud-onlyoffice
$ helm dep update
Error: no repository definition for onlyoffice-documentserver. Please add them via 'helm repo add'
Note that repositories must be URLs or aliases. For example, to refer to the stable
repository, use "https://kubernetes-charts.storage.googleapis.com/" or "@stable" instead of
"stable". Don't forget to add the repo, too ('helm repo add').

I can, however, add the repo as a URL instead of an alias:

  - name: onlyoffice-documentserver                                              
    version: 0.1.0                                                               
    repository: "git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora"

But now when I try to install, I get this: 

err: helm exited with status 1:
  Error: no repository definition for git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error
changing working directory back to "/root"
in /oas/source/repos/helmfiles/helmfile.d/20-nextcloud.yaml: helm exited with status 1:
  Error: no repository definition for git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error

So basically the same error, because however you specify the repo, you still need to have it listed when you run helm repo list, but with HELM_HOME overridden for plugin execution that doesn't work. For completeness, when I run helm repo list normally, I get this result: 

$ helm repo list
NAME                        URL                                                                                                                      
stable                      https://kubernetes-charts.storage.googleapis.com                                                                         
local                       http://127.0.0.1:8879/charts                                                                                             
jetstack                    https://charts.jetstack.io                                                                                               
onlyoffice-documentserver   git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora

Lastly, it might not be your case, but when you encounter aslafy-z/helm-git#16 try adding a sparse=0 query to the repo url.

I tried that, but it doesn't make a difference:

...
err: release "oas-test-files" in "20-nextcloud.yaml" failed: failed processing release oas-test-files: helm exited with status 1:
  Error: no repository definition for @onlyoffice-documentserver. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: plugin "helm-git" exited with error
  Error: plugin "diff" exited with error
changing working directory back to "/root"
in /oas/source/repos/helmfiles/helmfile.d/20-nextcloud.yaml: failed processing release oas-test-files: helm exited with status 1:
  Error: no repository definition for @onlyoffice-documentserver. Please add them via 'helm repo add'
  Error in plugin 'helm-git': Error while helm_dependency_update
  Error: plugin "helm-git" exited with error
  Error: plugin "diff" exited with error
$ helm repo list
NAME                        URL                                                                                                                               
stable                      https://kubernetes-charts.storage.googleapis.com                                                                                  
local                       http://127.0.0.1:8879/charts                                                                                                      
jetstack                    https://charts.jetstack.io                                                                                                        
onlyoffice-documentserver   git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora&sparse=0
nextcloud-onlyoffice        git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora
mrtndwrd commented 5 years ago

helm seems to empty HELM_HOME when calling a helm plugin

As far as I can tell it seems to not empty it but set it to whatever is in settings.Home, but I don't really understand why that doesn't default to /root/.helm in my case :thinking:

aslafy-z commented 5 years ago

Hi @mrtndwrd! Thanks for reporting, I'm unable to test now as it seems your git server is blocked at my workplace. Gonna try to debug when I'm back home. I added a Troubleshooting section in the helm-git repo, feel free to have a try. Thanks @mumoshu for your input!

mumoshu commented 5 years ago

I started to think if you've incorrect a repository url. A repository URL must point to the parent directory of the charts, not the chart directory itself.

This is not correct:

git+https://code.greenhost.net/openappstack/nextcloud@nextcloud-onlyoffice?ref=4-use-onlyoffice-instead-of-collabora

But this is:

git+https://code.greenhost.net/openappstack/nextcloud?ref=4-use-onlyoffice-instead-of-collabora

Now see your helmfile.yaml and if you still have the incorrect URL under repositories, try something like:

repositories:
  - name: nextcloud
    url: git+https://code.greenhost.net/openappstack/nextcloud?ref=4-use-onlyoffice-instead-of-collabora

releases:
- name: nextcloud-onlyoffice
  chart: nextcloud/nextcloud-onlyoffice

I can't access it via git+https as I have no read permission to it, but worth trying!

aslafy-z commented 5 years ago

@mumoshu Sadly, helm-git still needs a path section within the repo URL, even if it's empty (see https://github.com/aslafy-z/helm-git/issues/15). So it would become:

git+https://code.greenhost.net/openappstack/nextcloud@?ref=4-use-onlyoffice-instead-of-collabora

but from what I've seen (cached google page of https://code.greenhost.net/openappstack/nextcloud), the chart is in a nextcloud subfolder (according to the google cached page), it's also possible to use:

git+https://code.greenhost.net/openappstack/nextcloud@nextcloud?ref=4-use-onlyoffice-instead-of-collabora

I understand the URL syntax could be misleading, sorry for that. It can still be discussed, I'd be pleased to have your feedback!

mumoshu commented 5 years ago

Thanks for your support @aslafy-z!

From what I see he's using this branch https://code.greenhost.net/openappstack/nextcloud/tree/4-use-onlyoffice-instead-of-collabora for testing, and nextcloud seems like the repository name, where there are two charts(=directories) in it:

So I bet your first suggestion git+https://code.greenhost.net/openappstack/nextcloud@?ref=4-use-onlyoffice-instead-of-collabora works.

Now hoping either of the two options works :) Thansk again for your help.

mumoshu commented 5 years ago

Oh so do you mean that it would work in both cases, @nextcloud and @?

mrtndwrd commented 5 years ago

Thanks for all your quick replies! That's spot on, @mumoshu

Now when I try to use that URL for the repo, I run into a chicken-and-egg problem...

I updated my requirements.yaml to this:

dependencies:                                                                    
  - name: nextcloud                                                              
    version: 1.1.1                                                               
    repository: "@stable"                                                        
    tags:                                                                        
      - nextcloud                                                                
  - name: onlyoffice-documentserver                                              
    version: 0.1.0                                                               
    repository: "@nextcloud"                                                     
    tags:                                                                        
      - onlyoffice                                                               
      - nextcloud

Because the onlyoffice-documentserver chart would now be in the same repo as the nextcloud-onlyoffice chart, correct?

So then I try to add the repo as follows:

$ helm repo add nextcloud git+https://code.greenhost.net/openappstack/nextcloud@?ref=4-use-onlyoffice-instead-of-collabora

But I get this error: 

Error: no repository definition for @nextcloud. Please add them via 'helm repo add'
Error in plugin 'helm-git': Error while helm_dependency_update
Error: Looks like "git+https://code.greenhost.net/openappstack/nextcloud@?ref=4-use-onlyoffice-instead-of-collabora" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error
aslafy-z commented 5 years ago

Oh so do you mean that it would work in both cases, @nextcloud and @?

No it shouldn't with @nextcloud, I was referring to the wrong branch (master, where there is a nextcloud folder). It could have worked with only @ but yeah, chicken and egg problem as one chart refer to the other :(

You can try that:

You could also try to set repository key from your dep to git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora instead

mrtndwrd commented 5 years ago

Thanks @aslafy-z Yes that was my starting position. And when I run helm repo add like you suggest and then run helm dep update for nextcloud-onlyoffice it works. But when I have these repos as described in my first post and I run helmfile apply, it doesn't...

mrtndwrd commented 5 years ago

You could also try to set repository key from your dep to git+https://code.greenhost.net/openappstack/nextcloud@onlyoffice-documentserver?ref=4-use-onlyoffice-instead-of-collabora instead

I tried that too in this post: https://github.com/roboll/helmfile/issues/726#issuecomment-506245134

mumoshu commented 5 years ago

As far as I can tell it seems to not empty it but set it to whatever is in settings.Home, but I don't really understand why that doesn't default to /root/.helm in my case

Yeah that's interesting. I see no suspicious code that can result in the empty value, at least in the recent helm code-base.

On helm dependency build, settings are parsed from flags: https://github.com/helm/helm/blob/b0b0accdfc84e154b3d48ec334cd5b4f9b345667/cmd/helm/dependency_build.go#L78

HELM_HOME is derived from the settings.Home: https://github.com/helm/helm/blob/7cad59091a9451b2aa4f95aa882ea27e6b195f98/pkg/plugin/plugin.go#L182

And the whole envvars should be inherited to the plugin command, in this case helm-git in: https://github.com/helm/helm/blob/7cad59091a9451b2aa4f95aa882ea27e6b195f98/pkg/getter/plugingetter.go#L67-L68

🤔

mumoshu commented 5 years ago

@mrtndwrd Do you see anything suspicious when you run helm home --debug?

For my it's like this - where the first line is settings.Home:

$ helm home --debug
/Users/myuesr/.helm
Repository: /Users/myuesr/.helm/repository
RepositoryFile: /Users/myuesr/.helm/repository/repositories.yaml
Cache: /Users/myuesr/.helm/repository/cache
Stable CacheIndex: /Users/myuesr/.helm/repository/cache/stable-index.yaml
Starters: /Users/myuesr/.helm/starters
LocalRepository: /Users/myuesr/.helm/repository/local
Plugins: /Users/myuesr/.helm/plugins
mrtndwrd commented 5 years ago

Not really...

# helm home --debug
/root/.helm
Repository: /root/.helm/repository
RepositoryFile: /root/.helm/repository/repositories.yaml
Cache: /root/.helm/repository/cache
Stable CacheIndex: /root/.helm/repository/cache/stable-index.yaml
Starters: /root/.helm/starters
LocalRepository: /root/.helm/repository/local
Plugins: /root/.helm/plugins
dene14 commented 4 years ago

Actually we have the same issue when using both helm-diff and helm-git plugins... Very annoying as for us git references are simply unusable right now... Any workaround for that?

afolarin commented 2 years ago

was there a resolution to this?

metajiji commented 2 years ago

Workaround

sed -i 's| \-\-skip\-refresh||g' "$(helm env HELM_PLUGINS)/helm-git/helm-git-plugin.sh"

More info https://github.com/aslafy-z/helm-git/pull/155

// upd Dont do this! Fork bomb! helm dep up calls recursively helm dep up for each helm repos and every time helm calls helm-git plugin where plugin calls helm dep up - and here we gotcha infinity loop.

aslafy-z commented 2 years ago

This should now be fixed with https://github.com/aslafy-z/helm-git/releases/tag/v0.11.2. Thank you @metajiji for your work on this issue. Can you have a look @mrtndwrd?

yxxhero commented 2 years ago

@afolarin https://github.com/roboll/helmfile/issues/2148

mrtndwrd commented 2 years ago

Sorry, I can't confirm whether this solves the original issue because we've since moved to hosting our charts on Helm repositories and we've also replaced helmfile with flux2/helm-controller.

The charts and branches from this issue's description are also gone, so it would be quite a lot of work to bring them back.

Congrats on solving it!

7 Jun 2022 13:42:07 Zadkiel Aharonian @.***>:

This should now be fixed with https://github.com/aslafy-z/helm-git/releases/tag/v0.11.2. Thank you @metajiji[https://github.com/metajiji] for your work on this issue. Can you have a look @mrtndwrd[https://github.com/mrtndwrd]?

— Reply to this email directly, view it on GitHub[https://github.com/roboll/helmfile/issues/726#issuecomment-1148553013], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAGLJUZ43AJDTNTCIAGPAG3VN4YQZANCNFSM4H3RWLLA]. You are receiving this because you were mentioned.[Tracking image][https://github.com/notifications/beacon/AAGLJUYUIRIEU5ATXZFRWPTVN4YQZA5CNFSM4H3RWLLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIR2YONI.gif]