search

roboll / helmfile

Deploy Kubernetes Helm Charts
MIT License
4.05k stars 565 forks source link

Environment values not being used in nested states #1045

Open YingjunHu opened 4 years ago

YingjunHu commented 4 years ago

I've been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a specific environment but never was able to. Here's what my file structure and content look like. Structure:

├── environments
│   └── exp
│      └── values.yaml
├── helmfile.yaml
└── releases
    ├── release1.yaml
    └── release2.yaml

helmfile.yaml:

helmfiles:
 - "releases/release1.yaml"
 - "releases/release2.yaml"
environments:
  exp:
    values:
      - environments/exp/values.yaml

environments/exp/values.yaml:

testVersion: 1.0.0

releases/release1.yaml:

releases:
  - name: test-release
    chart: test/test-release
    version: {{ .Environment.Values.testVersion }}

Ideally when I run helmfile with arg -e exp, I expect all values in environments/exp/values.yaml to be injected to all helmfiles in releases folder. But I always got an error like executing "stringTemplate" at <.Environment.Values.testVersion>: map has no entry for key "testVersion"

When I enabled the debug level logging, I can see that values are loaded in my main helmfile.yaml from envvals_loader, but it seems like it's not passed to the sub helmfiles. Is this expected? And how do I achieve this if it's designed to be like this?

jrgwv commented 4 years ago

I have experienced a similar issue and am curious to see what the response / workaround / fix is for this.

YingjunHu commented 4 years ago

So I kinda found a solution but seems like there's another problem with it. I was able to inject the values from environments by separating the environments piece out of the main helmfile but create another file to store it and set that file as the base for all the sub helmfiles. Now the files looks like this:

├── commons
│   └── environments.yaml
├── environments
│   └── exp
│      └── values.yaml
├── helmfile.yaml
└── releases
    ├── release1.yaml
    └── release2.yaml

helmfile.yaml:

helmfiles:
 - "releases/release1.yaml"
 - "releases/release2.yaml"

commons/environments.yaml:

environments:
  exp:
    values:
      - ../environments/exp/values.yaml

releases/release1.yaml:

---
bases:
  - ../commons/environments.yaml
---
releases:
  - name: test-release
    chart: test/test-release
    version: {{ .Environment.Values.testVersion }}

But now it comes another problem that main helmfile will complain about err: no releases found that matches specified selector() and environment(exp), in any helmfile. And I can't set the environments.yaml file as the base for the main helmfile since the relative path of values file defined in environment.yaml is different for main and sub helfiles. And the workaround for the above problem is to add environments in the main helmfile again (even though it doesn't do anything). helmfiles:

helmfiles:
 - "releases/release1.yaml"
 - "releases/release2.yaml"
environments:
  exp:
    values:
      - environments/exp/values.yaml

And now I'm able to let all my helmfiles in releases folder working with environments. Even though I don't think it's the right/perfect way to write helmfile. Can someone point if I'm doing anything wrong here?

mumoshu commented 4 years ago

@YingjunHu Hey! To make each (sub-)helmfile modular, environment values are not automatically inherited to sub-helmfiles.

If you need to "override" environment values from the parent helmfile.yaml, you need to explicitly say:

helmfiles:
 - path: "releases/release1.yaml"
    values:
    - environments/exp/values.yaml
 - path: "releases/release2.yaml"
    values:
    - environments/exp/values.yaml
mumoshu commented 4 years ago

@YingjunHu I think you've almost managed it in your example at https://github.com/roboll/helmfile/issues/1045#issuecomment-567990337

The important point here is that a base helmfile is evaluated in the context(working directory, state values, etc) of current helmfile.yaml and that's why you are seeing

the base for the main helmfile since the relative path of values file defined in environment.yaml is different for main and sub helfiles.

Your last example seems ok to me. But you need a more DRY way to do it?

helmfiles:
 - "releases/release1.yaml"
 - "releases/release2.yaml"
environments:
  exp:
    values:
      - environments/exp/values.yaml

Fundamentally, Helmfile template isn't a full-fledged programming language so there might be no perfect way, but anyway I can suggest defining inline values within the environment.yaml:

environments:
  exp:
    values:
    - <contents of environments/exp/values.yaml = a yaml hash>
andrewnazarov commented 4 years ago

I'm using kinda the same workaround as stated here https://github.com/roboll/helmfile/issues/1045#issuecomment-567990337

Works quite ok, but looks not that great;)

tyuio9 commented 4 years ago

@YingjunHu I think, it may help you:

environments:
  production: {}
  development: {}

helmfiles:
  - path: releases/*/helmfile.yaml
    values:
      - environments/{{ .Environment.Name }}.yaml
mtb-xt commented 4 years ago

@YingjunHu this is what I did: If your environments are in one file, i.e. common/environments.yaml, you use this file as 'base' in your subhelmfiles:

#releases/something.yaml
bases:
- ../common/environments.yaml
- ../common/defaults.yaml

AND to avoid repetition in your top-level helmfile, use gotemplate black magic:

# helmfile.yaml
environments:
{{ $environments := (readFile "common/environments.yaml" | fromYaml ) }}{{ range $key, $env := (index $environments "environments") }}  {{ printf "%s:\n" $key }}{{ end }}
---
# Ordered list of releases.
helmfiles:
  - "releases/something.yaml"
valerius257 commented 4 years ago

@YingjunHu I think, it may help you:

environments:
  production: {}
  development: {}

helmfiles:
  - path: releases/*/helmfile.yaml
    values:
      - environments/{{ .Environment.Name }}.yaml

This is the closest workaround so far but if my understanding is correct than values from default environment are always used and values from a specific environment are used as overrides.

sshishov commented 4 years ago

I have similar issue but more difficult as we want to have modular releases:

├── commons
│   └── environments.yaml
├── environments
│   └── exp
│      └── values.yaml
├── helmfile.yaml
└── releases
    ├── module1
    |  ├── release1.yaml
    |  └── release2.yaml
    ├── module2
    |  ├── release1.yaml
    |  └── release2.yaml
    ├── module1.yaml (contains only `helmfiles:` to module)
    └── module2.yaml (contains only `helmfiles:` to module)

The problem is with relatives path here. Inside helmfile.yaml it should be one path for environments. on module1.yaml it should be another path and on release1.yaml it will be third path. And it become completely unmanagable when these files use some "external" files, for instance scripts/some.sh which cannot be referenced as every time the path is different.

mumoshu commented 4 years ago

@sshishov Thanks, that does seem hard. Do you have any ideal solution in your mind?

Anyway, I'd suggest passing the absolute path to the scripts directory via state values from the root helmfile.yaml down to module helmfiles, like:

helmfiles:
- path: module2/release1.yaml
  values:
  - scriptsDir: {{ .Values.scriptsDir }}

Then in release1.yaml you could use scriptsDir to build a absolute path to the script you want to load.

notjames commented 3 years ago

This is a good thread, but it's still not quite clear to me how to solve this problem. What's more is that everybody is going to have a different approach and there doesn't seem to be a one-way-fits-them-all. This would be a great topic from which to create documentation as I think this is probably one of the biggest problems to solve and it stems from the very beginning of a project when trying to determine how to structure the directories with respect to the project.

I've been working on my project for a couple of days now and some things are starting to gel for me for helmfile, but this is the one thing I'm still not getting.

I'm trying to keep this project as DRY as possible. I require to be able to deploy from the very basic application chart all the way to a full deployment, which is typically just a single cluster.

Given the following directory structure:

├── helmfile
│   ├── envs
│   │   ├── dev
│   │   │   └── cluster-a
│   │   ├── preprod
│   │   └── production
│   └── shlib
└── helmfile.d
    └── generic
        ├── 01a-network-and-proxies
        │   ├── ambassador
        │   ├── external-dns
        │   └── ingress-nginx
        ├── 01b-secrets-management
        │   ├── certmanager
        │   ├── dex
        │   ├── oauth2-proxy
        │   │   └── secrets
        │   ├── vault-operator
        │   │   └── secrets
        │   └── vault-secrets-webhook
        └── common

the way I'm framing this is:

an engineer can deploy dex by going to the dex directory under helmfile.d/generic/dex and running SOMEVAR=foo ANOTHERVAR=bar helmfile sync

The dex helmfile.yaml looks like: 

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
values:
  - aws_nlb_with_tls_termination_at_lb: {{ env "AWS_NLB" | default false }}
  - arm64_support: {{ env "ARM64" | default false }}
  - domain_name: {{ requiredEnv "DOMAIN_NAME" }}
  - cluster_id: {{ requiredEnv "CLUSTER_ID" }}

---
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: bluescape/dex
    version: {{ .Values.dex.version }}
    values:
      - values.yaml.gotmpl

most of the rest of my helmfile.yaml manifests look like this. This works very well for a single bundle deployment (such as dex here) or even deploying from the helmfile.d directory.

The problem occurs when I want to deploy cluster-a which I'm trying to do by changing directories to cluster-a (see above), which has a helmfile.yaml, and config.yaml. The idea was that the config.yaml which is simply a values definition manifest would override anything in the deployment.

❯ \cat helmfile.yaml 
environments:
  default:
    values:
      - config.yaml

---
helmfiles:
  - "../../../../helmfile.d/"
❯ \cat config.yaml 
---
values:
  - aws_nlb_with_tls_termination_at_lb: false
  - arm64_support: false
  - domain_name: "cluster-a.dev.domain.io"
  - cluster_id: "cluster-a"

However, when I test this I get a SIGSEGV nil pointer dereference:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x12cbc6e]

goroutine 1 [running]:
github.com/roboll/helmfile/pkg/state.(*Storage).normalizePath(0xc0002c5050, 0xc00003d560, 0x28, 0xc00028a1b0, 0x4e8491)
        /home/circleci/workspace/helmfile/pkg/state/storage.go:89 +0x3e
github.com/roboll/helmfile/pkg/state.(*Storage).ExpandPaths(0xc0002c5050, 0xc00003d560, 0x28, 0x1, 0x1, 0xc00019d1a0, 0x17, 0xc000323901)
        /home/circleci/workspace/helmfile/pkg/state/storage.go:74 +0x5a
github.com/roboll/helmfile/pkg/state.(*Storage).resolveFile(0xc0002c5050, 0x0, 0x17021dd, 0x12, 0xc00003d560, 0x28, 0x0, 0x199, 0x199, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/state/storage.go:32 +0xee
github.com/roboll/helmfile/pkg/state.(*EnvironmentValuesLoader).LoadEnvironmentValues(0xc00028a6a8, 0x0, 0xc000137140, 0x4, 0x4, 0x4, 0x4, 0x4)
        /home/circleci/workspace/helmfile/pkg/state/envvals_loader.go:48 +0x578
github.com/roboll/helmfile/pkg/state.(*HelmState).loadValuesEntries(0xc00037d900, 0x0, 0xc000137100, 0x4, 0x4, 0xc000094900, 0x17525e8, 0x1752c08, 0xc000250a00)
        /home/circleci/workspace/helmfile/pkg/state/create.go:360 +0x17e
github.com/roboll/helmfile/pkg/state.(*StateCreator).LoadEnvValues(0xc0005ded00, 0xc00037cf00, 0x16f43cc, 0x7, 0xc0001ca640, 0x0, 0xc00019c600, 0x14, 0xc00037cf00)
        /home/circleci/workspace/helmfile/pkg/state/create.go:156 +0x27d
github.com/roboll/helmfile/pkg/state.(*StateCreator).ParseAndLoad(0xc0005ded00, 0xc0000700e0, 0x6e, 0x70, 0x16ef515, 0x1, 0xc00019c600, 0x14, 0x16f43cc, 0x7, ...)
        /home/circleci/workspace/helmfile/pkg/state/create.go:189 +0x101
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderPrestate(0xc00041c280, 0xc0001ca640, 0x16ef515, 0x1, 0xc00019c600, 0x14, 0xc00022c0c2, 0x231, 0x231, 0x0, ...)
        /home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:55 +0x46d
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).twoPassRenderTemplateToYaml(0xc00041c280, 0xc00037ce70, 0x0, 0x16ef515, 0x1, 0xc00019c600, 0x14, 0xc00022c0c2, 0x231, 0x231, ...)
        /home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:99 +0x101
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderTemplatesToYamlWithEnv(...)
        /home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:81
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderAndLoad(0xc00041c280, 0x0, 0x0, 0x16ef515, 0x1, 0xc00050cff0, 0xd, 0xc000228000, 0x487, 0x687, ...)
        /home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:194 +0x430
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).loadFileWithOverrides(0xc00041c280, 0x0, 0x0, 0x16ef515, 0x1, 0xc00032500b, 0xd, 0xc000334501, 0x199, 0x203000, ...)
        /home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:113 +0x1ec
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).Load(0xc00041c280, 0xc00032500b, 0xd, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:65 +0x2f8
github.com/roboll/helmfile/pkg/app.(*App).loadDesiredStateFromYaml(0xc0000f7c20, 0xc00032500b, 0xd, 0xc00028ba40, 0x1, 0x1, 0x1, 0xc000226000, 0x92)
        /home/circleci/workspace/helmfile/pkg/app/app.go:630 +0x288
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1(0xc00032500b, 0xd, 0xc00032f000, 0x7a, 0x0, 0x0)
        /home/circleci/workspace/helmfile/pkg/app/app.go:682 +0x176
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles.func1(0xc000332620, 0x6f)
        /home/circleci/workspace/helmfile/pkg/app/app.go:596 +0x9d
github.com/roboll/helmfile/pkg/app.(*App).within(0xc0000f7c20, 0xc000325000, 0xa, 0xc00061be58, 0xc00061be18, 0x2)
        /home/circleci/workspace/helmfile/pkg/app/app.go:557 +0x33e
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles(0xc0000f7c20, 0xc000325000, 0x18, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:590 +0x2f2
github.com/roboll/helmfile/pkg/app.(*App).visitStates(0xc0000f7c20, 0xc000325000, 0x18, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:675 +0x10b
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1.2(0xc000000068, 0x1750d00)
        /home/circleci/workspace/helmfile/pkg/app/app.go:735 +0x31c
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1(0xc00032c410, 0xd, 0xc00034e5a0, 0x84, 0x0, 0x0)
        /home/circleci/workspace/helmfile/pkg/app/app.go:752 +0x916
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles.func1(0xc0003322a0, 0x64)
        /home/circleci/workspace/helmfile/pkg/app/app.go:596 +0x9d
github.com/roboll/helmfile/pkg/app.(*App).within(0xc0000f7c20, 0xc00032c3f0, 0x1f, 0xc00061c840, 0xc00061c800, 0x2)
        /home/circleci/workspace/helmfile/pkg/app/app.go:557 +0x33e
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles(0xc0000f7c20, 0xc000324840, 0x1f, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:590 +0x2f2
github.com/roboll/helmfile/pkg/app.(*App).visitStates(0xc0000f7c20, 0xc000324840, 0x1f, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:675 +0x10b
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1.2(0xc000000068, 0x1750d00)
        /home/circleci/workspace/helmfile/pkg/app/app.go:735 +0x31c
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1(0xc00032c017, 0xd, 0xc00032c060, 0x25, 0x0, 0x0)
        /home/circleci/workspace/helmfile/pkg/app/app.go:752 +0x916
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles.func1(0xc000330050, 0x44)
        /home/circleci/workspace/helmfile/pkg/app/app.go:596 +0x9d
github.com/roboll/helmfile/pkg/app.(*App).within(0xc0000f7c20, 0xc00032c000, 0x16, 0xc00061d228, 0xc00061d1e8, 0x2)
        /home/circleci/workspace/helmfile/pkg/app/app.go:557 +0x33e
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles(0xc0000f7c20, 0xc0004539a0, 0x16, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:590 +0x2f2
github.com/roboll/helmfile/pkg/app.(*App).visitStates(0xc0000f7c20, 0xc0004539a0, 0x16, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:675 +0x10b
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1.2(0xc000000068, 0x1750d00)
        /home/circleci/workspace/helmfile/pkg/app/app.go:735 +0x31c
github.com/roboll/helmfile/pkg/app.(*App).visitStates.func1(0x16fb8b9, 0xd, 0xc0000365a0, 0x52, 0x0, 0x0)
        /home/circleci/workspace/helmfile/pkg/app/app.go:752 +0x916
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles.func1(0x0, 0x7fccbd422108)
        /home/circleci/workspace/helmfile/pkg/app/app.go:596 +0x9d
github.com/roboll/helmfile/pkg/app.(*App).within(0xc0000f7c20, 0x16ef515, 0x1, 0xc00061dc10, 0xc00061dbd0, 0x2)
        /home/circleci/workspace/helmfile/pkg/app/app.go:538 +0x602
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles(0xc0000f7c20, 0x0, 0x0, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:590 +0x2f2
github.com/roboll/helmfile/pkg/app.(*App).visitStates(0xc0000f7c20, 0x0, 0x0, 0x27c0ad8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
        /home/circleci/workspace/helmfile/pkg/app/app.go:675 +0x10b
github.com/roboll/helmfile/pkg/app.(*App).visitStatesWithSelectorsAndRemoteSupport(0xc0000f7c20, 0x0, 0x0, 0xc00061df30, 0xc00061df98, 0x1, 0x1, 0xc00061df50, 0x133b749)
        /home/circleci/workspace/helmfile/pkg/app/app.go:930 +0x4c1
github.com/roboll/helmfile/pkg/app.(*App).ForEachState(0xc0000f7c20, 0xc00061dfa0, 0xc00061df98, 0x1, 0x1, 0x40a118, 0x16d5cc0)
        /home/circleci/workspace/helmfile/pkg/app/app.go:811 +0xb2
github.com/roboll/helmfile/pkg/app.(*App).Template(0xc0000f7c20, 0x1cf2640, 0xc00067e850, 0xc00067e850, 0xc00000f750)
        /home/circleci/workspace/helmfile/pkg/app/app.go:234 +0xc8
main.main.func5(0xc0000f7c20, 0xc0000c49a0, 0x0, 0xc00067e800, 0x0)
        /home/circleci/workspace/helmfile/main.go:266 +0x6d
main.action.func1(0xc0000c49a0, 0x0, 0xc0000a9720)
        /home/circleci/workspace/helmfile/main.go:793 +0xd4
github.com/urfave/cli.HandleAction(0x144d200, 0xc000044a80, 0xc0000c49a0, 0xc0000c49a0, 0x0)
        /home/circleci/workspace/helmfile/vendor/github.com/urfave/cli/app.go:524 +0x11a
github.com/urfave/cli.Command.Run(0x16f62d0, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x17386dd, 0x3d, 0x0, ...)
        /home/circleci/workspace/helmfile/vendor/github.com/urfave/cli/command.go:173 +0x57a
github.com/urfave/cli.(*App).Run(0xc0002da700, 0xc000030180, 0x3, 0x3, 0x0, 0x0)
        /home/circleci/workspace/helmfile/vendor/github.com/urfave/cli/app.go:277 +0x7c7
main.main()
        /home/circleci/workspace/helmfile/main.go:564 +0x3dce

When looking through the debug output it's clear to me what the problem is (though it shouldn't sigsegv ;) - the environment I try to set up in the cluster-a config at first gets read in fine. During runtime, the environment doesn't merge with the previously set environment. It just gets clobbered setting the values required to nil.

I tried doing something like this:

---                                                                                                                                                                                                                                                                                                                                                  
values:                                                                                                                                                                                                                                                                                                                                              
  - aws_nlb_with_tls_termination_at_lb: {{- if .Values.aws_nlb_with_tls_termination_at_lb }}{{ .Values.aws_nlb_with_tls_termination_at_lb }}{{- else}}{{ env "AWS_NLB" | default false }}{{- end }}                                                                                                                                                  
  - arm64_support: {{- if .Values.arm64_support }}{{ .Values.arm64_support }}{{- else}}{{ env "ARM64" | default false }}{{- end }}                                                                                                                                                                                                                   
  - domain_name: {{- if .Values.domain_name }}{{ .Values.domain_name }}{{- else}}{{ env "DOMAIN_NAME" }}{{- end }}                                                                                                                                                                                                                                   
  - cluster_id: {{- if .Values.cluster_id }}{{ .Values.cluster_id }}{{- else}}{{ env "CLUSTER_ID" }}{{- end }}

This doesn't work, though. The .Values always evaluate to false|nil which causes the last two values to be nil, which causes the SIGSEGV I believe.

For me, the documentation on this part of how helmfile works is not greatly understood. Some guidance would be helpful.

mumoshu commented 3 years ago

@notjames Hey! I may have missed details of your whole setup, but anyway- it looks like the issue is in:

environments:
  default:
    values:
      - config.yaml

---
helmfiles:
  - "../../../../helmfile.d/"

This doesn't make sense, as no environments (and its environment values defined in config.yaml) is supposed to be automatically inherited to any sub-helmfiles you've defined in helmfiles.

You need to somehow replicate that environments section in all the sub-helmfiles' helmfie.yaml. That way, each sub-helmfile can be run independently of the parent helmfile, which is the way helmfile thinks it should be.

Otherwise- if you really want to somehow make sub-helmfiles dependent on the parent, see https://github.com/roboll/helmfile/issues/1045#issuecomment-607538300

notjames commented 3 years ago 
environments:
  default:
    values:
      - config.yaml

---
helmfiles:
  - "../../../../helmfile.d/"

@mumoshu thank you for responding. I know I put a lot in my issue here. I think your response doesn't quite help me yet but that's likely because I gave you so much information that it's likely difficult to grok.

But perhaps I can sum it up like this. First, I love the concept that each helmfile can be run in its own right. That is exactly how I am trying to get this to work. So:

given the following scenarios: ---------------------------- SCENARIO 1 -------------------------------

DESIRE: I just want to deploy dex in an adhoc manner PWD: helmfile-project/helmfile.d/generic/01b-secrets-management/dex TREE:

dex/
├── helmfile.yaml
└── values.yaml.gotmpl

HELMFILE:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      app: dex
    chart: bluescape/dex
    version: {{ .Values.dex.version }}
    values:
      - values.yaml.gotmpl

I can run helmfile and dex will deploy successfully. Cool!

---------------------------- SCENARIO 2 -------------------------------

New scenario: Let's go up one directory:

DESIRE: I want to deploy everything in this directory with one helmfile sync command PWD: helmfile-project/helmfile.d/generic/01b-secrets-management TREE:

01b-secrets-management/
├── certmanager
│   ├── cluster-issuer-values.yaml
│   ├── config.yaml
│   ├── helmfile.yaml
│   ├── secrets.yaml
│   └── values.yaml.gotmpl
├── dex
│   ├── helmfile.yaml
│   └── values.yaml.gotmpl
├── helmfile.yaml
├── oauth2-proxy
│   ├── arm64-values.yaml.gotmpl
│   ├── create-ns
│   ├── generate-secret-cookie
│   ├── helmfile.yaml
│   ├── secrets
│   ├── values.yaml.gotmpl
│   └── wait_for_endpoint.sh
├── vault-operator
│   ├── helmfile.yaml
│   ├── secrets
│   │   └── vault-cr-secret-dec.yaml
│   └── values.yaml
└── vault-secrets-webhook
    ├── helmfile.yaml
    ├── values.yaml
    └── values.yaml.gotmpl

HELMFILE:

helmfiles:
  - "*/helmfile.yaml"
  - path: */helmfile.yaml
    selectors:
      - tier=secrets-managment

This works as desired. Any required variables that the helmfile manifests need, in this setup, is passed as environment variables IE VAR=foo VAR=bar helmfile sync and that's fine for running helmfile in an adhoc way.

---------------------------- SCENARIO 3 -------------------------------

OK...so hopefully this is making sense in the pattern I'm laying down.

New scenario:

NOW CHANGE IT UP A LITTLE: Up until now, I've made running helmfile adhoc to work fine and this is something we desire. But let's say that now I want to run helmfile from a directory containing definition configs for our environments so that instead of defining the variables on the command line with environment variables, I want to run helmfile from a directory with a configuration that defines everything in environments, which is what I've been trying to do and miserably failing and I can't quite figure out how to accomplish this. My attempt was:

DESIRE: run helmfile using a configuration (NOT ADHOC WITH ENV VARS) which defines static variables and uses them as helmfile values. PWD: bluescape/ops/helmfile-project/helmfile (important to note that this directory is parallel to helmfile.d TREE:

helmfile
└── envs
    ├── dev
    │   └── cluster-n
    │       ├── config.yaml
    │       └── helmfile.yaml
    ├── preprod
    └── production

I want to deploy cluster-n so I want to chdir to helmfile/envs/dev/cluster-a and simply run helmfile

My first question is: will helmfile support the method of release desired in this configuration

If so: the problem I'm having is that the following helmfile.yaml within helmfile/envs/dev/cluster-a which looks like and is apparently the helmfile.yaml which you address above, looks like: HELMFILE:

environments:
  default:
    values:
      - config.yaml

---
helmfiles:
  - "../../../../helmfile.d/"

This is meant to be a top-level configuration file with the intention, as mentioned, to be able to run helmfile like: helmfile sync from this path and whatever is defined in the top-level helmfile.yaml in this path will over-write the definitions in each of the release helmfile.yaml files "below" (defined in helmfile.d)

So I think I have a fundamental lack of understand of layering, but I believe I understand a little bit of it. If I can make this work the way I've outlined it here then that would rock.

Please note that this is from my understanding of the documentation, which is pretty complicated to understand when coming into this project with no context. Referentially, the documentation is great. Trying to define an actual working complicated project from the documentation is tough, because the documentation glances over the more sophisticated areas of helmfile (layering) in my estimation.

I'll try to understand what you've mentioned so far and fix things, but I don't quite fully understand what you're saying yet, so hopefully this makes more sense about what I'm trying to accomplish and you can address my mistakes. :)

notjames commented 3 years ago

@sshishov Thanks, that does seem hard. Do you have any ideal solution in your mind?

Anyway, I'd suggest passing the absolute path to the scripts directory via state values from the root helmfile.yaml down to module helmfiles, like:

helmfiles:
- path: module2/release1.yaml
  values:
  - scriptsDir: {{ .Values.scriptsDir }}

Then in release1.yaml you could use scriptsDir to build a absolute path to the script you want to load.

So this helmfile.yaml example is which one? :) It's not clear to me where I should define these settings. In other words, which is the "root" helmfile.yaml in the context of my question? Is it the one the one for the environment or the one in helmfile.d?