Helmfile doesnt error if helm-secrets fail

[philicious]

I noticed that helmfile doesnt error if there is a problem with helm-secrets decryption.

By mistake, new CI didnt have the plugin installed at all but helmfile still showed

Decrypting secret /...../secrets.yaml
Decrypting /...../secrets.yaml

Still, if plugin is installed, but decryption fails for a reason (e.g. not allowed to decrypt), helmfile will still show the same output.

Took me a while to figure out why it showed secrets in diff/sync locally but not in CI.

It would be great if helmfile would error or at least warn in such cases !

[mumoshu]

@philicious Hey! Thanks for reporting.

Just curious, but does helm secrets exit with non-zero status in that case? All Helmfile does is calling helm secrets and seeing the exit status to decide if it failed or not. So if it's the issue in helm secrets, I'm not sure how this can be resolved on helmfile side.

[philicious]

here are logs of the scenario where I found the problem:

1. helm secrets plugin is not even installed but helmfile diffs or syncs the release with exit code 0 = all good

139699d43dbe:~/project# helmfile -f deploy/helmfile.yaml --namespace backend --environment $STAGE diff
Decrypting secret /root/project/deploy/staging/secrets.yaml
Adding repo bahnx https://xxx.github.io/helm/
"xxx" has been added to your repositories
...
...
139699d43dbe:~/project# echo $?
0

2. helm secrets by now installed but authentication with the KMS backend hasnt happened and decryption obviously fails then. you can see, that it now prints a second Decrypting.. line, which likely is the piped output of helm-secrets plugin while the first line is from helmfile, just before calling helm secrets dec. but still, helmfile exits with 0.

139699d43dbe:~/project# helmfile -f deploy/helmfile.yaml --namespace backend --environment $STAGE diff
Decrypting secret /root/project/deploy/staging/secrets.yaml
Decrypting /root/project/deploy/staging/secrets.yaml

Adding repo bahnx https://xxx.github.io/helm/
"xxx" has been added to your repositories
...
...
139699d43dbe:~/project# echo $?
0

if I then try to decrypt with helm secrets, I do get a bad exit code.

139699d43dbe:~/project# helm tiller run -- helm secrets dec deploy/staging/secrets.yaml
Installed Helm version v2.15.2
Installed Tiller version v2.15.2
Helm and Tiller are the same version!
Starting Tiller...
Tiller namespace: kube-system
Running: helm secrets dec deploy/staging/secrets.yaml

Decrypting deploy/staging/secrets.yaml
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/xxx/locations/global/keyRings/xxx/cryptoKeys/team-xxx: FAILED
    - | Cannot create GCP KMS service: google: could not find
      | default credentials. See
      | https://developers.google.com/accounts/docs/application-default-credentials
      | for more information.

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.
Error: plugin "secrets" exited with error
Stopping Tiller...
Error: plugin "tiller" exited with error
139699d43dbe:~/project# echo $?
1

as helmfile happily exited, it took me a little while to figure out I missed helm-secrets in this new CI setup and that KMS auth hadnt been properly initialized. I only saw CI being green, logs looked good, but for unknown reason the secrets werent deployed ?! then I debugged it by running helm secrets decmanually and then realized what was going on

[philicious]

@mumoshu any update on this ? we just had another encounter of this bug and debugging can be really annoying if there is no error shown

[mumoshu]

@philicious Unforutnately, I can't reproduce this... Would you mind sharing the version number of your helmfile?

Also, some debug logs obtained by running helmfile --log-level=debug YOURCMD would be helpful

[mumoshu]

With helmfile v0.118.7, this is what I get:

$ helm version -c
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

$ helmfile template
Fetching incubator/raw
Decrypting secret /path/to/helmfile/issue1163/secret.enc.yaml
Decrypting /path/to/helmfile/issue1163/secret.enc.yaml

in ./helmfile.yaml: helm exited with status 1:
  Failed to get the data key required to decrypt the SOPS file.

  Group 0: FAILED
    : FAILED
      - | Error creating AWS session: No valid ARN found in ""

  Recovery failed because no master key was able to decrypt the file. In
  order for SOPS to recover the file, at least one key has to be successful,
  but none were.
  Error: plugin "secrets" exited with error

$ echo $?
1

[philicious]

I am seeing this again on a new macbook:

# helmfile version v0.119.0

Decrypting secret /Users/xx/git/helm/charts/rabbitmq/staging/secrets.yaml
Decrypting /Users/xx/git/helm/charts/rabbitmq/staging/secrets.yaml

in ./xxx.yaml: in .helmfiles[4]: in rabbitmq/helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:12:38: executing "stringTemplate" at <.Values.managementPassword>: map has no entry for key "managementPassword"

the helmfile doesnt forward the helm-secrets error but only complains about the secret values then not being present.

doing a helm tiller run -- helm secrets dec rabbitmq/staging/secrets.yaml does show me the error.

however helmfile exits with bad exit code at least ! my initial report was with v0.93.2 and that exited with 0 = all good

so now it would be great if helmfile would also pipe the error of helm-secrets so its more obvious what the problem might be

[dbluxo]

Same here with version v0.134.0