search

pelotech / drone-helm3

Plugin for drone to deploy helm charts using helm3
Apache License 2.0
31 stars 36 forks source link

Expand environment variables into Values and StringValues #34

Closed ErinCall closed 4 years ago

ErinCall commented 4 years ago

In order to pass secrets to helm, a user might put the following in their .drone.yml:

settings:
  values: "ssl_key=${SSL_KEY}"
environment:
  ssl_key:
    from_secret: ssl_key

When reading config from the environment, check cfg.Values and cfg.StringValues against the pattern \$\{\w+\} and substitute the corresponding environment variables. They need to respect the Prefix setting, using the same semantics as for regular config: use the non-prefixed form if it's present, but the prefixed form should override the non-prefixed.

Blocked on #9 and #19 (both fixed in #32).

ErinCall commented 4 years ago

Taking this out of version 1.0—it turns out drone does that interpolation for us already, so this would only be useful for making them respect the prefix setting, and none of the .drone.ymls I've found would need that feature.

ErinCall commented 4 years ago

Not needed at all—we're dropping the prefix setting (see #48), so this has no use case.

vitobotta commented 4 years ago

Hi @ErinCall

I am having a problem with environment variables not being expanded. Example take from my .drone.yml:

    settings:
      ...
      values: 
        - "image.tag=v${DRONE_COMMIT_SHA:0:7}"
        - "rails_master_key=${RAILS_MASTER_KEY}"
    environment:
      ...
      RAILS_MASTER_KEY:
        from_secret: rails_master_key

For some reason DRONE_COMMIT_SHA is expanded but my custom variables like RAILS_MASTER_KEY are not and the values remain empty. Any idea of what I am doing wrong? Thanks!

ErinCall commented 4 years ago

@vitobotta hm, I'm not sure—Drone should be taking care of it.

IIUC, Drone secrets aren't available on pull-request builds (or at least it didn't in 0.8; I can't find any information one way or the other in the 1.0 docs). Could that be the cause?

vitobotta commented 4 years ago

Hi @ErinCall I am pushing to master, no PRs. I have just changed plugin, everything was working fine before with the old plugin. Weird

ErinCall commented 4 years ago

Well heck, maybe we need this after all. Would you be willing to do a run with debug: true and put the output here? I'm mostly trying to see whether the plugin is getting PLUGIN_VALUES=rails_master_key=${RAILS_MASTER_KEY} or just PLUGIN_VALUES=rails_master_key=, but more context is always better :)

vitobotta commented 4 years ago

With debug: true I can see that all of them are empty apart from the image tag taken from the drone commit sha:

Values:image.tag=vdcda3a3,rails_master_key=,rails_secret_key_base=,lockbox_master_key=,mysql.master_url=,mysql.slave_url=,mail.username=,mail.password=,skylight.authentication=,sentry_dsn=,s3.access_key_id=,s3.secret_access_key=

There's no PLUGIN_VALUES text in the debug output

vitobotta commented 4 years ago

Here's the whole output:

Generated config: {Command:upgrade DroneEvent:push UpdateDependencies:false AddRepos:[] RepoCAFile: Debug:true Values:image.tag=vdcda3a3,rails_master_key=,rails_secret_key_base=,lockbox_master_key=,mysql.master_url=,mysql.slave_url=,mail.username=,mail.password=,skylight.authentication=,sentry_dsn=,s3.access_key_id=,s3.secret_access_key= StringValues: ValuesFiles:[./helm/values/myapp/staging/values.yaml] Namespace:myapp-staging KubeToken:(redacted) SkipTLSVerify:true Certificate: APIServer:https://kubernetes.default.svc.cluster.local:443 ServiceAccount: ChartVersion: DryRun:false Wait:true ReuseValues:false KeepHistory:false Timeout: Chart:./helm/chart Release:myapp-staging Force:false AtomicUpgrade:false CleanupOnFail:false LintStrictly:false Stdout:0xc0000a8008 Stderr:0xc0000a8010}
calling *run.InitKube.Prepare (step 0)
loading kubeconfig template from /root/.kube/config.tpl
creating kubeconfig file at /root/.kube/config
calling *run.Upgrade.Prepare (step 1)
Generated command: '/usr/bin/helm --debug --namespace myapp-staging upgrade --install --wait --set image.tag=vdcda3a3,rails_master_key=,rails_secret_key_base=,lockbox_master_key=,mysql.master_url=,mysql.slave_url=,mail.username=,mail.password=,skylight.authentication=,sentry_dsn=,s3.access_key_id=,s3.secret_access_key= --values ./helm/values/myapp/staging/values.yaml myapp-staging ./helm/chart'
calling *run.InitKube.Execute (step 0)
writing kubeconfig file to /root/.kube/config
calling *run.Upgrade.Execute (step 1)
history.go:52: [debug] getting history for release myapp-staging
upgrade.go:79: [debug] preparing upgrade for myapp-staging
upgrade.go:87: [debug] performing update for myapp-staging
upgrade.go:225: [debug] creating upgraded release for myapp-staging
client.go:222: [debug] Starting delete for "myapp-config" ConfigMap
client.go:89: [debug] creating 1 resource(s)
client.go:222: [debug] Starting delete for "myapp-secrets" Secret
client.go:89: [debug] creating 1 resource(s)
client.go:89: [debug] creating 1 resource(s)
upgrade.go:280: [debug] warning: Upgrade "myapp-staging" failed: pre-upgrade hooks failed: warning: Hook pre-upgrade myapp/templates/migrate_job.yaml failed: jobs.batch "myapp-db-migrate" already exists
Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade myapp/templates/migrate_job.yaml failed: jobs.batch "myapp-db-migrate" already exists
helm.go:76: [debug] pre-upgrade hooks failed: warning: Hook pre-upgrade myapp/templates/migrate_job.yaml failed: jobs.batch "myapp-db-migrate" already exists
UPGRADE FAILED
main.newUpgradeCmd.func1
    /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:136
github.com/spf13/cobra.(*Command).execute
    /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
    /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
    /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
    /home/circleci/helm.sh/helm/cmd/helm/helm.go:75
runtime.main
    /usr/local/go/src/runtime/proc.go:203
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1357
while executing *run.Upgrade step: exit status 1
vitobotta commented 4 years ago

And this is the deploy step in drone.yml:

  - name: deploy-staging
    image: pelotech/drone-helm3
    settings:
      mode: upgrade
      chart: ./helm/chart
      release: myapp-staging
      namespace: myapp-staging
      debug: true
      wait_for_upgrade: true
      skip_tls_verify: true
      values_files:
        - ./helm/values/myapp/staging/values.yaml
      values: 
        - "image.tag=v${DRONE_COMMIT_SHA:0:7}"
        - "rails_master_key=${RAILS_MASTER_KEY}"
        - "rails_secret_key_base=${RAILS_SECRET_KEY_BASE}"
        - "lockbox_master_key=${LOCKBOX_MASTER_KEY}"
        - "mysql.master_url=${MYSQL_MASTER_URL}"
        - "mysql.slave_url=${MYSQL_SLAVE_URL}"
        - "mail.username=${MAIL_USERNAME}"
        - "mail.password=${MAIL_PASSWORD}"
        - "skylight.authentication=${SKYLIGHT_AUTHENTICATION}"
        - "sentry_dsn=${SENTRY_DSN}"
        - "s3.access_key_id=${S3_ACCESS_KEY_ID}"
        - "s3.secret_access_key=${S3_SECRET_ACCESS_KEY}"
    environment:
      KUBE_API_SERVER: https://kubernetes.default.svc.cluster.local:443
      SKYLIGHT_ENABLED: "true"
      KUBE_TOKEN:
        from_secret: kubernetes_token
      RAILS_MASTER_KEY:
        from_secret: rails_master_key
      RAILS_SECRET_KEY_BASE:
        from_secret: rails_secret_key_base
      LOCKBOX_MASTER_KEY:
        from_secret: lockbox_master_key
      MYSQL_MASTER_URL:
        from_secret: staging_mysql_master_url
      MYSQL_SLAVE_URL:
        from_secret: staging_mysql_slave_url
      MAIL_USERNAME:
        from_secret: staging_smtp_user
      MAIL_PASSWORD:
        from_secret: staging_smtp_password        
      SKYLIGHT_AUTHENTICATION:
        from_secret: staging_skylight_authentication
      SENTRY_DSN: ""
      S3_ACCESS_KEY_ID: 
        from_secret: s3_access_key_id
      S3_SECRET_ACCESS_KEY: 
        from_secret: s3_secret_access_key
vitobotta commented 4 years ago

@ErinCall anything else that might be useful? Thanks!

ErinCall commented 4 years ago

Great, thanks for all that information. I'll get a start on implementing this shortly.

(sorry about the unclear PLUGIN_VALUES mention--Drone turns settings into PLUGIN_SETTINGNAME env vars when it invokes a plugin)

vitobotta commented 4 years ago

Thanks a lot, I am stuck so I appreciate a lot! :)

ErinCall commented 4 years ago

Er...I said the exact opposite of what I should have. Drone is turning the ${...}s into empty strings before it invokes the plugin, so the plugin never gets to see which environment variables to look up. Sorry about that 😞

It's strange that it worked with the old plugin, though. Can you show what your drone.yml looked like before you switched to this plugin?

Also, you're using Drone 1.0, right?

vitobotta commented 4 years ago

@ErinCall 1.6.4 to be precise. The drone.yml was identical apart from the plugin used and a couple of settings that have been renamed. Everything else was the same :)

vitobotta commented 4 years ago

Why the DRONE_COMMIT_SHA variable is being expanded though?

ErinCall commented 4 years ago

Well heck, I'm not sure what the problem might be. I'll do some digging and see what I can find out.

vitobotta commented 4 years ago

Thanks! @ErinCall

ErinCall commented 4 years ago

Ok, well, I don't know why it was working in drone-helm, but I have enough to go on.

  1. DRONE_COMMIT_SHA is showing up because Drone only allows interpolating values from a specific list of variables, including that one.
  2. The other variables expanding to empty strings is expected behavior. In order to pass them into the plugin, they'll need to be escaped with a second dollar sign (rails_master_key=$${RAILS_MASTER_KEY}) or listed without the curly braces (rails_master_key=$RAILS_MASTER_KEY).
  3. We do need to implement this feature.

Thanks for bringing this up and for all the helpful context! I'll get it ready for you shortly :)

vitobotta commented 4 years ago

@ErinCall You're awesome :) So you are going to publish an update? Or should I add the second dollar sign for now?

ErinCall commented 4 years ago

@vitobotta both :) I've created #77, which can do string interpolation as long as you avoid Drone's syntax—the entries in values should look like either $${RAILS_MASTER_KEY} or $RAILS_MASTER_KEY.

I built an image from that branch and pushed it to docker-hub, if you'd like to try it out.

vitobotta commented 4 years ago

@ErinCall Testing it now. :)

vitobotta commented 4 years ago

@ErinCall I think the env variables are being replaced now but I get this:

Error: UPGRADE FAILED: query: failed to query with labels: Unauthorized
--
27 | helm.go:76: [debug] Unauthorized
28 | query: failed to query with labels

helm.sh/helm/v3/pkg/storage/driver.(*Secrets).Query
--
31 | /home/circleci/helm.sh/helm/pkg/storage/driver/secrets.go:121
32 | helm.sh/helm/v3/pkg/storage.(*Storage).DeployedAll
33 | /home/circleci/helm.sh/helm/pkg/storage/storage.go:133
34 | helm.sh/helm/v3/pkg/storage.(*Storage).Deployed
35 | /home/circleci/helm.sh/helm/pkg/storage/storage.go:113
36 | helm.sh/helm/v3/pkg/action.(*Upgrade).prepareUpgrade
37 | /home/circleci/helm.sh/helm/pkg/action/upgrade.go:122
38 | helm.sh/helm/v3/pkg/action.(*Upgrade).Run
39 | /home/circleci/helm.sh/helm/pkg/action/upgrade.go:80
40 | main.newUpgradeCmd.func1
41 | /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:134
42 | github.com/spf13/cobra.(*Command).execute
43 | /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
44 | github.com/spf13/cobra.(*Command).ExecuteC
45 | /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
46 | github.com/spf13/cobra.(*Command).Execute
47 | /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
48 | main.main
49 | /home/circleci/helm.sh/helm/cmd/helm/helm.go:75
50 | runtime.main
51 | /usr/local/go/src/runtime/proc.go:203
52 | runtime.goexit
53 | /usr/local/go/src/runtime/asm_amd64.s:1357
54 | UPGRADE FAILED
55 | main.newUpgradeCmd.func1
56 | /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:136
57 | github.com/spf13/cobra.(*Command).execute
58 | /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
59 | github.com/spf13/cobra.(*Command).ExecuteC
60 | /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
61 | github.com/spf13/cobra.(*Command).Execute
62 | /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
63 | main.main
64 | /home/circleci/helm.sh/helm/cmd/helm/helm.go:75
65 | runtime.main
66 | /usr/local/go/src/runtime/proc.go:203
67 | runtime.goexit
68 | /usr/local/go/src/runtime/asm_amd64.s:1357
69 | while executing *run.Upgrade step: exit status 1
ErinCall commented 4 years ago

I'm not sure what would cause that one. It sounds like it's just a problem with the RBAC for the helm k8s user, except that the error you posted earlier showed helm getting farther along than it did this time.

Are you able to run helm upgrade directly from the command line?

vitobotta commented 4 years ago

@ErinCall My bad! I migrated to a new cluster and when I restored my Drone setup with Velero I forgot to update the Kubernetes token secret. All is working with that image, thanks! :) Do you mind updating this thread when the final image is out? :)

ErinCall commented 4 years ago

@vitobotta The PR is merged and published as version 0.14.0!

vitobotta commented 4 years ago

@ErinCall wonderful, thanks!